2023-10-06T04:53:30.865 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-10-06T04:53:30.866 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2023-10-05_23:54:57-fs-wip-yuri3-testing-2023-10-05-0943-quincy-distro-default-smithi/7412314 --verbose --name yuriw-2023-10-05_23:54:57-fs-wip-yuri3-testing-2023-10-05-0943-quincy-distro-default-smithi --description fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-readahead} -- /home/teuthworker/archive/yuriw-2023-10-05_23:54:57-fs-wip-yuri3-testing-2023-10-05-0943-quincy-distro-default-smithi/7412314/orig.config.yaml 2023-10-06T04:53:30.911 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-06T04:53:31.012 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-10-05_23:54:57-fs-wip-yuri3-testing-2023-10-05-0943-quincy-distro-default-smithi/7412314 branch: wip-yuri3-testing-2023-10-05-0943-quincy description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-readahead} email: yweinste@redhat.com first_in_suite: false job_id: '7412314' kernel: client: sha1: distro kdb: true sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: yuriw-2023-10-05_23:54:57-fs-wip-yuri3-testing-2023-10-05-0943-quincy-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.4' overrides: admin_socket: branch: wip-yuri3-testing-2023-10-05-0943-quincy ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: 68e53a4a7d4ae06d4fc1a0a7cfef840d1d942564 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 68e53a4a7d4ae06d4fc1a0a7cfef840d1d942564 kclient: syntax: v2 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri3-testing-2023-10-05-0943-quincy sha1: 68e53a4a7d4ae06d4fc1a0a7cfef840d1d942564 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 4002 sha1: 68e53a4a7d4ae06d4fc1a0a7cfef840d1d942564 sleep_before_teardown: 0 subset: 111/12000 suite: fs suite_branch: wip-yuri3-testing-2023-10-05-0943-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_68e53a4a7d4ae06d4fc1a0a7cfef840d1d942564/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 68e53a4a7d4ae06d4fc1a0a7cfef840d1d942564 targets: smithi098.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAdLN1jqITOyOs2/LG9xBqbeR1dU57UfESY/Js/od2mIfqcEEbswvtH7X0gj0zBZVDFXiTLyYoFCX8PqhzYepWI= smithi174.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKTO+hRfafQJmPn7d061Qdv6KXPKph9+gBvigvYovlxzFKbU3ueuwcnmiJKd4IHyY+NWJGJPMku58GXgpv6R/oQ= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - ceph: null - kclient: null - cephfs_test_runner: fail_on_skip: false modules: - tasks.cephfs.test_readahead teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 54e62bcbac4e53d9685e08328b790d3b20d71cae timestamp: 2023-10-05_23:54:57 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-10-06T04:53:31.012 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_68e53a4a7d4ae06d4fc1a0a7cfef840d1d942564/qa; will attempt to use it 2023-10-06T04:53:31.013 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_68e53a4a7d4ae06d4fc1a0a7cfef840d1d942564/qa/tasks 2023-10-06T04:53:31.014 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-10-06T04:53:31.015 INFO:teuthology.task.internal:Checking packages... 2023-10-06T04:53:31.038 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '68e53a4a7d4ae06d4fc1a0a7cfef840d1d942564' 2023-10-06T04:53:31.038 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-10-06T04:53:31.038 INFO:teuthology.packaging:ref: None 2023-10-06T04:53:31.038 INFO:teuthology.packaging:tag: None 2023-10-06T04:53:31.038 INFO:teuthology.packaging:branch: wip-yuri3-testing-2023-10-05-0943-quincy 2023-10-06T04:53:31.038 INFO:teuthology.packaging:sha1: 68e53a4a7d4ae06d4fc1a0a7cfef840d1d942564 2023-10-06T04:53:31.039 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri3-testing-2023-10-05-0943-quincy 2023-10-06T04:53:31.194 INFO:teuthology.task.internal:Found packages for ceph version 17.2.6-1274.g68e53a4a 2023-10-06T04:53:31.194 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-10-06T04:53:31.199 INFO:teuthology.task.internal:no buildpackages task found 2023-10-06T04:53:31.199 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-10-06T04:53:31.222 INFO:teuthology.task.internal:Saving configuration 2023-10-06T04:53:31.242 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-10-06T04:53:31.247 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-06T04:53:31.273 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi098.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-05_23:54:57-fs-wip-yuri3-testing-2023-10-05-0943-quincy-distro-default-smithi/7412314', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-06 04:48:18.746010', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAdLN1jqITOyOs2/LG9xBqbeR1dU57UfESY/Js/od2mIfqcEEbswvtH7X0gj0zBZVDFXiTLyYoFCX8PqhzYepWI='} 2023-10-06T04:53:31.296 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi174.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-05_23:54:57-fs-wip-yuri3-testing-2023-10-05-0943-quincy-distro-default-smithi/7412314', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-06 04:48:18.747732', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKTO+hRfafQJmPn7d061Qdv6KXPKph9+gBvigvYovlxzFKbU3ueuwcnmiJKd4IHyY+NWJGJPMku58GXgpv6R/oQ='} 2023-10-06T04:53:31.296 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-10-06T04:53:31.301 INFO:teuthology.task.internal:roles: ubuntu@smithi098.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2023-10-06T04:53:31.302 INFO:teuthology.task.internal:roles: ubuntu@smithi174.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2023-10-06T04:53:31.302 INFO:teuthology.run_tasks:Running task console_log... 2023-10-06T04:53:31.361 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fc7ea4f4b80>, signals=[15]) 2023-10-06T04:53:31.361 INFO:teuthology.run_tasks:Running task internal.connect... 2023-10-06T04:53:31.367 INFO:teuthology.task.internal:Opening connections... 2023-10-06T04:53:31.367 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi098.front.sepia.ceph.com 2023-10-06T04:53:31.369 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-06T04:53:31.437 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi174.front.sepia.ceph.com 2023-10-06T04:53:31.438 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-06T04:53:31.503 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-10-06T04:53:31.509 DEBUG:teuthology.orchestra.run.smithi098:> uname -m 2023-10-06T04:53:31.527 INFO:teuthology.orchestra.run.smithi098.stdout:x86_64 2023-10-06T04:53:31.527 DEBUG:teuthology.orchestra.run.smithi098:> cat /etc/os-release 2023-10-06T04:53:31.583 INFO:teuthology.orchestra.run.smithi098.stdout:NAME="Red Hat Enterprise Linux" 2023-10-06T04:53:31.583 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION="8.4 (Ootpa)" 2023-10-06T04:53:31.583 INFO:teuthology.orchestra.run.smithi098.stdout:ID="rhel" 2023-10-06T04:53:31.583 INFO:teuthology.orchestra.run.smithi098.stdout:ID_LIKE="fedora" 2023-10-06T04:53:31.584 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION_ID="8.4" 2023-10-06T04:53:31.584 INFO:teuthology.orchestra.run.smithi098.stdout:PLATFORM_ID="platform:el8" 2023-10-06T04:53:31.584 INFO:teuthology.orchestra.run.smithi098.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2023-10-06T04:53:31.584 INFO:teuthology.orchestra.run.smithi098.stdout:ANSI_COLOR="0;31" 2023-10-06T04:53:31.584 INFO:teuthology.orchestra.run.smithi098.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2023-10-06T04:53:31.584 INFO:teuthology.orchestra.run.smithi098.stdout:HOME_URL="https://www.redhat.com/" 2023-10-06T04:53:31.584 INFO:teuthology.orchestra.run.smithi098.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-10-06T04:53:31.584 INFO:teuthology.orchestra.run.smithi098.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-06T04:53:31.584 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T04:53:31.584 INFO:teuthology.orchestra.run.smithi098.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-10-06T04:53:31.584 INFO:teuthology.orchestra.run.smithi098.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2023-10-06T04:53:31.585 INFO:teuthology.orchestra.run.smithi098.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-10-06T04:53:31.585 INFO:teuthology.orchestra.run.smithi098.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2023-10-06T04:53:31.585 INFO:teuthology.lock.ops:Updating smithi098.front.sepia.ceph.com on lock server 2023-10-06T04:53:31.611 DEBUG:teuthology.orchestra.run.smithi174:> uname -m 2023-10-06T04:53:31.627 INFO:teuthology.orchestra.run.smithi174.stdout:x86_64 2023-10-06T04:53:31.627 DEBUG:teuthology.orchestra.run.smithi174:> cat /etc/os-release 2023-10-06T04:53:31.683 INFO:teuthology.orchestra.run.smithi174.stdout:NAME="Red Hat Enterprise Linux" 2023-10-06T04:53:31.683 INFO:teuthology.orchestra.run.smithi174.stdout:VERSION="8.4 (Ootpa)" 2023-10-06T04:53:31.683 INFO:teuthology.orchestra.run.smithi174.stdout:ID="rhel" 2023-10-06T04:53:31.683 INFO:teuthology.orchestra.run.smithi174.stdout:ID_LIKE="fedora" 2023-10-06T04:53:31.683 INFO:teuthology.orchestra.run.smithi174.stdout:VERSION_ID="8.4" 2023-10-06T04:53:31.683 INFO:teuthology.orchestra.run.smithi174.stdout:PLATFORM_ID="platform:el8" 2023-10-06T04:53:31.684 INFO:teuthology.orchestra.run.smithi174.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2023-10-06T04:53:31.684 INFO:teuthology.orchestra.run.smithi174.stdout:ANSI_COLOR="0;31" 2023-10-06T04:53:31.684 INFO:teuthology.orchestra.run.smithi174.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2023-10-06T04:53:31.684 INFO:teuthology.orchestra.run.smithi174.stdout:HOME_URL="https://www.redhat.com/" 2023-10-06T04:53:31.684 INFO:teuthology.orchestra.run.smithi174.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-10-06T04:53:31.684 INFO:teuthology.orchestra.run.smithi174.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-06T04:53:31.684 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-10-06T04:53:31.685 INFO:teuthology.orchestra.run.smithi174.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-10-06T04:53:31.685 INFO:teuthology.orchestra.run.smithi174.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2023-10-06T04:53:31.685 INFO:teuthology.orchestra.run.smithi174.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-10-06T04:53:31.685 INFO:teuthology.orchestra.run.smithi174.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2023-10-06T04:53:31.685 INFO:teuthology.lock.ops:Updating smithi174.front.sepia.ceph.com on lock server 2023-10-06T04:53:31.710 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-10-06T04:53:31.719 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-10-06T04:53:31.731 INFO:teuthology.task.internal:Checking for old test directory... 2023-10-06T04:53:31.731 DEBUG:teuthology.orchestra.run.smithi098:> test '!' -e /home/ubuntu/cephtest 2023-10-06T04:53:31.734 DEBUG:teuthology.orchestra.run.smithi174:> test '!' -e /home/ubuntu/cephtest 2023-10-06T04:53:31.748 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-10-06T04:53:31.754 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-10-06T04:53:31.754 DEBUG:teuthology.orchestra.run.smithi098:> test -z $(ls -A /var/lib/ceph) 2023-10-06T04:53:31.790 DEBUG:teuthology.orchestra.run.smithi174:> test -z $(ls -A /var/lib/ceph) 2023-10-06T04:53:31.804 INFO:teuthology.orchestra.run.smithi098.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2023-10-06T04:53:31.804 INFO:teuthology.orchestra.run.smithi174.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2023-10-06T04:53:31.806 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-10-06T04:53:31.862 INFO:teuthology.run_tasks:Running task kernel... 2023-10-06T04:53:31.877 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'} 2023-10-06T04:53:31.878 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}, 'client.2': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}, 'client.3': {'sha1': 'distro'}} 2023-10-06T04:53:31.878 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}}, timeout 300 2023-10-06T04:53:31.878 DEBUG:teuthology.orchestra.run.smithi098:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-06T04:53:31.879 DEBUG:teuthology.orchestra.run.smithi174:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-06T04:53:31.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-06T04:53:31.893 DEBUG:teuthology.orchestra.run.smithi174:> uname -r 2023-10-06T04:53:31.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-06T04:53:31.895 DEBUG:teuthology.orchestra.run.smithi098:> uname -r 2023-10-06T04:53:31.948 INFO:teuthology.orchestra.run.smithi174.stdout:4.18.0-305.25.1.el8_4.x86_64 2023-10-06T04:53:31.948 INFO:teuthology.task.kernel:Running kernel on smithi174: 4.18.0-305.25.1.el8_4.x86_64 2023-10-06T04:53:31.949 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum install -y kernel 2023-10-06T04:53:31.951 INFO:teuthology.orchestra.run.smithi098.stdout:4.18.0-305.25.1.el8_4.x86_64 2023-10-06T04:53:31.951 INFO:teuthology.task.kernel:Running kernel on smithi098: 4.18.0-305.25.1.el8_4.x86_64 2023-10-06T04:53:31.951 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum install -y kernel 2023-10-06T04:53:33.411 INFO:teuthology.orchestra.run.smithi174.stdout:Updating Subscription Management repositories. 2023-10-06T04:53:33.411 INFO:teuthology.orchestra.run.smithi174.stdout:Unable to read consumer identity 2023-10-06T04:53:33.413 INFO:teuthology.orchestra.run.smithi098.stdout:Updating Subscription Management repositories. 2023-10-06T04:53:33.414 INFO:teuthology.orchestra.run.smithi098.stdout:Unable to read consumer identity 2023-10-06T04:53:34.731 INFO:teuthology.orchestra.run.smithi174.stdout:Copr repo for python3-asyncssh owned by ceph 27 kB/s | 3.5 kB 00:00 2023-10-06T04:53:34.819 INFO:teuthology.orchestra.run.smithi098.stdout:Copr repo for python3-asyncssh owned by ceph 27 kB/s | 3.5 kB 00:00 2023-10-06T04:53:35.253 INFO:teuthology.orchestra.run.smithi174.stdout:Extra Packages for Enterprise Linux 33 MB/s | 16 MB 00:00 2023-10-06T04:53:35.821 INFO:teuthology.orchestra.run.smithi098.stdout:Extra Packages for Enterprise Linux 16 MB/s | 16 MB 00:00 2023-10-06T04:53:38.551 INFO:teuthology.orchestra.run.smithi174.stdout:lab-extras 4.4 MB/s | 24 kB 00:00 2023-10-06T04:53:39.108 INFO:teuthology.orchestra.run.smithi098.stdout:lab-extras 4.4 MB/s | 24 kB 00:00 2023-10-06T04:53:39.267 INFO:teuthology.orchestra.run.smithi174.stdout:Package kernel-4.18.0-80.el8.x86_64 is already installed. 2023-10-06T04:53:39.267 INFO:teuthology.orchestra.run.smithi174.stdout:Package kernel-4.18.0-425.3.1.el8.x86_64 is already installed. 2023-10-06T04:53:39.267 INFO:teuthology.orchestra.run.smithi174.stdout:Package kernel-4.18.0-305.25.1.el8_4.x86_64 is already installed. 2023-10-06T04:53:39.435 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2023-10-06T04:53:39.436 INFO:teuthology.orchestra.run.smithi174.stdout:Nothing to do. 2023-10-06T04:53:39.436 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2023-10-06T04:53:39.547 DEBUG:teuthology.orchestra.run.smithi174:> echo no | sudo yum reinstall kernel || true 2023-10-06T04:53:39.864 INFO:teuthology.orchestra.run.smithi098.stdout:Package kernel-4.18.0-80.el8.x86_64 is already installed. 2023-10-06T04:53:39.864 INFO:teuthology.orchestra.run.smithi098.stdout:Package kernel-4.18.0-425.3.1.el8.x86_64 is already installed. 2023-10-06T04:53:39.864 INFO:teuthology.orchestra.run.smithi098.stdout:Package kernel-4.18.0-305.25.1.el8_4.x86_64 is already installed. 2023-10-06T04:53:39.890 INFO:teuthology.orchestra.run.smithi174.stdout:Updating Subscription Management repositories. 2023-10-06T04:53:39.891 INFO:teuthology.orchestra.run.smithi174.stdout:Unable to read consumer identity 2023-10-06T04:53:40.025 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2023-10-06T04:53:40.026 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2023-10-06T04:53:40.026 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2023-10-06T04:53:40.162 DEBUG:teuthology.orchestra.run.smithi098:> echo no | sudo yum reinstall kernel || true 2023-10-06T04:53:40.185 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:00:02 ago on Fri 06 Oct 2023 04:53:38 AM UTC. 2023-10-06T04:53:40.314 INFO:teuthology.orchestra.run.smithi174.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-10-06T04:53:40.314 INFO:teuthology.orchestra.run.smithi174.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-06T04:53:40.314 INFO:teuthology.orchestra.run.smithi174.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-06T04:53:40.346 INFO:teuthology.orchestra.run.smithi174.stderr:Error: No packages marked for reinstall. 2023-10-06T04:53:40.396 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum reinstall -y kernel || true 2023-10-06T04:53:40.546 INFO:teuthology.orchestra.run.smithi098.stdout:Updating Subscription Management repositories. 2023-10-06T04:53:40.546 INFO:teuthology.orchestra.run.smithi098.stdout:Unable to read consumer identity 2023-10-06T04:53:40.737 INFO:teuthology.orchestra.run.smithi174.stdout:Updating Subscription Management repositories. 2023-10-06T04:53:40.737 INFO:teuthology.orchestra.run.smithi174.stdout:Unable to read consumer identity 2023-10-06T04:53:40.832 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:00:01 ago on Fri 06 Oct 2023 04:53:39 AM UTC. 2023-10-06T04:53:40.980 INFO:teuthology.orchestra.run.smithi098.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-10-06T04:53:40.980 INFO:teuthology.orchestra.run.smithi098.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-06T04:53:40.980 INFO:teuthology.orchestra.run.smithi098.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-06T04:53:41.016 INFO:teuthology.orchestra.run.smithi098.stderr:Error: No packages marked for reinstall. 2023-10-06T04:53:41.021 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:00:03 ago on Fri 06 Oct 2023 04:53:38 AM UTC. 2023-10-06T04:53:41.066 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum reinstall -y kernel || true 2023-10-06T04:53:41.174 INFO:teuthology.orchestra.run.smithi174.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-10-06T04:53:41.174 INFO:teuthology.orchestra.run.smithi174.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-06T04:53:41.174 INFO:teuthology.orchestra.run.smithi174.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-06T04:53:41.189 INFO:teuthology.orchestra.run.smithi174.stderr:Error: No packages marked for reinstall. 2023-10-06T04:53:41.239 DEBUG:teuthology.orchestra.run.smithi174:> rpm -q kernel | sort -rV | head -n 1 2023-10-06T04:53:41.365 INFO:teuthology.orchestra.run.smithi174.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-10-06T04:53:41.366 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-10-06T04:53:41.366 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-305.25.1.el8_4.x86_64 Expected: 4.18.0-425.3.1.el8.x86_64 2023-10-06T04:53:41.366 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-06T04:53:41.366 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-06T04:53:41.366 INFO:teuthology.task.kernel:Installing distro kernel on client.1... 2023-10-06T04:53:41.366 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi174.front.sepia.ceph.com, path=None, version=distro) 2023-10-06T04:53:41.367 DEBUG:teuthology.orchestra.run.smithi174:> rpm -q kernel | sort -rV | head -n 1 2023-10-06T04:53:41.409 INFO:teuthology.orchestra.run.smithi098.stdout:Updating Subscription Management repositories. 2023-10-06T04:53:41.409 INFO:teuthology.orchestra.run.smithi098.stdout:Unable to read consumer identity 2023-10-06T04:53:41.491 INFO:teuthology.orchestra.run.smithi174.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-10-06T04:53:41.491 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-10-06T04:53:41.491 DEBUG:teuthology.orchestra.run.smithi174:> sudo rpm -qi grub2-tools 2023-10-06T04:53:41.519 INFO:teuthology.orchestra.run.smithi174.stdout:Name : grub2-tools 2023-10-06T04:53:41.519 INFO:teuthology.orchestra.run.smithi174.stdout:Epoch : 1 2023-10-06T04:53:41.519 INFO:teuthology.orchestra.run.smithi174.stdout:Version : 2.02 2023-10-06T04:53:41.519 INFO:teuthology.orchestra.run.smithi174.stdout:Release : 99.el8_4.1 2023-10-06T04:53:41.519 INFO:teuthology.orchestra.run.smithi174.stdout:Architecture: x86_64 2023-10-06T04:53:41.520 INFO:teuthology.orchestra.run.smithi174.stdout:Install Date: Wed 21 Dec 2022 01:02:32 PM UTC 2023-10-06T04:53:41.520 INFO:teuthology.orchestra.run.smithi174.stdout:Group : System Environment/Base 2023-10-06T04:53:41.520 INFO:teuthology.orchestra.run.smithi174.stdout:Size : 9264631 2023-10-06T04:53:41.520 INFO:teuthology.orchestra.run.smithi174.stdout:License : GPLv3+ 2023-10-06T04:53:41.520 INFO:teuthology.orchestra.run.smithi174.stdout:Signature : RSA/SHA256, Thu 09 Sep 2021 09:41:55 AM UTC, Key ID 199e2f91fd431d51 2023-10-06T04:53:41.520 INFO:teuthology.orchestra.run.smithi174.stdout:Source RPM : grub2-2.02-99.el8_4.1.src.rpm 2023-10-06T04:53:41.520 INFO:teuthology.orchestra.run.smithi174.stdout:Build Date : Tue 18 May 2021 07:46:34 AM UTC 2023-10-06T04:53:41.520 INFO:teuthology.orchestra.run.smithi174.stdout:Build Host : x86-vm-07.build.eng.bos.redhat.com 2023-10-06T04:53:41.520 INFO:teuthology.orchestra.run.smithi174.stdout:Relocations : (not relocatable) 2023-10-06T04:53:41.520 INFO:teuthology.orchestra.run.smithi174.stdout:Packager : Red Hat, Inc. 2023-10-06T04:53:41.520 INFO:teuthology.orchestra.run.smithi174.stdout:Vendor : Red Hat, Inc. 2023-10-06T04:53:41.521 INFO:teuthology.orchestra.run.smithi174.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-06T04:53:41.521 INFO:teuthology.orchestra.run.smithi174.stdout:Summary : Support tools for GRUB. 2023-10-06T04:53:41.521 INFO:teuthology.orchestra.run.smithi174.stdout:Description : 2023-10-06T04:53:41.521 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-10-06T04:53:41.521 INFO:teuthology.orchestra.run.smithi174.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-06T04:53:41.521 INFO:teuthology.orchestra.run.smithi174.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-06T04:53:41.521 INFO:teuthology.orchestra.run.smithi174.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-06T04:53:41.521 INFO:teuthology.orchestra.run.smithi174.stdout:hardware devices. 2023-10-06T04:53:41.521 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-10-06T04:53:41.521 INFO:teuthology.orchestra.run.smithi174.stdout:This subpackage provides tools for support of all platforms. 2023-10-06T04:53:41.522 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-06T04:53:41.522 INFO:teuthology.task.kernel:Updating grub on smithi174 to boot 4.18.0-425.3.1.el8.x86_64 2023-10-06T04:53:41.523 DEBUG:teuthology.orchestra.run.smithi174:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-06T04:53:41.700 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:00:02 ago on Fri 06 Oct 2023 04:53:39 AM UTC. 2023-10-06T04:53:41.835 INFO:teuthology.orchestra.run.smithi098.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-10-06T04:53:41.835 INFO:teuthology.orchestra.run.smithi098.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-06T04:53:41.835 INFO:teuthology.orchestra.run.smithi098.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-06T04:53:41.850 INFO:teuthology.orchestra.run.smithi098.stderr:Error: No packages marked for reinstall. 2023-10-06T04:53:41.904 DEBUG:teuthology.orchestra.run.smithi098:> rpm -q kernel | sort -rV | head -n 1 2023-10-06T04:53:42.029 INFO:teuthology.orchestra.run.smithi098.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-10-06T04:53:42.029 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-10-06T04:53:42.029 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-305.25.1.el8_4.x86_64 Expected: 4.18.0-425.3.1.el8.x86_64 2023-10-06T04:53:42.029 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-06T04:53:42.029 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-06T04:53:42.029 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2023-10-06T04:53:42.029 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi098.front.sepia.ceph.com, path=None, version=distro) 2023-10-06T04:53:42.030 DEBUG:teuthology.orchestra.run.smithi098:> rpm -q kernel | sort -rV | head -n 1 2023-10-06T04:53:42.155 INFO:teuthology.orchestra.run.smithi098.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-10-06T04:53:42.155 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-10-06T04:53:42.155 DEBUG:teuthology.orchestra.run.smithi098:> sudo rpm -qi grub2-tools 2023-10-06T04:53:42.182 INFO:teuthology.orchestra.run.smithi098.stdout:Name : grub2-tools 2023-10-06T04:53:42.182 INFO:teuthology.orchestra.run.smithi098.stdout:Epoch : 1 2023-10-06T04:53:42.182 INFO:teuthology.orchestra.run.smithi098.stdout:Version : 2.02 2023-10-06T04:53:42.182 INFO:teuthology.orchestra.run.smithi098.stdout:Release : 99.el8_4.1 2023-10-06T04:53:42.183 INFO:teuthology.orchestra.run.smithi098.stdout:Architecture: x86_64 2023-10-06T04:53:42.183 INFO:teuthology.orchestra.run.smithi098.stdout:Install Date: Wed 21 Dec 2022 01:02:32 PM UTC 2023-10-06T04:53:42.183 INFO:teuthology.orchestra.run.smithi098.stdout:Group : System Environment/Base 2023-10-06T04:53:42.183 INFO:teuthology.orchestra.run.smithi098.stdout:Size : 9264631 2023-10-06T04:53:42.183 INFO:teuthology.orchestra.run.smithi098.stdout:License : GPLv3+ 2023-10-06T04:53:42.183 INFO:teuthology.orchestra.run.smithi098.stdout:Signature : RSA/SHA256, Thu 09 Sep 2021 09:41:55 AM UTC, Key ID 199e2f91fd431d51 2023-10-06T04:53:42.183 INFO:teuthology.orchestra.run.smithi098.stdout:Source RPM : grub2-2.02-99.el8_4.1.src.rpm 2023-10-06T04:53:42.183 INFO:teuthology.orchestra.run.smithi098.stdout:Build Date : Tue 18 May 2021 07:46:34 AM UTC 2023-10-06T04:53:42.183 INFO:teuthology.orchestra.run.smithi098.stdout:Build Host : x86-vm-07.build.eng.bos.redhat.com 2023-10-06T04:53:42.183 INFO:teuthology.orchestra.run.smithi098.stdout:Relocations : (not relocatable) 2023-10-06T04:53:42.183 INFO:teuthology.orchestra.run.smithi098.stdout:Packager : Red Hat, Inc. 2023-10-06T04:53:42.183 INFO:teuthology.orchestra.run.smithi098.stdout:Vendor : Red Hat, Inc. 2023-10-06T04:53:42.183 INFO:teuthology.orchestra.run.smithi098.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-06T04:53:42.184 INFO:teuthology.orchestra.run.smithi098.stdout:Summary : Support tools for GRUB. 2023-10-06T04:53:42.184 INFO:teuthology.orchestra.run.smithi098.stdout:Description : 2023-10-06T04:53:42.184 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T04:53:42.184 INFO:teuthology.orchestra.run.smithi098.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-06T04:53:42.184 INFO:teuthology.orchestra.run.smithi098.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-06T04:53:42.184 INFO:teuthology.orchestra.run.smithi098.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-06T04:53:42.184 INFO:teuthology.orchestra.run.smithi098.stdout:hardware devices. 2023-10-06T04:53:42.184 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T04:53:42.184 INFO:teuthology.orchestra.run.smithi098.stdout:This subpackage provides tools for support of all platforms. 2023-10-06T04:53:42.185 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-06T04:53:42.185 INFO:teuthology.task.kernel:Updating grub on smithi098 to boot 4.18.0-425.3.1.el8.x86_64 2023-10-06T04:53:42.185 DEBUG:teuthology.orchestra.run.smithi098:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-06T04:53:42.907 INFO:teuthology.orchestra.run.smithi174.stderr:Generating grub configuration file ... 2023-10-06T04:53:43.578 INFO:teuthology.orchestra.run.smithi098.stderr:Generating grub configuration file ... 2023-10-06T04:53:44.984 INFO:teuthology.orchestra.run.smithi174.stderr:done 2023-10-06T04:53:44.986 DEBUG:teuthology.orchestra.run.smithi174:> mktemp 2023-10-06T04:53:45.000 INFO:teuthology.orchestra.run.smithi174.stdout:/tmp/tmp.j6Tm6tLSsC 2023-10-06T04:53:45.000 DEBUG:teuthology.orchestra.run.smithi174:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.j6Tm6tLSsC 2023-10-06T04:53:45.096 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod 0666 /tmp/tmp.j6Tm6tLSsC 2023-10-06T04:53:45.223 DEBUG:teuthology.orchestra.remote:smithi174:/tmp/tmp.j6Tm6tLSsC is 6KB 2023-10-06T04:53:45.270 DEBUG:teuthology.orchestra.run.smithi174:> rm -fr /tmp/tmp.j6Tm6tLSsC 2023-10-06T04:53:45.284 DEBUG:teuthology.orchestra.run.smithi174:> sudo /bin/ls /boot/loader/entries || true 2023-10-06T04:53:45.355 INFO:teuthology.orchestra.run.smithi174.stdout:298e228f3b5e401e8ac0f5ae159a2e06-0-rescue.conf 2023-10-06T04:53:45.355 INFO:teuthology.orchestra.run.smithi174.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-305.25.1.el8_4.x86_64.conf 2023-10-06T04:53:45.355 INFO:teuthology.orchestra.run.smithi174.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64.conf 2023-10-06T04:53:45.355 INFO:teuthology.orchestra.run.smithi174.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-80.el8.x86_64.conf 2023-10-06T04:53:45.356 DEBUG:teuthology.orchestra.run.smithi174:> sudo grub2-set-default 298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64 2023-10-06T04:53:45.521 DEBUG:teuthology.orchestra.run.smithi174:> sudo shutdown -r now 2023-10-06T04:53:45.699 INFO:teuthology.orchestra.run.smithi098.stderr:done 2023-10-06T04:53:45.701 DEBUG:teuthology.orchestra.run.smithi098:> mktemp 2023-10-06T04:53:45.715 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/tmp.TB5J3ekuuD 2023-10-06T04:53:45.716 DEBUG:teuthology.orchestra.run.smithi098:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.TB5J3ekuuD 2023-10-06T04:53:45.790 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0666 /tmp/tmp.TB5J3ekuuD 2023-10-06T04:53:45.948 DEBUG:teuthology.orchestra.remote:smithi098:/tmp/tmp.TB5J3ekuuD is 6KB 2023-10-06T04:53:45.997 DEBUG:teuthology.orchestra.run.smithi098:> rm -fr /tmp/tmp.TB5J3ekuuD 2023-10-06T04:53:46.010 DEBUG:teuthology.orchestra.run.smithi098:> sudo /bin/ls /boot/loader/entries || true 2023-10-06T04:53:46.077 INFO:teuthology.orchestra.run.smithi098.stdout:298e228f3b5e401e8ac0f5ae159a2e06-0-rescue.conf 2023-10-06T04:53:46.077 INFO:teuthology.orchestra.run.smithi098.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-305.25.1.el8_4.x86_64.conf 2023-10-06T04:53:46.077 INFO:teuthology.orchestra.run.smithi098.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64.conf 2023-10-06T04:53:46.077 INFO:teuthology.orchestra.run.smithi098.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-80.el8.x86_64.conf 2023-10-06T04:53:46.079 DEBUG:teuthology.orchestra.run.smithi098:> sudo grub2-set-default 298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64 2023-10-06T04:53:46.228 DEBUG:teuthology.orchestra.run.smithi098:> sudo shutdown -r now 2023-10-06T04:54:15.551 INFO:teuthology.task.kernel:Checking client client.1 for new kernel version... 2023-10-06T04:54:15.551 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2023-10-06T04:54:15.552 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-06T04:54:16.232 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2023-10-06T04:54:16.232 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2023-10-06T04:54:16.233 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-06T04:54:34.106 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.174 2023-10-06T04:54:34.618 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.98 2023-10-06T04:54:43.114 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2023-10-06T04:54:43.115 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-06T04:54:43.620 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2023-10-06T04:54:43.620 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-06T04:54:46.174 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.174 2023-10-06T04:54:46.682 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.98 2023-10-06T04:54:58.187 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2023-10-06T04:54:58.188 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-06T04:54:58.686 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2023-10-06T04:54:58.686 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-06T04:55:16.602 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.174 2023-10-06T04:55:17.114 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.98 2023-10-06T04:55:31.618 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2023-10-06T04:55:31.619 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-06T04:55:31.927 DEBUG:teuthology.orchestra.run.smithi174:> true 2023-10-06T04:55:32.115 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2023-10-06T04:55:32.115 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-06T04:55:32.282 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi174.front.sepia.ceph.com' 2023-10-06T04:55:32.282 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "4.18.0-425.3.1.el8.x86_64"... 2023-10-06T04:55:32.283 DEBUG:teuthology.orchestra.run.smithi174:> uname -r 2023-10-06T04:55:32.296 INFO:teuthology.orchestra.run.smithi174.stdout:4.18.0-425.3.1.el8.x86_64 2023-10-06T04:55:32.296 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-425.3.1.el8.x86_64 vs 4.18.0-425.3.1.el8.x86_64 2023-10-06T04:55:32.297 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-06T04:55:32.297 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2023-10-06T04:55:32.505 DEBUG:teuthology.orchestra.run.smithi098:> true 2023-10-06T04:55:32.766 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi098.front.sepia.ceph.com' 2023-10-06T04:55:32.766 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "4.18.0-425.3.1.el8.x86_64"... 2023-10-06T04:55:32.766 DEBUG:teuthology.orchestra.run.smithi098:> uname -r 2023-10-06T04:55:32.780 INFO:teuthology.orchestra.run.smithi098.stdout:4.18.0-425.3.1.el8.x86_64 2023-10-06T04:55:32.780 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-425.3.1.el8.x86_64 vs 4.18.0-425.3.1.el8.x86_64 2023-10-06T04:55:32.780 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-06T04:55:32.780 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2023-10-06T04:55:33.297 DEBUG:teuthology.parallel:result is None 2023-10-06T04:55:33.781 DEBUG:teuthology.parallel:result is None 2023-10-06T04:55:33.781 INFO:teuthology.run_tasks:Running task internal.base... 2023-10-06T04:55:33.789 INFO:teuthology.task.internal:Creating test directory... 2023-10-06T04:55:33.789 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-06T04:55:33.801 DEBUG:teuthology.orchestra.run.smithi174:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-06T04:55:34.038 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-10-06T04:55:34.046 INFO:teuthology.run_tasks:Running task internal.archive... 2023-10-06T04:55:34.060 INFO:teuthology.task.internal:Creating archive directory... 2023-10-06T04:55:34.061 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-06T04:55:34.113 DEBUG:teuthology.orchestra.run.smithi174:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-06T04:55:34.471 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-10-06T04:55:34.479 INFO:teuthology.task.internal:Enabling coredump saving... 2023-10-06T04:55:34.480 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-06T04:55:34.550 DEBUG:teuthology.orchestra.run.smithi174:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-06T04:55:35.195 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-06T04:55:35.220 INFO:teuthology.orchestra.run.smithi174.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-06T04:55:35.396 INFO:teuthology.orchestra.run.smithi174.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-06T04:55:35.433 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-06T04:55:35.457 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-10-06T04:55:35.464 INFO:teuthology.task.internal:Configuring sudo... 2023-10-06T04:55:35.465 DEBUG:teuthology.orchestra.run.smithi098:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-06T04:55:35.498 DEBUG:teuthology.orchestra.run.smithi174:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-06T04:55:35.921 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-10-06T04:55:35.930 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-10-06T04:55:35.930 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-06T04:55:35.932 DEBUG:teuthology.orchestra.run.smithi174:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-06T04:55:35.979 DEBUG:teuthology.orchestra.run.smithi098:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-06T04:55:36.028 DEBUG:teuthology.orchestra.run.smithi098:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-06T04:55:36.148 DEBUG:teuthology.orchestra.run.smithi098:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-06T04:55:36.191 DEBUG:teuthology.orchestra.run.smithi098:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-06T04:55:36.228 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-10-06T04:55:36.228 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-06T04:55:36.292 DEBUG:teuthology.orchestra.run.smithi174:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-06T04:55:36.334 DEBUG:teuthology.orchestra.run.smithi174:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-06T04:55:36.384 DEBUG:teuthology.orchestra.run.smithi174:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-06T04:55:36.467 DEBUG:teuthology.orchestra.run.smithi174:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-06T04:55:36.507 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2023-10-06T04:55:36.507 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-06T04:55:36.571 DEBUG:teuthology.orchestra.run.smithi098:> sudo service rsyslog restart 2023-10-06T04:55:36.573 DEBUG:teuthology.orchestra.run.smithi174:> sudo service rsyslog restart 2023-10-06T04:55:36.694 INFO:teuthology.orchestra.run.smithi098.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-06T04:55:36.869 INFO:teuthology.orchestra.run.smithi174.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-06T04:55:36.912 INFO:teuthology.run_tasks:Running task internal.timer... 2023-10-06T04:55:36.923 INFO:teuthology.task.internal:Starting timer... 2023-10-06T04:55:36.924 INFO:teuthology.run_tasks:Running task pcp... 2023-10-06T04:55:36.960 INFO:teuthology.run_tasks:Running task selinux... 2023-10-06T04:55:36.970 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:logrotate_t:s0']} 2023-10-06T04:55:36.970 DEBUG:teuthology.orchestra.run.smithi098:> sudo service auditd rotate 2023-10-06T04:55:37.018 INFO:teuthology.orchestra.run.smithi098.stdout:Rotating logs: [ OK ] 2023-10-06T04:55:37.020 DEBUG:teuthology.orchestra.run.smithi174:> sudo service auditd rotate 2023-10-06T04:55:37.091 INFO:teuthology.orchestra.run.smithi174.stdout:Rotating logs: [ OK ] 2023-10-06T04:55:37.092 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-10-06T04:55:37.093 DEBUG:teuthology.orchestra.run.smithi098:> /usr/sbin/getenforce 2023-10-06T04:55:37.110 INFO:teuthology.orchestra.run.smithi098.stdout:Permissive 2023-10-06T04:55:37.110 DEBUG:teuthology.orchestra.run.smithi174:> /usr/sbin/getenforce 2023-10-06T04:55:37.125 INFO:teuthology.orchestra.run.smithi174.stdout:Permissive 2023-10-06T04:55:37.125 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi098.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi174.front.sepia.ceph.com': 'permissive'} 2023-10-06T04:55:37.125 DEBUG:teuthology.orchestra.run.smithi098:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-06T04:55:37.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-06T04:55:37.176 DEBUG:teuthology.orchestra.run.smithi174:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-06T04:55:37.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-06T04:55:37.201 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-10-06T04:55:37.201 DEBUG:teuthology.orchestra.run.smithi098:> sudo /usr/sbin/setenforce permissive 2023-10-06T04:55:37.240 DEBUG:teuthology.orchestra.run.smithi174:> sudo /usr/sbin/setenforce permissive 2023-10-06T04:55:37.277 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-10-06T04:55:37.287 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-10-06T04:55:37.287 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-10-06T04:55:37.308 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-10-06T04:55:37.309 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi098.front.sepia.ceph.com,smithi174.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-10-06T05:03:47.649 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi098.front.sepia.ceph.com'), Remote(name='ubuntu@smithi174.front.sepia.ceph.com')] 2023-10-06T05:03:47.650 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2023-10-06T05:03:47.650 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-06T05:03:47.721 DEBUG:teuthology.orchestra.run.smithi098:> true 2023-10-06T05:03:47.743 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi098.front.sepia.ceph.com' 2023-10-06T05:03:47.744 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2023-10-06T05:03:47.744 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-06T05:03:47.812 DEBUG:teuthology.orchestra.run.smithi174:> true 2023-10-06T05:03:47.881 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi174.front.sepia.ceph.com' 2023-10-06T05:03:47.881 INFO:teuthology.run_tasks:Running task clock... 2023-10-06T05:03:47.895 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-10-06T05:03:47.895 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-06T05:03:47.896 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-06T05:03:47.898 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-06T05:03:47.898 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-06T05:03:47.925 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-10-06T05:03:47.940 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-10-06T05:03:47.951 INFO:teuthology.orchestra.run.smithi174.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-10-06T05:03:47.965 INFO:teuthology.orchestra.run.smithi174.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-10-06T05:03:47.977 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: ntpd: command not found 2023-10-06T05:03:47.987 INFO:teuthology.orchestra.run.smithi098.stdout:506 Cannot talk to daemon 2023-10-06T05:03:47.996 INFO:teuthology.orchestra.run.smithi174.stderr:sudo: ntpd: command not found 2023-10-06T05:03:48.001 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-10-06T05:03:48.007 INFO:teuthology.orchestra.run.smithi174.stdout:506 Cannot talk to daemon 2023-10-06T05:03:48.014 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-10-06T05:03:48.021 INFO:teuthology.orchestra.run.smithi174.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-10-06T05:03:48.034 INFO:teuthology.orchestra.run.smithi174.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-10-06T05:03:48.078 INFO:teuthology.orchestra.run.smithi098.stderr:bash: ntpq: command not found 2023-10-06T05:03:48.080 INFO:teuthology.orchestra.run.smithi098.stdout:210 Number of sources = 4 2023-10-06T05:03:48.080 INFO:teuthology.orchestra.run.smithi098.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-06T05:03:48.080 INFO:teuthology.orchestra.run.smithi098.stdout:=============================================================================== 2023-10-06T05:03:48.081 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-06T05:03:48.081 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-06T05:03:48.081 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-06T05:03:48.081 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-06T05:03:48.112 INFO:teuthology.orchestra.run.smithi174.stderr:bash: ntpq: command not found 2023-10-06T05:03:48.115 INFO:teuthology.orchestra.run.smithi174.stdout:210 Number of sources = 4 2023-10-06T05:03:48.115 INFO:teuthology.orchestra.run.smithi174.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-06T05:03:48.115 INFO:teuthology.orchestra.run.smithi174.stdout:=============================================================================== 2023-10-06T05:03:48.115 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-06T05:03:48.115 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-06T05:03:48.115 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-06T05:03:48.115 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-06T05:03:48.116 INFO:teuthology.run_tasks:Running task install... 2023-10-06T05:03:48.128 DEBUG:teuthology.task.install:project ceph 2023-10-06T05:03:48.128 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '68e53a4a7d4ae06d4fc1a0a7cfef840d1d942564'}} 2023-10-06T05:03:48.128 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'flavor': 'default', 'sha1': '68e53a4a7d4ae06d4fc1a0a7cfef840d1d942564'} 2023-10-06T05:03:48.128 INFO:teuthology.task.install:Using flavor: default 2023-10-06T05:03:48.134 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-10-06T05:03:48.134 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-10-06T05:03:48.135 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '68e53a4a7d4ae06d4fc1a0a7cfef840d1d942564', 'tag': None, 'wait_for_package': False} 2023-10-06T05:03:48.135 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=68e53a4a7d4ae06d4fc1a0a7cfef840d1d942564 2023-10-06T05:03:48.137 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '68e53a4a7d4ae06d4fc1a0a7cfef840d1d942564', 'tag': None, 'wait_for_package': False} 2023-10-06T05:03:48.137 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=68e53a4a7d4ae06d4fc1a0a7cfef840d1d942564 2023-10-06T05:03:48.353 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-05-0943-quincy/68e53a4a7d4ae06d4fc1a0a7cfef840d1d942564/centos/8/flavors/default/ 2023-10-06T05:03:48.353 INFO:teuthology.task.install.rpm:Package version is 17.2.6-1274.g68e53a4a 2023-10-06T05:03:48.368 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-05-0943-quincy/68e53a4a7d4ae06d4fc1a0a7cfef840d1d942564/centos/8/flavors/default/ 2023-10-06T05:03:48.369 INFO:teuthology.task.install.rpm:Package version is 17.2.6-1274.g68e53a4a 2023-10-06T05:03:48.747 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-05-0943-quincy/68e53a4a7d4ae06d4fc1a0a7cfef840d1d942564/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-05-0943-quincy/68e53a4a7d4ae06d4fc1a0a7cfef840d1d942564/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-05-0943-quincy/68e53a4a7d4ae06d4fc1a0a7cfef840d1d942564/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-10-06T05:03:48.748 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-10-06T05:03:48.748 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-10-06T05:03:48.750 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-05-0943-quincy/68e53a4a7d4ae06d4fc1a0a7cfef840d1d942564/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-05-0943-quincy/68e53a4a7d4ae06d4fc1a0a7cfef840d1d942564/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-10-05-0943-quincy/68e53a4a7d4ae06d4fc1a0a7cfef840d1d942564/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-10-06T05:03:48.750 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2023-10-06T05:03:48.750 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-10-06T05:03:48.776 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel on remote rpm x86_64 2023-10-06T05:03:48.776 DEBUG:teuthology.orchestra.run.smithi098:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/68e53a4a7d4ae06d4fc1a0a7cfef840d1d942564/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-10-06T05:03:48.777 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel on remote rpm x86_64 2023-10-06T05:03:48.777 DEBUG:teuthology.orchestra.run.smithi174:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/68e53a4a7d4ae06d4fc1a0a7cfef840d1d942564/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-10-06T05:03:48.841 DEBUG:teuthology.orchestra.run.smithi098:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-10-06T05:03:48.843 DEBUG:teuthology.orchestra.run.smithi174:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-10-06T05:03:48.916 DEBUG:teuthology.orchestra.run.smithi098:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-10-06T05:03:48.918 DEBUG:teuthology.orchestra.run.smithi174:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-10-06T05:03:48.979 INFO:teuthology.orchestra.run.smithi098.stdout:check_obsoletes = 1 2023-10-06T05:03:48.981 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum clean all 2023-10-06T05:03:48.982 INFO:teuthology.orchestra.run.smithi174.stdout:check_obsoletes = 1 2023-10-06T05:03:48.984 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum clean all 2023-10-06T05:03:49.367 INFO:teuthology.orchestra.run.smithi098.stdout:Updating Subscription Management repositories. 2023-10-06T05:03:49.374 INFO:teuthology.orchestra.run.smithi174.stdout:Updating Subscription Management repositories. 2023-10-06T05:03:49.666 INFO:teuthology.orchestra.run.smithi098.stdout:44 files removed 2023-10-06T05:03:49.673 INFO:teuthology.orchestra.run.smithi174.stdout:44 files removed 2023-10-06T05:03:49.719 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel 2023-10-06T05:03:49.723 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel 2023-10-06T05:03:50.069 INFO:teuthology.orchestra.run.smithi098.stdout:Updating Subscription Management repositories. 2023-10-06T05:03:50.070 INFO:teuthology.orchestra.run.smithi174.stdout:Updating Subscription Management repositories. 2023-10-06T05:03:51.103 INFO:teuthology.orchestra.run.smithi174.stdout:ceph packages for x86_64 141 kB/s | 78 kB 00:00 2023-10-06T05:03:51.129 INFO:teuthology.orchestra.run.smithi098.stdout:ceph packages for x86_64 129 kB/s | 78 kB 00:00 2023-10-06T05:03:51.525 INFO:teuthology.orchestra.run.smithi174.stdout:ceph noarch packages 24 kB/s | 9.2 kB 00:00 2023-10-06T05:03:51.600 INFO:teuthology.orchestra.run.smithi098.stdout:ceph noarch packages 21 kB/s | 9.2 kB 00:00 2023-10-06T05:03:52.320 INFO:teuthology.orchestra.run.smithi098.stdout:ceph source packages 2.1 kB/s | 1.4 kB 00:00 2023-10-06T05:03:52.335 INFO:teuthology.orchestra.run.smithi174.stdout:ceph source packages 1.8 kB/s | 1.4 kB 00:00 2023-10-06T05:03:52.431 INFO:teuthology.orchestra.run.smithi098.stdout:Copr repo for python3-asyncssh owned by ceph 42 kB/s | 3.5 kB 00:00 2023-10-06T05:03:52.447 INFO:teuthology.orchestra.run.smithi174.stdout:Copr repo for python3-asyncssh owned by ceph 42 kB/s | 3.5 kB 00:00 2023-10-06T05:03:53.682 INFO:teuthology.orchestra.run.smithi098.stdout:Extra Packages for Enterprise Linux 13 MB/s | 16 MB 00:01 2023-10-06T05:03:53.778 INFO:teuthology.orchestra.run.smithi174.stdout:Extra Packages for Enterprise Linux 12 MB/s | 16 MB 00:01 2023-10-06T05:03:56.961 INFO:teuthology.orchestra.run.smithi098.stdout:lab-extras 3.9 MB/s | 24 kB 00:00 2023-10-06T05:03:57.018 INFO:teuthology.orchestra.run.smithi174.stdout:lab-extras 4.4 MB/s | 24 kB 00:00 2023-10-06T05:03:57.318 INFO:teuthology.orchestra.run.smithi098.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 120 MB/s | 33 MB 00:00 2023-10-06T05:03:57.390 INFO:teuthology.orchestra.run.smithi174.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 113 MB/s | 33 MB 00:00 2023-10-06T05:04:02.424 INFO:teuthology.orchestra.run.smithi098.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 121 MB/s | 38 MB 00:00 2023-10-06T05:04:02.565 INFO:teuthology.orchestra.run.smithi174.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 125 MB/s | 38 MB 00:00 2023-10-06T05:04:10.820 INFO:teuthology.orchestra.run.smithi098.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-10-06T05:04:10.820 INFO:teuthology.orchestra.run.smithi098.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-10-06T05:04:10.831 INFO:teuthology.orchestra.run.smithi098.stdout:Package NetworkManager-1:1.30.0-13.el8_4.x86_64 is already installed. 2023-10-06T05:04:10.832 INFO:teuthology.orchestra.run.smithi098.stdout:Package iproute-5.9.0-4.el8.x86_64 is already installed. 2023-10-06T05:04:10.833 INFO:teuthology.orchestra.run.smithi098.stdout:Package util-linux-2.32.1-27.el8.x86_64 is already installed. 2023-10-06T05:04:10.834 INFO:teuthology.orchestra.run.smithi098.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-10-06T05:04:10.835 INFO:teuthology.orchestra.run.smithi098.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-10-06T05:04:10.838 INFO:teuthology.orchestra.run.smithi098.stdout:Package xfsdump-3.1.8-2.el8.x86_64 is already installed. 2023-10-06T05:04:10.838 INFO:teuthology.orchestra.run.smithi098.stdout:Package xfsprogs-5.0.0-8.el8.x86_64 is already installed. 2023-10-06T05:04:10.903 INFO:teuthology.orchestra.run.smithi174.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-10-06T05:04:10.903 INFO:teuthology.orchestra.run.smithi174.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-10-06T05:04:10.914 INFO:teuthology.orchestra.run.smithi174.stdout:Package NetworkManager-1:1.30.0-13.el8_4.x86_64 is already installed. 2023-10-06T05:04:10.915 INFO:teuthology.orchestra.run.smithi174.stdout:Package iproute-5.9.0-4.el8.x86_64 is already installed. 2023-10-06T05:04:10.916 INFO:teuthology.orchestra.run.smithi174.stdout:Package util-linux-2.32.1-27.el8.x86_64 is already installed. 2023-10-06T05:04:10.917 INFO:teuthology.orchestra.run.smithi174.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-10-06T05:04:10.918 INFO:teuthology.orchestra.run.smithi174.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-10-06T05:04:10.921 INFO:teuthology.orchestra.run.smithi174.stdout:Package xfsdump-3.1.8-2.el8.x86_64 is already installed. 2023-10-06T05:04:10.922 INFO:teuthology.orchestra.run.smithi174.stdout:Package xfsprogs-5.0.0-8.el8.x86_64 is already installed. 2023-10-06T05:04:11.325 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2023-10-06T05:04:11.336 INFO:teuthology.orchestra.run.smithi098.stdout:====================================================================================================================== 2023-10-06T05:04:11.336 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2023-10-06T05:04:11.336 INFO:teuthology.orchestra.run.smithi098.stdout:====================================================================================================================== 2023-10-06T05:04:11.336 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2023-10-06T05:04:11.336 INFO:teuthology.orchestra.run.smithi098.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-10-06T05:04:11.337 INFO:teuthology.orchestra.run.smithi098.stdout: ceph x86_64 2:17.2.6-1274.g68e53a4a.el8 ceph 6.4 k 2023-10-06T05:04:11.337 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-base x86_64 2:17.2.6-1274.g68e53a4a.el8 ceph 5.2 M 2023-10-06T05:04:11.337 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-fuse x86_64 2:17.2.6-1274.g68e53a4a.el8 ceph 877 k 2023-10-06T05:04:11.337 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-immutable-object-cache x86_64 2:17.2.6-1274.g68e53a4a.el8 ceph 162 k 2023-10-06T05:04:11.337 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr x86_64 2:17.2.6-1274.g68e53a4a.el8 ceph 1.6 M 2023-10-06T05:04:11.337 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-cephadm noarch 2:17.2.6-1274.g68e53a4a.el8 ceph-noarch 121 k 2023-10-06T05:04:11.337 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-dashboard noarch 2:17.2.6-1274.g68e53a4a.el8 ceph-noarch 1.7 M 2023-10-06T05:04:11.337 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.6-1274.g68e53a4a.el8 ceph-noarch 7.4 M 2023-10-06T05:04:11.337 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-rook noarch 2:17.2.6-1274.g68e53a4a.el8 ceph-noarch 47 k 2023-10-06T05:04:11.337 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-radosgw x86_64 2:17.2.6-1274.g68e53a4a.el8 ceph 17 M 2023-10-06T05:04:11.337 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-test x86_64 2:17.2.6-1274.g68e53a4a.el8 ceph 52 M 2023-10-06T05:04:11.337 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-volume noarch 2:17.2.6-1274.g68e53a4a.el8 ceph-noarch 267 k 2023-10-06T05:04:11.338 INFO:teuthology.orchestra.run.smithi098.stdout: cephadm noarch 2:17.2.6-1274.g68e53a4a.el8 ceph-noarch 80 k 2023-10-06T05:04:11.338 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-mirror x86_64 2:17.2.6-1274.g68e53a4a.el8 ceph 255 k 2023-10-06T05:04:11.338 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-top noarch 2:17.2.6-1274.g68e53a4a.el8 ceph-noarch 18 k 2023-10-06T05:04:11.338 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-libelf-devel x86_64 0.182-3.el8 rhel-8-for-x86_64-baseos-rpms 59 k 2023-10-06T05:04:11.338 INFO:teuthology.orchestra.run.smithi098.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-10-06T05:04:11.338 INFO:teuthology.orchestra.run.smithi098.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-10-06T05:04:11.338 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs-devel x86_64 2:17.2.6-1274.g68e53a4a.el8 ceph 25 k 2023-10-06T05:04:11.338 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs2 x86_64 2:17.2.6-1274.g68e53a4a.el8 ceph 797 k 2023-10-06T05:04:11.338 INFO:teuthology.orchestra.run.smithi098.stdout: librados-devel x86_64 2:17.2.6-1274.g68e53a4a.el8 ceph 130 k 2023-10-06T05:04:11.338 INFO:teuthology.orchestra.run.smithi098.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-10-06T05:04:11.338 INFO:teuthology.orchestra.run.smithi098.stdout: libuuid-devel x86_64 2.32.1-27.el8 rhel-8-for-x86_64-baseos-rpms 97 k 2023-10-06T05:04:11.339 INFO:teuthology.orchestra.run.smithi098.stdout: openssl-devel x86_64 1:1.1.1g-15.el8_3 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-10-06T05:04:11.339 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cephfs x86_64 2:17.2.6-1274.g68e53a4a.el8 ceph 190 k 2023-10-06T05:04:11.339 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rados x86_64 2:17.2.6-1274.g68e53a4a.el8 ceph 371 k 2023-10-06T05:04:11.339 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rbd x86_64 2:17.2.6-1274.g68e53a4a.el8 ceph 358 k 2023-10-06T05:04:11.339 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rgw x86_64 2:17.2.6-1274.g68e53a4a.el8 ceph 107 k 2023-10-06T05:04:11.339 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-fuse x86_64 2:17.2.6-1274.g68e53a4a.el8 ceph 91 k 2023-10-06T05:04:11.339 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-mirror x86_64 2:17.2.6-1274.g68e53a4a.el8 ceph 3.5 M 2023-10-06T05:04:11.339 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-nbd x86_64 2:17.2.6-1274.g68e53a4a.el8 ceph 173 k 2023-10-06T05:04:11.339 INFO:teuthology.orchestra.run.smithi098.stdout: xfsprogs-devel x86_64 5.0.0-8.el8 rhel-8-for-x86_64-baseos-rpms 70 k 2023-10-06T05:04:11.339 INFO:teuthology.orchestra.run.smithi098.stdout:Upgrading: 2023-10-06T05:04:11.339 INFO:teuthology.orchestra.run.smithi098.stdout: librados2 x86_64 2:17.2.6-1274.g68e53a4a.el8 ceph 3.4 M 2023-10-06T05:04:11.339 INFO:teuthology.orchestra.run.smithi098.stdout: librbd1 x86_64 2:17.2.6-1274.g68e53a4a.el8 ceph 3.5 M 2023-10-06T05:04:11.340 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2023-10-06T05:04:11.340 INFO:teuthology.orchestra.run.smithi098.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-10-06T05:04:11.340 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-common x86_64 2:17.2.6-1274.g68e53a4a.el8 ceph 23 M 2023-10-06T05:04:11.340 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-grafana-dashboards noarch 2:17.2.6-1274.g68e53a4a.el8 ceph-noarch 23 k 2023-10-06T05:04:11.340 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mds x86_64 2:17.2.6-1274.g68e53a4a.el8 ceph 2.3 M 2023-10-06T05:04:11.340 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-modules-core noarch 2:17.2.6-1274.g68e53a4a.el8 ceph-noarch 238 k 2023-10-06T05:04:11.340 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mon x86_64 2:17.2.6-1274.g68e53a4a.el8 ceph 3.9 M 2023-10-06T05:04:11.340 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-osd x86_64 2:17.2.6-1274.g68e53a4a.el8 ceph 17 M 2023-10-06T05:04:11.340 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-prometheus-alerts noarch 2:17.2.6-1274.g68e53a4a.el8 ceph-noarch 14 k 2023-10-06T05:04:11.340 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-selinux x86_64 2:17.2.6-1274.g68e53a4a.el8 ceph 24 k 2023-10-06T05:04:11.340 INFO:teuthology.orchestra.run.smithi098.stdout: fuse x86_64 2.9.7-12.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-10-06T05:04:11.341 INFO:teuthology.orchestra.run.smithi098.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-10-06T05:04:11.341 INFO:teuthology.orchestra.run.smithi098.stdout: jq x86_64 1.5-12.el8 rhel-8-for-x86_64-appstream-rpms 161 k 2023-10-06T05:04:11.341 INFO:teuthology.orchestra.run.smithi098.stdout: keyutils-libs-devel x86_64 1.5.10-6.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-10-06T05:04:11.341 INFO:teuthology.orchestra.run.smithi098.stdout: krb5-devel x86_64 1.18.2-8.3.el8_4 rhel-8-for-x86_64-baseos-rpms 559 k 2023-10-06T05:04:11.341 INFO:teuthology.orchestra.run.smithi098.stdout: libcephsqlite x86_64 2:17.2.6-1274.g68e53a4a.el8 ceph 175 k 2023-10-06T05:04:11.341 INFO:teuthology.orchestra.run.smithi098.stdout: libcom_err-devel x86_64 1.45.6-1.el8 rhel-8-for-x86_64-baseos-rpms 38 k 2023-10-06T05:04:11.341 INFO:teuthology.orchestra.run.smithi098.stdout: libgfortran x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 639 k 2023-10-06T05:04:11.341 INFO:teuthology.orchestra.run.smithi098.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-10-06T05:04:11.341 INFO:teuthology.orchestra.run.smithi098.stdout: libquadmath x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 169 k 2023-10-06T05:04:11.341 INFO:teuthology.orchestra.run.smithi098.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-10-06T05:04:11.341 INFO:teuthology.orchestra.run.smithi098.stdout: libradosstriper1 x86_64 2:17.2.6-1274.g68e53a4a.el8 ceph 506 k 2023-10-06T05:04:11.342 INFO:teuthology.orchestra.run.smithi098.stdout: librdkafka x86_64 0.11.4-1.el8 rhel-8-for-x86_64-appstream-rpms 353 k 2023-10-06T05:04:11.342 INFO:teuthology.orchestra.run.smithi098.stdout: librgw2 x86_64 2:17.2.6-1274.g68e53a4a.el8 ceph 5.6 M 2023-10-06T05:04:11.342 INFO:teuthology.orchestra.run.smithi098.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-10-06T05:04:11.342 INFO:teuthology.orchestra.run.smithi098.stdout: libsepol-devel x86_64 2.9-2.el8 rhel-8-for-x86_64-baseos-rpms 86 k 2023-10-06T05:04:11.342 INFO:teuthology.orchestra.run.smithi098.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-10-06T05:04:11.342 INFO:teuthology.orchestra.run.smithi098.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-10-06T05:04:11.342 INFO:teuthology.orchestra.run.smithi098.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-10-06T05:04:11.342 INFO:teuthology.orchestra.run.smithi098.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-10-06T05:04:11.342 INFO:teuthology.orchestra.run.smithi098.stdout: openblas x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.6 M 2023-10-06T05:04:11.342 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-threads x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-10-06T05:04:11.343 INFO:teuthology.orchestra.run.smithi098.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-10-06T05:04:11.343 INFO:teuthology.orchestra.run.smithi098.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-10-06T05:04:11.343 INFO:teuthology.orchestra.run.smithi098.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-10-06T05:04:11.343 INFO:teuthology.orchestra.run.smithi098.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-10-06T05:04:11.343 INFO:teuthology.orchestra.run.smithi098.stdout: python3-babel noarch 2.5.1-5.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-10-06T05:04:11.343 INFO:teuthology.orchestra.run.smithi098.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-10-06T05:04:11.343 INFO:teuthology.orchestra.run.smithi098.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-10-06T05:04:11.343 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-10-06T05:04:11.343 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-argparse x86_64 2:17.2.6-1274.g68e53a4a.el8 ceph 45 k 2023-10-06T05:04:11.343 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-common x86_64 2:17.2.6-1274.g68e53a4a.el8 ceph 115 k 2023-10-06T05:04:11.343 INFO:teuthology.orchestra.run.smithi098.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-10-06T05:04:11.343 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-10-06T05:04:11.345 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-10-06T05:04:11.345 INFO:teuthology.orchestra.run.smithi098.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-10-06T05:04:11.345 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-10-06T05:04:11.345 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-10-06T05:04:11.345 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 rhel-8-for-x86_64-appstream-rpms 538 k 2023-10-06T05:04:11.345 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-10-06T05:04:11.345 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-10-06T05:04:11.345 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-10-06T05:04:11.346 INFO:teuthology.orchestra.run.smithi098.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-10-06T05:04:11.346 INFO:teuthology.orchestra.run.smithi098.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-10-06T05:04:11.346 INFO:teuthology.orchestra.run.smithi098.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-10-06T05:04:11.346 INFO:teuthology.orchestra.run.smithi098.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-10-06T05:04:11.346 INFO:teuthology.orchestra.run.smithi098.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-10-06T05:04:11.346 INFO:teuthology.orchestra.run.smithi098.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-10-06T05:04:11.346 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-10-06T05:04:11.346 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-10-06T05:04:11.346 INFO:teuthology.orchestra.run.smithi098.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-10-06T05:04:11.346 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-10-06T05:04:11.346 INFO:teuthology.orchestra.run.smithi098.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-10-06T05:04:11.346 INFO:teuthology.orchestra.run.smithi098.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-10-06T05:04:11.347 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-10-06T05:04:11.347 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-10-06T05:04:11.347 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-10-06T05:04:11.347 INFO:teuthology.orchestra.run.smithi098.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-10-06T05:04:11.347 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-10-06T05:04:11.347 INFO:teuthology.orchestra.run.smithi098.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-10-06T05:04:11.347 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-10-06T05:04:11.347 INFO:teuthology.orchestra.run.smithi098.stdout: python3-scipy x86_64 1.0.0-20.module+el8.1.0+3334+5cb623d7 rhel-8-for-x86_64-appstream-rpms 15 M 2023-10-06T05:04:11.347 INFO:teuthology.orchestra.run.smithi098.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-10-06T05:04:11.347 INFO:teuthology.orchestra.run.smithi098.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-10-06T05:04:11.347 INFO:teuthology.orchestra.run.smithi098.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-10-06T05:04:11.347 INFO:teuthology.orchestra.run.smithi098.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-10-06T05:04:11.348 INFO:teuthology.orchestra.run.smithi098.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-10-06T05:04:11.348 INFO:teuthology.orchestra.run.smithi098.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-10-06T05:04:11.348 INFO:teuthology.orchestra.run.smithi098.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-10-06T05:04:11.348 INFO:teuthology.orchestra.run.smithi098.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-10-06T05:04:11.348 INFO:teuthology.orchestra.run.smithi098.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-10-06T05:04:11.348 INFO:teuthology.orchestra.run.smithi098.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-10-06T05:04:11.348 INFO:teuthology.orchestra.run.smithi098.stdout: socat x86_64 1.7.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 302 k 2023-10-06T05:04:11.348 INFO:teuthology.orchestra.run.smithi098.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-10-06T05:04:11.348 INFO:teuthology.orchestra.run.smithi098.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-10-06T05:04:11.348 INFO:teuthology.orchestra.run.smithi098.stdout: zlib-devel x86_64 1.2.11-17.el8 rhel-8-for-x86_64-baseos-rpms 58 k 2023-10-06T05:04:11.348 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:04:11.349 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2023-10-06T05:04:11.349 INFO:teuthology.orchestra.run.smithi098.stdout:====================================================================================================================== 2023-10-06T05:04:11.349 INFO:teuthology.orchestra.run.smithi098.stdout:Install 118 Packages 2023-10-06T05:04:11.349 INFO:teuthology.orchestra.run.smithi098.stdout:Upgrade 2 Packages 2023-10-06T05:04:11.349 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:04:11.349 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 201 M 2023-10-06T05:04:11.349 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2023-10-06T05:04:11.413 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2023-10-06T05:04:11.425 INFO:teuthology.orchestra.run.smithi174.stdout:====================================================================================================================== 2023-10-06T05:04:11.425 INFO:teuthology.orchestra.run.smithi174.stdout: Package Arch Version Repository Size 2023-10-06T05:04:11.425 INFO:teuthology.orchestra.run.smithi174.stdout:====================================================================================================================== 2023-10-06T05:04:11.425 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2023-10-06T05:04:11.425 INFO:teuthology.orchestra.run.smithi174.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-10-06T05:04:11.425 INFO:teuthology.orchestra.run.smithi174.stdout: ceph x86_64 2:17.2.6-1274.g68e53a4a.el8 ceph 6.4 k 2023-10-06T05:04:11.425 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-base x86_64 2:17.2.6-1274.g68e53a4a.el8 ceph 5.2 M 2023-10-06T05:04:11.426 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-fuse x86_64 2:17.2.6-1274.g68e53a4a.el8 ceph 877 k 2023-10-06T05:04:11.426 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-immutable-object-cache x86_64 2:17.2.6-1274.g68e53a4a.el8 ceph 162 k 2023-10-06T05:04:11.426 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr x86_64 2:17.2.6-1274.g68e53a4a.el8 ceph 1.6 M 2023-10-06T05:04:11.426 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-cephadm noarch 2:17.2.6-1274.g68e53a4a.el8 ceph-noarch 121 k 2023-10-06T05:04:11.426 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-dashboard noarch 2:17.2.6-1274.g68e53a4a.el8 ceph-noarch 1.7 M 2023-10-06T05:04:11.426 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.6-1274.g68e53a4a.el8 ceph-noarch 7.4 M 2023-10-06T05:04:11.426 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-rook noarch 2:17.2.6-1274.g68e53a4a.el8 ceph-noarch 47 k 2023-10-06T05:04:11.426 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-radosgw x86_64 2:17.2.6-1274.g68e53a4a.el8 ceph 17 M 2023-10-06T05:04:11.426 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-test x86_64 2:17.2.6-1274.g68e53a4a.el8 ceph 52 M 2023-10-06T05:04:11.426 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-volume noarch 2:17.2.6-1274.g68e53a4a.el8 ceph-noarch 267 k 2023-10-06T05:04:11.426 INFO:teuthology.orchestra.run.smithi174.stdout: cephadm noarch 2:17.2.6-1274.g68e53a4a.el8 ceph-noarch 80 k 2023-10-06T05:04:11.426 INFO:teuthology.orchestra.run.smithi174.stdout: cephfs-mirror x86_64 2:17.2.6-1274.g68e53a4a.el8 ceph 255 k 2023-10-06T05:04:11.427 INFO:teuthology.orchestra.run.smithi174.stdout: cephfs-top noarch 2:17.2.6-1274.g68e53a4a.el8 ceph-noarch 18 k 2023-10-06T05:04:11.427 INFO:teuthology.orchestra.run.smithi174.stdout: elfutils-libelf-devel x86_64 0.182-3.el8 rhel-8-for-x86_64-baseos-rpms 59 k 2023-10-06T05:04:11.427 INFO:teuthology.orchestra.run.smithi174.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-10-06T05:04:11.427 INFO:teuthology.orchestra.run.smithi174.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-10-06T05:04:11.427 INFO:teuthology.orchestra.run.smithi174.stdout: libcephfs-devel x86_64 2:17.2.6-1274.g68e53a4a.el8 ceph 25 k 2023-10-06T05:04:11.427 INFO:teuthology.orchestra.run.smithi174.stdout: libcephfs2 x86_64 2:17.2.6-1274.g68e53a4a.el8 ceph 797 k 2023-10-06T05:04:11.427 INFO:teuthology.orchestra.run.smithi174.stdout: librados-devel x86_64 2:17.2.6-1274.g68e53a4a.el8 ceph 130 k 2023-10-06T05:04:11.427 INFO:teuthology.orchestra.run.smithi174.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-10-06T05:04:11.427 INFO:teuthology.orchestra.run.smithi174.stdout: libuuid-devel x86_64 2.32.1-27.el8 rhel-8-for-x86_64-baseos-rpms 97 k 2023-10-06T05:04:11.427 INFO:teuthology.orchestra.run.smithi174.stdout: openssl-devel x86_64 1:1.1.1g-15.el8_3 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-10-06T05:04:11.427 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cephfs x86_64 2:17.2.6-1274.g68e53a4a.el8 ceph 190 k 2023-10-06T05:04:11.427 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rados x86_64 2:17.2.6-1274.g68e53a4a.el8 ceph 371 k 2023-10-06T05:04:11.428 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rbd x86_64 2:17.2.6-1274.g68e53a4a.el8 ceph 358 k 2023-10-06T05:04:11.428 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rgw x86_64 2:17.2.6-1274.g68e53a4a.el8 ceph 107 k 2023-10-06T05:04:11.428 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-fuse x86_64 2:17.2.6-1274.g68e53a4a.el8 ceph 91 k 2023-10-06T05:04:11.428 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-mirror x86_64 2:17.2.6-1274.g68e53a4a.el8 ceph 3.5 M 2023-10-06T05:04:11.428 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-nbd x86_64 2:17.2.6-1274.g68e53a4a.el8 ceph 173 k 2023-10-06T05:04:11.428 INFO:teuthology.orchestra.run.smithi174.stdout: xfsprogs-devel x86_64 5.0.0-8.el8 rhel-8-for-x86_64-baseos-rpms 70 k 2023-10-06T05:04:11.428 INFO:teuthology.orchestra.run.smithi174.stdout:Upgrading: 2023-10-06T05:04:11.428 INFO:teuthology.orchestra.run.smithi174.stdout: librados2 x86_64 2:17.2.6-1274.g68e53a4a.el8 ceph 3.4 M 2023-10-06T05:04:11.428 INFO:teuthology.orchestra.run.smithi174.stdout: librbd1 x86_64 2:17.2.6-1274.g68e53a4a.el8 ceph 3.5 M 2023-10-06T05:04:11.428 INFO:teuthology.orchestra.run.smithi174.stdout:Installing dependencies: 2023-10-06T05:04:11.428 INFO:teuthology.orchestra.run.smithi174.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-10-06T05:04:11.429 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-common x86_64 2:17.2.6-1274.g68e53a4a.el8 ceph 23 M 2023-10-06T05:04:11.429 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-grafana-dashboards noarch 2:17.2.6-1274.g68e53a4a.el8 ceph-noarch 23 k 2023-10-06T05:04:11.429 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mds x86_64 2:17.2.6-1274.g68e53a4a.el8 ceph 2.3 M 2023-10-06T05:04:11.429 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-modules-core noarch 2:17.2.6-1274.g68e53a4a.el8 ceph-noarch 238 k 2023-10-06T05:04:11.429 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mon x86_64 2:17.2.6-1274.g68e53a4a.el8 ceph 3.9 M 2023-10-06T05:04:11.429 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-osd x86_64 2:17.2.6-1274.g68e53a4a.el8 ceph 17 M 2023-10-06T05:04:11.429 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-prometheus-alerts noarch 2:17.2.6-1274.g68e53a4a.el8 ceph-noarch 14 k 2023-10-06T05:04:11.429 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-selinux x86_64 2:17.2.6-1274.g68e53a4a.el8 ceph 24 k 2023-10-06T05:04:11.429 INFO:teuthology.orchestra.run.smithi174.stdout: fuse x86_64 2.9.7-12.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-10-06T05:04:11.429 INFO:teuthology.orchestra.run.smithi174.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-10-06T05:04:11.429 INFO:teuthology.orchestra.run.smithi174.stdout: jq x86_64 1.5-12.el8 rhel-8-for-x86_64-appstream-rpms 161 k 2023-10-06T05:04:11.430 INFO:teuthology.orchestra.run.smithi174.stdout: keyutils-libs-devel x86_64 1.5.10-6.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-10-06T05:04:11.430 INFO:teuthology.orchestra.run.smithi174.stdout: krb5-devel x86_64 1.18.2-8.3.el8_4 rhel-8-for-x86_64-baseos-rpms 559 k 2023-10-06T05:04:11.430 INFO:teuthology.orchestra.run.smithi174.stdout: libcephsqlite x86_64 2:17.2.6-1274.g68e53a4a.el8 ceph 175 k 2023-10-06T05:04:11.430 INFO:teuthology.orchestra.run.smithi174.stdout: libcom_err-devel x86_64 1.45.6-1.el8 rhel-8-for-x86_64-baseos-rpms 38 k 2023-10-06T05:04:11.430 INFO:teuthology.orchestra.run.smithi174.stdout: libgfortran x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 639 k 2023-10-06T05:04:11.430 INFO:teuthology.orchestra.run.smithi174.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-10-06T05:04:11.430 INFO:teuthology.orchestra.run.smithi174.stdout: libquadmath x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 169 k 2023-10-06T05:04:11.430 INFO:teuthology.orchestra.run.smithi174.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-10-06T05:04:11.430 INFO:teuthology.orchestra.run.smithi174.stdout: libradosstriper1 x86_64 2:17.2.6-1274.g68e53a4a.el8 ceph 506 k 2023-10-06T05:04:11.430 INFO:teuthology.orchestra.run.smithi174.stdout: librdkafka x86_64 0.11.4-1.el8 rhel-8-for-x86_64-appstream-rpms 353 k 2023-10-06T05:04:11.430 INFO:teuthology.orchestra.run.smithi174.stdout: librgw2 x86_64 2:17.2.6-1274.g68e53a4a.el8 ceph 5.6 M 2023-10-06T05:04:11.430 INFO:teuthology.orchestra.run.smithi174.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-10-06T05:04:11.431 INFO:teuthology.orchestra.run.smithi174.stdout: libsepol-devel x86_64 2.9-2.el8 rhel-8-for-x86_64-baseos-rpms 86 k 2023-10-06T05:04:11.431 INFO:teuthology.orchestra.run.smithi174.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-10-06T05:04:11.431 INFO:teuthology.orchestra.run.smithi174.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-10-06T05:04:11.431 INFO:teuthology.orchestra.run.smithi174.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-10-06T05:04:11.431 INFO:teuthology.orchestra.run.smithi174.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-10-06T05:04:11.431 INFO:teuthology.orchestra.run.smithi174.stdout: openblas x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.6 M 2023-10-06T05:04:11.431 INFO:teuthology.orchestra.run.smithi174.stdout: openblas-threads x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-10-06T05:04:11.431 INFO:teuthology.orchestra.run.smithi174.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-10-06T05:04:11.431 INFO:teuthology.orchestra.run.smithi174.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-10-06T05:04:11.431 INFO:teuthology.orchestra.run.smithi174.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-10-06T05:04:11.431 INFO:teuthology.orchestra.run.smithi174.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-10-06T05:04:11.432 INFO:teuthology.orchestra.run.smithi174.stdout: python3-babel noarch 2.5.1-5.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-10-06T05:04:11.432 INFO:teuthology.orchestra.run.smithi174.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-10-06T05:04:11.432 INFO:teuthology.orchestra.run.smithi174.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-10-06T05:04:11.432 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-10-06T05:04:11.432 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ceph-argparse x86_64 2:17.2.6-1274.g68e53a4a.el8 ceph 45 k 2023-10-06T05:04:11.432 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ceph-common x86_64 2:17.2.6-1274.g68e53a4a.el8 ceph 115 k 2023-10-06T05:04:11.432 INFO:teuthology.orchestra.run.smithi174.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-10-06T05:04:11.432 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-10-06T05:04:11.432 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-10-06T05:04:11.432 INFO:teuthology.orchestra.run.smithi174.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-10-06T05:04:11.432 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-10-06T05:04:11.432 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-10-06T05:04:11.433 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 rhel-8-for-x86_64-appstream-rpms 538 k 2023-10-06T05:04:11.433 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-10-06T05:04:11.433 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-10-06T05:04:11.433 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-10-06T05:04:11.433 INFO:teuthology.orchestra.run.smithi174.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-10-06T05:04:11.433 INFO:teuthology.orchestra.run.smithi174.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-10-06T05:04:11.433 INFO:teuthology.orchestra.run.smithi174.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-10-06T05:04:11.433 INFO:teuthology.orchestra.run.smithi174.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-10-06T05:04:11.433 INFO:teuthology.orchestra.run.smithi174.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-10-06T05:04:11.433 INFO:teuthology.orchestra.run.smithi174.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-10-06T05:04:11.433 INFO:teuthology.orchestra.run.smithi174.stdout: python3-numpy x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-10-06T05:04:11.433 INFO:teuthology.orchestra.run.smithi174.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-10-06T05:04:11.434 INFO:teuthology.orchestra.run.smithi174.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-10-06T05:04:11.434 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-10-06T05:04:11.434 INFO:teuthology.orchestra.run.smithi174.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-10-06T05:04:11.434 INFO:teuthology.orchestra.run.smithi174.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-10-06T05:04:11.434 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-10-06T05:04:11.434 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-10-06T05:04:11.434 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-10-06T05:04:11.434 INFO:teuthology.orchestra.run.smithi174.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-10-06T05:04:11.434 INFO:teuthology.orchestra.run.smithi174.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-10-06T05:04:11.434 INFO:teuthology.orchestra.run.smithi174.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-10-06T05:04:11.434 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-10-06T05:04:11.434 INFO:teuthology.orchestra.run.smithi174.stdout: python3-scipy x86_64 1.0.0-20.module+el8.1.0+3334+5cb623d7 rhel-8-for-x86_64-appstream-rpms 15 M 2023-10-06T05:04:11.435 INFO:teuthology.orchestra.run.smithi174.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-10-06T05:04:11.435 INFO:teuthology.orchestra.run.smithi174.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-10-06T05:04:11.435 INFO:teuthology.orchestra.run.smithi174.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-10-06T05:04:11.435 INFO:teuthology.orchestra.run.smithi174.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-10-06T05:04:11.435 INFO:teuthology.orchestra.run.smithi174.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-10-06T05:04:11.435 INFO:teuthology.orchestra.run.smithi174.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-10-06T05:04:11.435 INFO:teuthology.orchestra.run.smithi174.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-10-06T05:04:11.435 INFO:teuthology.orchestra.run.smithi174.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-10-06T05:04:11.435 INFO:teuthology.orchestra.run.smithi174.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-10-06T05:04:11.435 INFO:teuthology.orchestra.run.smithi174.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-10-06T05:04:11.435 INFO:teuthology.orchestra.run.smithi174.stdout: socat x86_64 1.7.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 302 k 2023-10-06T05:04:11.436 INFO:teuthology.orchestra.run.smithi174.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-10-06T05:04:11.436 INFO:teuthology.orchestra.run.smithi174.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-10-06T05:04:11.436 INFO:teuthology.orchestra.run.smithi174.stdout: zlib-devel x86_64 1.2.11-17.el8 rhel-8-for-x86_64-baseos-rpms 58 k 2023-10-06T05:04:11.436 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-10-06T05:04:11.436 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2023-10-06T05:04:11.436 INFO:teuthology.orchestra.run.smithi174.stdout:====================================================================================================================== 2023-10-06T05:04:11.436 INFO:teuthology.orchestra.run.smithi174.stdout:Install 118 Packages 2023-10-06T05:04:11.436 INFO:teuthology.orchestra.run.smithi174.stdout:Upgrade 2 Packages 2023-10-06T05:04:11.436 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-10-06T05:04:11.437 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 201 M 2023-10-06T05:04:11.437 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2023-10-06T05:04:11.568 INFO:teuthology.orchestra.run.smithi098.stdout:(1/120): ceph-17.2.6-1274.g68e53a4a.el8.x86_64. 35 kB/s | 6.4 kB 00:00 2023-10-06T05:04:11.802 INFO:teuthology.orchestra.run.smithi174.stdout:(1/120): ceph-17.2.6-1274.g68e53a4a.el8.x86_64. 30 kB/s | 6.4 kB 00:00 2023-10-06T05:04:11.850 INFO:teuthology.orchestra.run.smithi098.stdout:(2/120): ceph-fuse-17.2.6-1274.g68e53a4a.el8.x8 3.0 MB/s | 877 kB 00:00 2023-10-06T05:04:11.920 INFO:teuthology.orchestra.run.smithi098.stdout:(3/120): ceph-immutable-object-cache-17.2.6-127 2.3 MB/s | 162 kB 00:00 2023-10-06T05:04:12.225 INFO:teuthology.orchestra.run.smithi098.stdout:(4/120): ceph-mds-17.2.6-1274.g68e53a4a.el8.x86 7.6 MB/s | 2.3 MB 00:00 2023-10-06T05:04:12.441 INFO:teuthology.orchestra.run.smithi098.stdout:(5/120): ceph-mgr-17.2.6-1274.g68e53a4a.el8.x86 7.3 MB/s | 1.6 MB 00:00 2023-10-06T05:04:12.951 INFO:teuthology.orchestra.run.smithi174.stdout:(2/120): ceph-fuse-17.2.6-1274.g68e53a4a.el8.x8 764 kB/s | 877 kB 00:01 2023-10-06T05:04:13.199 INFO:teuthology.orchestra.run.smithi174.stdout:(3/120): ceph-immutable-object-cache-17.2.6-127 651 kB/s | 162 kB 00:00 2023-10-06T05:04:13.246 INFO:teuthology.orchestra.run.smithi098.stdout:(6/120): ceph-mon-17.2.6-1274.g68e53a4a.el8.x86 4.9 MB/s | 3.9 MB 00:00 2023-10-06T05:04:14.101 INFO:teuthology.orchestra.run.smithi098.stdout:(7/120): ceph-base-17.2.6-1274.g68e53a4a.el8.x8 1.9 MB/s | 5.2 MB 00:02 2023-10-06T05:04:14.959 INFO:teuthology.orchestra.run.smithi174.stdout:(4/120): ceph-mds-17.2.6-1274.g68e53a4a.el8.x86 1.3 MB/s | 2.3 MB 00:01 2023-10-06T05:04:15.751 INFO:teuthology.orchestra.run.smithi174.stdout:(5/120): ceph-base-17.2.6-1274.g68e53a4a.el8.x8 1.2 MB/s | 5.2 MB 00:04 2023-10-06T05:04:15.965 INFO:teuthology.orchestra.run.smithi098.stdout:(8/120): ceph-osd-17.2.6-1274.g68e53a4a.el8.x86 6.1 MB/s | 17 MB 00:02 2023-10-06T05:04:16.010 INFO:teuthology.orchestra.run.smithi098.stdout:(9/120): ceph-selinux-17.2.6-1274.g68e53a4a.el8 546 kB/s | 24 kB 00:00 2023-10-06T05:04:16.037 INFO:teuthology.orchestra.run.smithi174.stdout:(6/120): ceph-mgr-17.2.6-1274.g68e53a4a.el8.x86 1.5 MB/s | 1.6 MB 00:01 2023-10-06T05:04:18.091 INFO:teuthology.orchestra.run.smithi174.stdout:(7/120): ceph-mon-17.2.6-1274.g68e53a4a.el8.x86 1.7 MB/s | 3.9 MB 00:02 2023-10-06T05:04:19.596 INFO:teuthology.orchestra.run.smithi098.stdout:(10/120): ceph-radosgw-17.2.6-1274.g68e53a4a.el 3.1 MB/s | 17 MB 00:05 2023-10-06T05:04:20.311 INFO:teuthology.orchestra.run.smithi098.stdout:(11/120): cephfs-mirror-17.2.6-1274.g68e53a4a.e 357 kB/s | 255 kB 00:00 2023-10-06T05:04:20.362 INFO:teuthology.orchestra.run.smithi098.stdout:(12/120): libcephfs-devel-17.2.6-1274.g68e53a4a 488 kB/s | 25 kB 00:00 2023-10-06T05:04:20.606 INFO:teuthology.orchestra.run.smithi098.stdout:(13/120): libcephfs2-17.2.6-1274.g68e53a4a.el8. 3.2 MB/s | 797 kB 00:00 2023-10-06T05:04:20.668 INFO:teuthology.orchestra.run.smithi098.stdout:(14/120): libcephsqlite-17.2.6-1274.g68e53a4a.e 2.8 MB/s | 175 kB 00:00 2023-10-06T05:04:20.718 INFO:teuthology.orchestra.run.smithi098.stdout:(15/120): librados-devel-17.2.6-1274.g68e53a4a. 2.6 MB/s | 130 kB 00:00 2023-10-06T05:04:20.876 INFO:teuthology.orchestra.run.smithi098.stdout:(16/120): libradosstriper1-17.2.6-1274.g68e53a4 3.1 MB/s | 506 kB 00:00 2023-10-06T05:04:21.805 INFO:teuthology.orchestra.run.smithi174.stdout:(8/120): ceph-common-17.2.6-1274.g68e53a4a.el8. 2.3 MB/s | 23 MB 00:10 2023-10-06T05:04:21.867 INFO:teuthology.orchestra.run.smithi174.stdout:(9/120): ceph-selinux-17.2.6-1274.g68e53a4a.el8 391 kB/s | 24 kB 00:00 2023-10-06T05:04:22.522 INFO:teuthology.orchestra.run.smithi098.stdout:(17/120): ceph-common-17.2.6-1274.g68e53a4a.el8 2.1 MB/s | 23 MB 00:11 2023-10-06T05:04:22.590 INFO:teuthology.orchestra.run.smithi098.stdout:(18/120): python3-ceph-argparse-17.2.6-1274.g68 658 kB/s | 45 kB 00:00 2023-10-06T05:04:22.620 INFO:teuthology.orchestra.run.smithi098.stdout:(19/120): librgw2-17.2.6-1274.g68e53a4a.el8.x86 3.2 MB/s | 5.6 MB 00:01 2023-10-06T05:04:22.662 INFO:teuthology.orchestra.run.smithi098.stdout:(20/120): python3-cephfs-17.2.6-1274.g68e53a4a. 4.5 MB/s | 190 kB 00:00 2023-10-06T05:04:22.686 INFO:teuthology.orchestra.run.smithi098.stdout:(21/120): python3-ceph-common-17.2.6-1274.g68e5 1.2 MB/s | 115 kB 00:00 2023-10-06T05:04:22.742 INFO:teuthology.orchestra.run.smithi098.stdout:(22/120): python3-rados-17.2.6-1274.g68e53a4a.e 4.5 MB/s | 371 kB 00:00 2023-10-06T05:04:22.780 INFO:teuthology.orchestra.run.smithi098.stdout:(23/120): python3-rgw-17.2.6-1274.g68e53a4a.el8 2.8 MB/s | 107 kB 00:00 2023-10-06T05:04:22.798 INFO:teuthology.orchestra.run.smithi098.stdout:(24/120): python3-rbd-17.2.6-1274.g68e53a4a.el8 3.1 MB/s | 358 kB 00:00 2023-10-06T05:04:22.816 INFO:teuthology.orchestra.run.smithi098.stdout:(25/120): rbd-fuse-17.2.6-1274.g68e53a4a.el8.x8 2.5 MB/s | 91 kB 00:00 2023-10-06T05:04:22.859 INFO:teuthology.orchestra.run.smithi098.stdout:(26/120): rbd-nbd-17.2.6-1274.g68e53a4a.el8.x86 3.9 MB/s | 173 kB 00:00 2023-10-06T05:04:22.900 INFO:teuthology.orchestra.run.smithi098.stdout:(27/120): ceph-grafana-dashboards-17.2.6-1274.g 574 kB/s | 23 kB 00:00 2023-10-06T05:04:22.938 INFO:teuthology.orchestra.run.smithi098.stdout:(28/120): ceph-mgr-cephadm-17.2.6-1274.g68e53a4 3.1 MB/s | 121 kB 00:00 2023-10-06T05:04:23.307 INFO:teuthology.orchestra.run.smithi098.stdout:(29/120): ceph-mgr-dashboard-17.2.6-1274.g68e53 4.6 MB/s | 1.7 MB 00:00 2023-10-06T05:04:23.658 INFO:teuthology.orchestra.run.smithi098.stdout:(30/120): rbd-mirror-17.2.6-1274.g68e53a4a.el8. 4.0 MB/s | 3.5 MB 00:00 2023-10-06T05:04:23.724 INFO:teuthology.orchestra.run.smithi098.stdout:(31/120): ceph-mgr-modules-core-17.2.6-1274.g68 3.5 MB/s | 238 kB 00:00 2023-10-06T05:04:23.780 INFO:teuthology.orchestra.run.smithi098.stdout:(32/120): ceph-mgr-rook-17.2.6-1274.g68e53a4a.e 844 kB/s | 47 kB 00:00 2023-10-06T05:04:23.827 INFO:teuthology.orchestra.run.smithi098.stdout:(33/120): ceph-prometheus-alerts-17.2.6-1274.g6 314 kB/s | 14 kB 00:00 2023-10-06T05:04:23.863 INFO:teuthology.orchestra.run.smithi174.stdout:(10/120): ceph-osd-17.2.6-1274.g68e53a4a.el8.x8 2.1 MB/s | 17 MB 00:07 2023-10-06T05:04:23.875 INFO:teuthology.orchestra.run.smithi098.stdout:(34/120): ceph-volume-17.2.6-1274.g68e53a4a.el8 5.4 MB/s | 267 kB 00:00 2023-10-06T05:04:23.943 INFO:teuthology.orchestra.run.smithi098.stdout:(35/120): cephadm-17.2.6-1274.g68e53a4a.el8.noa 1.2 MB/s | 80 kB 00:00 2023-10-06T05:04:23.974 INFO:teuthology.orchestra.run.smithi174.stdout:(11/120): cephfs-mirror-17.2.6-1274.g68e53a4a.e 2.3 MB/s | 255 kB 00:00 2023-10-06T05:04:24.010 INFO:teuthology.orchestra.run.smithi174.stdout:(12/120): libcephfs-devel-17.2.6-1274.g68e53a4a 687 kB/s | 25 kB 00:00 2023-10-06T05:04:24.162 INFO:teuthology.orchestra.run.smithi174.stdout:(13/120): libcephfs2-17.2.6-1274.g68e53a4a.el8. 5.1 MB/s | 797 kB 00:00 2023-10-06T05:04:24.282 INFO:teuthology.orchestra.run.smithi174.stdout:(14/120): ceph-radosgw-17.2.6-1274.g68e53a4a.el 2.7 MB/s | 17 MB 00:06 2023-10-06T05:04:24.284 INFO:teuthology.orchestra.run.smithi174.stdout:(15/120): libcephsqlite-17.2.6-1274.g68e53a4a.e 1.4 MB/s | 175 kB 00:00 2023-10-06T05:04:24.322 INFO:teuthology.orchestra.run.smithi174.stdout:(16/120): librados-devel-17.2.6-1274.g68e53a4a. 3.2 MB/s | 130 kB 00:00 2023-10-06T05:04:24.362 INFO:teuthology.orchestra.run.smithi098.stdout:(36/120): ceph-test-17.2.6-1274.g68e53a4a.el8.x 6.2 MB/s | 52 MB 00:08 2023-10-06T05:04:24.363 INFO:teuthology.orchestra.run.smithi098.stdout:(37/120): cephfs-top-17.2.6-1274.g68e53a4a.el8. 42 kB/s | 18 kB 00:00 2023-10-06T05:04:24.373 INFO:teuthology.orchestra.run.smithi174.stdout:(17/120): libradosstriper1-17.2.6-1274.g68e53a4 5.5 MB/s | 506 kB 00:00 2023-10-06T05:04:24.410 INFO:teuthology.orchestra.run.smithi174.stdout:(18/120): python3-ceph-argparse-17.2.6-1274.g68 1.2 MB/s | 45 kB 00:00 2023-10-06T05:04:24.447 INFO:teuthology.orchestra.run.smithi098.stdout:(38/120): liboath-2.6.2-3.el8.x86_64.rpm 705 kB/s | 59 kB 00:00 2023-10-06T05:04:24.457 INFO:teuthology.orchestra.run.smithi174.stdout:(19/120): python3-ceph-common-17.2.6-1274.g68e5 2.4 MB/s | 115 kB 00:00 2023-10-06T05:04:24.487 INFO:teuthology.orchestra.run.smithi098.stdout:(39/120): libunwind-1.3.1-3.el8.x86_64.rpm 1.8 MB/s | 75 kB 00:00 2023-10-06T05:04:24.494 INFO:teuthology.orchestra.run.smithi098.stdout:(40/120): gperftools-libs-2.7-9.el8.x86_64.rpm 2.3 MB/s | 306 kB 00:00 2023-10-06T05:04:24.506 INFO:teuthology.orchestra.run.smithi174.stdout:(20/120): python3-cephfs-17.2.6-1274.g68e53a4a. 3.8 MB/s | 190 kB 00:00 2023-10-06T05:04:24.522 INFO:teuthology.orchestra.run.smithi098.stdout:(41/120): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 1.6 MB/s | 44 kB 00:00 2023-10-06T05:04:24.549 INFO:teuthology.orchestra.run.smithi098.stdout:(42/120): python3-beautifulsoup4-4.6.3-2.el8.1. 6.7 MB/s | 185 kB 00:00 2023-10-06T05:04:24.565 INFO:teuthology.orchestra.run.smithi098.stdout:(43/120): python3-asyncssh-2.7.0-2.el8.noarch.r 6.0 MB/s | 476 kB 00:00 2023-10-06T05:04:24.569 INFO:teuthology.orchestra.run.smithi098.stdout:(44/120): python3-cachetools-3.1.1-4.el8.noarch 1.6 MB/s | 33 kB 00:00 2023-10-06T05:04:24.584 INFO:teuthology.orchestra.run.smithi098.stdout:(45/120): python3-certifi-2018.10.15-7.el8.noar 843 kB/s | 16 kB 00:00 2023-10-06T05:04:24.595 INFO:teuthology.orchestra.run.smithi174.stdout:(21/120): python3-rados-17.2.6-1274.g68e53a4a.e 4.1 MB/s | 371 kB 00:00 2023-10-06T05:04:24.596 INFO:teuthology.orchestra.run.smithi098.stdout:(46/120): python3-cheroot-8.5.2-1.el8.noarch.rp 6.4 MB/s | 173 kB 00:00 2023-10-06T05:04:24.613 INFO:teuthology.orchestra.run.smithi098.stdout:(47/120): python3-cherrypy-18.4.0-1.el8.noarch. 13 MB/s | 384 kB 00:00 2023-10-06T05:04:24.618 INFO:teuthology.orchestra.run.smithi098.stdout:(48/120): python3-google-auth-1.1.1-10.el8.noar 3.7 MB/s | 81 kB 00:00 2023-10-06T05:04:24.632 INFO:teuthology.orchestra.run.smithi098.stdout:(49/120): python3-jaraco-6.2-6.el8.noarch.rpm 580 kB/s | 11 kB 00:00 2023-10-06T05:04:24.638 INFO:teuthology.orchestra.run.smithi098.stdout:(50/120): python3-jaraco-functools-2.0-4.el8.no 905 kB/s | 18 kB 00:00 2023-10-06T05:04:24.675 INFO:teuthology.orchestra.run.smithi098.stdout:(51/120): ceph-mgr-diskprediction-local-17.2.6- 5.4 MB/s | 7.4 MB 00:01 2023-10-06T05:04:24.676 INFO:teuthology.orchestra.run.smithi098.stdout:(52/120): python3-logutils-0.3.5-11.el8.noarch. 1.2 MB/s | 49 kB 00:00 2023-10-06T05:04:24.677 INFO:teuthology.orchestra.run.smithi174.stdout:(22/120): python3-rbd-17.2.6-1274.g68e53a4a.el8 4.2 MB/s | 358 kB 00:00 2023-10-06T05:04:24.699 INFO:teuthology.orchestra.run.smithi098.stdout:(53/120): python3-kubernetes-11.0.0-6.el8.noarc 18 MB/s | 1.2 MB 00:00 2023-10-06T05:04:24.699 INFO:teuthology.orchestra.run.smithi098.stdout:(54/120): python3-natsort-7.1.1-2.el8.noarch.rp 2.6 MB/s | 60 kB 00:00 2023-10-06T05:04:24.719 INFO:teuthology.orchestra.run.smithi098.stdout:(55/120): python3-portend-2.6-1.el8.noarch.rpm 805 kB/s | 16 kB 00:00 2023-10-06T05:04:24.720 INFO:teuthology.orchestra.run.smithi098.stdout:(56/120): python3-pecan-1.3.2-9.el8.noarch.rpm 13 MB/s | 283 kB 00:00 2023-10-06T05:04:24.739 INFO:teuthology.orchestra.run.smithi098.stdout:(57/120): python3-repoze-lru-0.7-6.el8.noarch.r 1.7 MB/s | 34 kB 00:00 2023-10-06T05:04:24.742 INFO:teuthology.orchestra.run.smithi098.stdout:(58/120): python3-routes-2.4.1-12.el8.noarch.rp 9.2 MB/s | 196 kB 00:00 2023-10-06T05:04:24.759 INFO:teuthology.orchestra.run.smithi098.stdout:(59/120): python3-more-itertools-7.2.0-3.el8.no 708 kB/s | 59 kB 00:00 2023-10-06T05:04:24.760 INFO:teuthology.orchestra.run.smithi098.stdout:(60/120): python3-rsa-4.9-2.el8.noarch.rpm 3.0 MB/s | 64 kB 00:00 2023-10-06T05:04:24.762 INFO:teuthology.orchestra.run.smithi098.stdout:(61/120): python3-simplegeneric-0.8.1-17.el8.no 957 kB/s | 19 kB 00:00 2023-10-06T05:04:24.769 INFO:teuthology.orchestra.run.smithi174.stdout:(23/120): python3-rgw-17.2.6-1274.g68e53a4a.el8 1.1 MB/s | 107 kB 00:00 2023-10-06T05:04:24.780 INFO:teuthology.orchestra.run.smithi098.stdout:(62/120): python3-tempora-1.14.1-5.el8.noarch.r 1.5 MB/s | 29 kB 00:00 2023-10-06T05:04:24.781 INFO:teuthology.orchestra.run.smithi098.stdout:(63/120): python3-trustme-0.6.0-4.el8.noarch.rp 1.4 MB/s | 27 kB 00:00 2023-10-06T05:04:24.783 INFO:teuthology.orchestra.run.smithi098.stdout:(64/120): python3-singledispatch-3.4.0.3-18.el8 1.0 MB/s | 24 kB 00:00 2023-10-06T05:04:24.804 INFO:teuthology.orchestra.run.smithi098.stdout:(65/120): python3-webob-1.8.5-1.el8.1.noarch.rp 11 MB/s | 251 kB 00:00 2023-10-06T05:04:24.806 INFO:teuthology.orchestra.run.smithi174.stdout:(24/120): rbd-fuse-17.2.6-1274.g68e53a4a.el8.x8 2.4 MB/s | 91 kB 00:00 2023-10-06T05:04:24.811 INFO:teuthology.orchestra.run.smithi098.stdout:(66/120): python3-waitress-1.4.3-1.el8.noarch.r 8.3 MB/s | 255 kB 00:00 2023-10-06T05:04:24.816 INFO:teuthology.orchestra.run.smithi098.stdout:(67/120): python3-websocket-client-0.56.0-5.el8 1.8 MB/s | 61 kB 00:00 2023-10-06T05:04:24.824 INFO:teuthology.orchestra.run.smithi098.stdout:(68/120): python3-webtest-2.0.33-1.el8.noarch.r 4.2 MB/s | 85 kB 00:00 2023-10-06T05:04:24.829 INFO:teuthology.orchestra.run.smithi098.stdout:(69/120): xmlstarlet-1.6.1-20.el8.x86_64.rpm 15 MB/s | 70 kB 00:00 2023-10-06T05:04:24.831 INFO:teuthology.orchestra.run.smithi098.stdout:(70/120): python3-zc-lockfile-2.0-2.el8.noarch. 1.1 MB/s | 23 kB 00:00 2023-10-06T05:04:24.871 INFO:teuthology.orchestra.run.smithi098.stdout:(71/120): python3-prettytable-0.7.2-14.el8.noar 1.1 MB/s | 44 kB 00:00 2023-10-06T05:04:24.886 INFO:teuthology.orchestra.run.smithi098.stdout:(72/120): python3-jsonpatch-1.21-2.el8.noarch.r 1.8 MB/s | 27 kB 00:00 2023-10-06T05:04:24.902 INFO:teuthology.orchestra.run.smithi098.stdout:(73/120): jq-1.5-12.el8.x86_64.rpm 10 MB/s | 161 kB 00:00 2023-10-06T05:04:24.920 INFO:teuthology.orchestra.run.smithi098.stdout:(74/120): flex-2.6.1-9.el8.x86_64.rpm 17 MB/s | 320 kB 00:00 2023-10-06T05:04:24.936 INFO:teuthology.orchestra.run.smithi098.stdout:(75/120): thrift-0.13.0-2.el8.x86_64.rpm 15 MB/s | 1.7 MB 00:00 2023-10-06T05:04:24.943 INFO:teuthology.orchestra.run.smithi098.stdout:(76/120): python3-werkzeug-0.12.2-4.el8.noarch. 20 MB/s | 457 kB 00:00 2023-10-06T05:04:24.958 INFO:teuthology.orchestra.run.smithi098.stdout:(77/120): python3-markupsafe-0.23-19.el8.x86_64 2.5 MB/s | 39 kB 00:00 2023-10-06T05:04:24.984 INFO:teuthology.orchestra.run.smithi098.stdout:(78/120): bison-3.0.4-10.el8.x86_64.rpm 27 MB/s | 688 kB 00:00 2023-10-06T05:04:25.005 INFO:teuthology.orchestra.run.smithi098.stdout:(79/120): lttng-ust-2.8.1-11.el8.x86_64.rpm 12 MB/s | 259 kB 00:00 2023-10-06T05:04:25.026 INFO:teuthology.orchestra.run.smithi098.stdout:(80/120): python3-numpy-f2py-1.14.3-9.el8.x86_6 11 MB/s | 225 kB 00:00 2023-10-06T05:04:25.041 INFO:teuthology.orchestra.run.smithi098.stdout:(81/120): python3-babel-2.5.1-5.el8.noarch.rpm 45 MB/s | 4.8 MB 00:00 2023-10-06T05:04:25.043 INFO:teuthology.orchestra.run.smithi098.stdout:(82/120): python3-pyasn1-0.3.7-6.el8.noarch.rpm 7.2 MB/s | 126 kB 00:00 2023-10-06T05:04:25.072 INFO:teuthology.orchestra.run.smithi174.stdout:(25/120): librgw2-17.2.6-1274.g68e53a4a.el8.x86 7.4 MB/s | 5.6 MB 00:00 2023-10-06T05:04:25.119 INFO:teuthology.orchestra.run.smithi174.stdout:(26/120): rbd-nbd-17.2.6-1274.g68e53a4a.el8.x86 3.7 MB/s | 173 kB 00:00 2023-10-06T05:04:25.155 INFO:teuthology.orchestra.run.smithi174.stdout:(27/120): ceph-grafana-dashboards-17.2.6-1274.g 640 kB/s | 23 kB 00:00 2023-10-06T05:04:25.168 INFO:teuthology.orchestra.run.smithi098.stdout:(83/120): python3-scipy-1.0.0-20.module+el8.1.0 43 MB/s | 15 MB 00:00 2023-10-06T05:04:25.171 INFO:teuthology.orchestra.run.smithi098.stdout:(84/120): oniguruma-6.8.2-2.el8.x86_64.rpm 1.4 MB/s | 187 kB 00:00 2023-10-06T05:04:25.192 INFO:teuthology.orchestra.run.smithi174.stdout:(28/120): ceph-mgr-cephadm-17.2.6-1274.g68e53a4 3.2 MB/s | 121 kB 00:00 2023-10-06T05:04:25.193 INFO:teuthology.orchestra.run.smithi098.stdout:(85/120): python3-jinja2-2.10.1-2.el8_0.noarch. 22 MB/s | 538 kB 00:00 2023-10-06T05:04:25.194 INFO:teuthology.orchestra.run.smithi098.stdout:(86/120): socat-1.7.3.3-2.el8.x86_64.rpm 13 MB/s | 302 kB 00:00 2023-10-06T05:04:25.215 INFO:teuthology.orchestra.run.smithi098.stdout:(87/120): python3-numpy-1.14.3-9.el8.x86_64.rpm 21 MB/s | 3.7 MB 00:00 2023-10-06T05:04:25.217 INFO:teuthology.orchestra.run.smithi098.stdout:(88/120): python3-pyasn1-modules-0.3.7-6.el8.no 4.8 MB/s | 110 kB 00:00 2023-10-06T05:04:25.218 INFO:teuthology.orchestra.run.smithi098.stdout:(89/120): python3-mako-1.0.6-13.el8.noarch.rpm 6.0 MB/s | 157 kB 00:00 2023-10-06T05:04:25.234 INFO:teuthology.orchestra.run.smithi098.stdout:(90/120): python3-jsonpointer-1.10-11.el8.noarc 1.1 MB/s | 20 kB 00:00 2023-10-06T05:04:25.236 INFO:teuthology.orchestra.run.smithi098.stdout:(91/120): librdkafka-0.11.4-1.el8.x86_64.rpm 16 MB/s | 353 kB 00:00 2023-10-06T05:04:25.243 INFO:teuthology.orchestra.run.smithi098.stdout:(92/120): libtool-2.4.6-25.el8.x86_64.rpm 28 MB/s | 709 kB 00:00 2023-10-06T05:04:25.252 INFO:teuthology.orchestra.run.smithi098.stdout:(93/120): python3-pytz-2017.2-9.el8.noarch.rpm 3.0 MB/s | 54 kB 00:00 2023-10-06T05:04:25.281 INFO:teuthology.orchestra.run.smithi098.stdout:(94/120): automake-1.16.1-7.el8.noarch.rpm 25 MB/s | 713 kB 00:00 2023-10-06T05:04:25.299 INFO:teuthology.orchestra.run.smithi174.stdout:(29/120): rbd-mirror-17.2.6-1274.g68e53a4a.el8. 7.0 MB/s | 3.5 MB 00:00 2023-10-06T05:04:25.323 INFO:teuthology.orchestra.run.smithi098.stdout:(95/120): openblas-threads-0.3.12-1.el8.x86_64. 55 MB/s | 4.7 MB 00:00 2023-10-06T05:04:25.323 INFO:teuthology.orchestra.run.smithi098.stdout:(96/120): libaio-devel-0.3.112-1.el8.x86_64.rpm 440 kB/s | 19 kB 00:00 2023-10-06T05:04:25.343 INFO:teuthology.orchestra.run.smithi098.stdout:(97/120): openblas-0.3.12-1.el8.x86_64.rpm 46 MB/s | 4.6 MB 00:00 2023-10-06T05:04:25.351 INFO:teuthology.orchestra.run.smithi098.stdout:(98/120): python3-oauthlib-2.1.0-1.el8.noarch.r 5.4 MB/s | 155 kB 00:00 2023-10-06T05:04:25.358 INFO:teuthology.orchestra.run.smithi098.stdout:(99/120): python3-jwt-1.6.1-2.el8.noarch.rpm 1.2 MB/s | 43 kB 00:00 2023-10-06T05:04:25.376 INFO:teuthology.orchestra.run.smithi098.stdout:(100/120): pcre2-utf32-10.32-2.el8.x86_64.rpm 6.4 MB/s | 220 kB 00:00 2023-10-06T05:04:25.388 INFO:teuthology.orchestra.run.smithi098.stdout:(101/120): keyutils-libs-devel-1.5.10-6.el8.x86 1.3 MB/s | 48 kB 00:00 2023-10-06T05:04:25.391 INFO:teuthology.orchestra.run.smithi098.stdout:(102/120): pcre2-utf16-10.32-2.el8.x86_64.rpm 6.6 MB/s | 229 kB 00:00 2023-10-06T05:04:25.407 INFO:teuthology.orchestra.run.smithi174.stdout:(30/120): ceph-mgr-dashboard-17.2.6-1274.g68e53 7.9 MB/s | 1.7 MB 00:00 2023-10-06T05:04:25.419 INFO:teuthology.orchestra.run.smithi098.stdout:(103/120): libverto-devel-0.3.0-5.el8.x86_64.rp 640 kB/s | 18 kB 00:00 2023-10-06T05:04:25.429 INFO:teuthology.orchestra.run.smithi098.stdout:(104/120): python3-requests-oauthlib-1.0.0-1.el 812 kB/s | 43 kB 00:00 2023-10-06T05:04:25.447 INFO:teuthology.orchestra.run.smithi174.stdout:(31/120): ceph-mgr-modules-core-17.2.6-1274.g68 5.9 MB/s | 238 kB 00:00 2023-10-06T05:04:25.465 INFO:teuthology.orchestra.run.smithi098.stdout:(105/120): openssl-devel-1.1.1g-15.el8_3.x86_64 30 MB/s | 2.3 MB 00:00 2023-10-06T05:04:25.467 INFO:teuthology.orchestra.run.smithi098.stdout:(106/120): fuse-2.9.7-12.el8.x86_64.rpm 2.1 MB/s | 83 kB 00:00 2023-10-06T05:04:25.473 INFO:teuthology.orchestra.run.smithi098.stdout:(107/120): pcre2-devel-10.32-2.el8.x86_64.rpm 11 MB/s | 605 kB 00:00 2023-10-06T05:04:25.483 INFO:teuthology.orchestra.run.smithi174.stdout:(32/120): ceph-mgr-rook-17.2.6-1274.g68e53a4a.e 1.3 MB/s | 47 kB 00:00 2023-10-06T05:04:25.495 INFO:teuthology.orchestra.run.smithi098.stdout:(108/120): libquadmath-8.4.1-1.el8.x86_64.rpm 5.9 MB/s | 169 kB 00:00 2023-10-06T05:04:25.514 INFO:teuthology.orchestra.run.smithi098.stdout:(109/120): elfutils-libelf-devel-0.182-3.el8.x8 1.2 MB/s | 59 kB 00:00 2023-10-06T05:04:25.519 INFO:teuthology.orchestra.run.smithi174.stdout:(33/120): ceph-prometheus-alerts-17.2.6-1274.g6 400 kB/s | 14 kB 00:00 2023-10-06T05:04:25.525 INFO:teuthology.orchestra.run.smithi098.stdout:(110/120): libcom_err-devel-1.45.6-1.el8.x86_64 732 kB/s | 38 kB 00:00 2023-10-06T05:04:25.525 INFO:teuthology.orchestra.run.smithi098.stdout:(111/120): libsepol-devel-2.9-2.el8.x86_64.rpm 2.8 MB/s | 86 kB 00:00 2023-10-06T05:04:25.546 INFO:teuthology.orchestra.run.smithi098.stdout:(112/120): zlib-devel-1.2.11-17.el8.x86_64.rpm 1.7 MB/s | 58 kB 00:00 2023-10-06T05:04:25.558 INFO:teuthology.orchestra.run.smithi174.stdout:(34/120): ceph-volume-17.2.6-1274.g68e53a4a.el8 6.7 MB/s | 267 kB 00:00 2023-10-06T05:04:25.561 INFO:teuthology.orchestra.run.smithi098.stdout:(113/120): krb5-devel-1.18.2-8.3.el8_4.x86_64.r 15 MB/s | 559 kB 00:00 2023-10-06T05:04:25.562 INFO:teuthology.orchestra.run.smithi098.stdout:(114/120): librabbitmq-0.9.0-3.el8.x86_64.rpm 1.2 MB/s | 47 kB 00:00 2023-10-06T05:04:25.592 INFO:teuthology.orchestra.run.smithi098.stdout:(115/120): libgfortran-8.4.1-1.el8.x86_64.rpm 14 MB/s | 639 kB 00:00 2023-10-06T05:04:25.593 INFO:teuthology.orchestra.run.smithi098.stdout:(116/120): xfsprogs-devel-5.0.0-8.el8.x86_64.rp 2.2 MB/s | 70 kB 00:00 2023-10-06T05:04:25.595 INFO:teuthology.orchestra.run.smithi174.stdout:(35/120): cephadm-17.2.6-1274.g68e53a4a.el8.noa 2.1 MB/s | 80 kB 00:00 2023-10-06T05:04:25.596 INFO:teuthology.orchestra.run.smithi098.stdout:(117/120): libselinux-devel-2.9-5.el8.x86_64.rp 5.8 MB/s | 200 kB 00:00 2023-10-06T05:04:25.622 INFO:teuthology.orchestra.run.smithi098.stdout:(118/120): libuuid-devel-2.32.1-27.el8.x86_64.r 3.2 MB/s | 97 kB 00:00 2023-10-06T05:04:25.632 INFO:teuthology.orchestra.run.smithi174.stdout:(36/120): cephfs-top-17.2.6-1274.g68e53a4a.el8. 496 kB/s | 18 kB 00:00 2023-10-06T05:04:25.863 INFO:teuthology.orchestra.run.smithi174.stdout:(37/120): gperftools-libs-2.7-9.el8.x86_64.rpm 1.3 MB/s | 306 kB 00:00 2023-10-06T05:04:25.900 INFO:teuthology.orchestra.run.smithi174.stdout:(38/120): liboath-2.6.2-3.el8.x86_64.rpm 1.5 MB/s | 59 kB 00:00 2023-10-06T05:04:25.940 INFO:teuthology.orchestra.run.smithi174.stdout:(39/120): libunwind-1.3.1-3.el8.x86_64.rpm 1.9 MB/s | 75 kB 00:00 2023-10-06T05:04:26.012 INFO:teuthology.orchestra.run.smithi174.stdout:(40/120): python3-asyncssh-2.7.0-2.el8.noarch.r 6.4 MB/s | 476 kB 00:00 2023-10-06T05:04:26.047 INFO:teuthology.orchestra.run.smithi174.stdout:(41/120): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 1.2 MB/s | 44 kB 00:00 2023-10-06T05:04:26.075 INFO:teuthology.orchestra.run.smithi098.stdout:(119/120): librados2-17.2.6-1274.g68e53a4a.el8. 7.1 MB/s | 3.4 MB 00:00 2023-10-06T05:04:26.085 INFO:teuthology.orchestra.run.smithi174.stdout:(42/120): python3-beautifulsoup4-4.6.3-2.el8.1. 4.7 MB/s | 185 kB 00:00 2023-10-06T05:04:26.251 INFO:teuthology.orchestra.run.smithi098.stdout:(120/120): librbd1-17.2.6-1274.g68e53a4a.el8.x8 5.4 MB/s | 3.5 MB 00:00 2023-10-06T05:04:26.254 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2023-10-06T05:04:26.254 INFO:teuthology.orchestra.run.smithi098.stdout:Total 14 MB/s | 201 MB 00:14 2023-10-06T05:04:26.301 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2023-10-06T05:04:26.326 INFO:teuthology.orchestra.run.smithi174.stdout:(43/120): ceph-mgr-diskprediction-local-17.2.6- 7.2 MB/s | 7.4 MB 00:01 2023-10-06T05:04:26.331 INFO:teuthology.orchestra.run.smithi174.stdout:(44/120): python3-cachetools-3.1.1-4.el8.noarch 134 kB/s | 33 kB 00:00 2023-10-06T05:04:26.383 INFO:teuthology.orchestra.run.smithi174.stdout:(45/120): python3-cheroot-8.5.2-1.el8.noarch.rp 3.3 MB/s | 173 kB 00:00 2023-10-06T05:04:26.413 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2023-10-06T05:04:26.413 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2023-10-06T05:04:26.431 INFO:teuthology.orchestra.run.smithi174.stdout:(46/120): python3-certifi-2018.10.15-7.el8.noar 156 kB/s | 16 kB 00:00 2023-10-06T05:04:26.447 INFO:teuthology.orchestra.run.smithi174.stdout:(47/120): python3-cherrypy-18.4.0-1.el8.noarch. 5.9 MB/s | 384 kB 00:00 2023-10-06T05:04:26.481 INFO:teuthology.orchestra.run.smithi174.stdout:(48/120): python3-jaraco-6.2-6.el8.noarch.rpm 324 kB/s | 11 kB 00:00 2023-10-06T05:04:26.516 INFO:teuthology.orchestra.run.smithi174.stdout:(49/120): python3-jaraco-functools-2.0-4.el8.no 510 kB/s | 18 kB 00:00 2023-10-06T05:04:26.535 INFO:teuthology.orchestra.run.smithi174.stdout:(50/120): python3-google-auth-1.1.1-10.el8.noar 777 kB/s | 81 kB 00:00 2023-10-06T05:04:26.583 INFO:teuthology.orchestra.run.smithi174.stdout:(51/120): python3-logutils-0.3.5-11.el8.noarch. 1.0 MB/s | 49 kB 00:00 2023-10-06T05:04:26.633 INFO:teuthology.orchestra.run.smithi174.stdout:(52/120): python3-more-itertools-7.2.0-3.el8.no 1.1 MB/s | 59 kB 00:00 2023-10-06T05:04:26.639 INFO:teuthology.orchestra.run.smithi174.stdout:(53/120): python3-kubernetes-11.0.0-6.el8.noarc 9.5 MB/s | 1.2 MB 00:00 2023-10-06T05:04:26.683 INFO:teuthology.orchestra.run.smithi174.stdout:(54/120): python3-natsort-7.1.1-2.el8.noarch.rp 1.2 MB/s | 60 kB 00:00 2023-10-06T05:04:26.711 INFO:teuthology.orchestra.run.smithi174.stdout:(55/120): python3-pecan-1.3.2-9.el8.noarch.rpm 3.9 MB/s | 283 kB 00:00 2023-10-06T05:04:26.721 INFO:teuthology.orchestra.run.smithi174.stdout:(56/120): python3-portend-2.6-1.el8.noarch.rpm 414 kB/s | 16 kB 00:00 2023-10-06T05:04:26.746 INFO:teuthology.orchestra.run.smithi174.stdout:(57/120): python3-repoze-lru-0.7-6.el8.noarch.r 951 kB/s | 34 kB 00:00 2023-10-06T05:04:26.784 INFO:teuthology.orchestra.run.smithi174.stdout:(58/120): python3-rsa-4.9-2.el8.noarch.rpm 1.7 MB/s | 64 kB 00:00 2023-10-06T05:04:26.805 INFO:teuthology.orchestra.run.smithi174.stdout:(59/120): python3-routes-2.4.1-12.el8.noarch.rp 2.3 MB/s | 196 kB 00:00 2023-10-06T05:04:26.841 INFO:teuthology.orchestra.run.smithi174.stdout:(60/120): python3-simplegeneric-0.8.1-17.el8.no 328 kB/s | 19 kB 00:00 2023-10-06T05:04:26.842 INFO:teuthology.orchestra.run.smithi174.stdout:(61/120): python3-singledispatch-3.4.0.3-18.el8 650 kB/s | 24 kB 00:00 2023-10-06T05:04:27.241 INFO:teuthology.orchestra.run.smithi174.stdout:(62/120): ceph-test-17.2.6-1274.g68e53a4a.el8.x 9.6 MB/s | 52 MB 00:05 2023-10-06T05:04:27.241 INFO:teuthology.orchestra.run.smithi174.stdout:(63/120): python3-tempora-1.14.1-5.el8.noarch.r 73 kB/s | 29 kB 00:00 2023-10-06T05:04:27.242 INFO:teuthology.orchestra.run.smithi174.stdout:(64/120): python3-trustme-0.6.0-4.el8.noarch.rp 69 kB/s | 27 kB 00:00 2023-10-06T05:04:27.281 INFO:teuthology.orchestra.run.smithi174.stdout:(65/120): python3-websocket-client-0.56.0-5.el8 1.5 MB/s | 61 kB 00:00 2023-10-06T05:04:27.292 INFO:teuthology.orchestra.run.smithi174.stdout:(66/120): python3-webob-1.8.5-1.el8.1.noarch.rp 4.9 MB/s | 251 kB 00:00 2023-10-06T05:04:27.323 INFO:teuthology.orchestra.run.smithi174.stdout:(67/120): python3-webtest-2.0.33-1.el8.noarch.r 2.0 MB/s | 85 kB 00:00 2023-10-06T05:04:27.327 INFO:teuthology.orchestra.run.smithi174.stdout:(68/120): python3-zc-lockfile-2.0-2.el8.noarch. 665 kB/s | 23 kB 00:00 2023-10-06T05:04:27.332 INFO:teuthology.orchestra.run.smithi174.stdout:(69/120): xmlstarlet-1.6.1-20.el8.x86_64.rpm 16 MB/s | 70 kB 00:00 2023-10-06T05:04:27.462 INFO:teuthology.orchestra.run.smithi174.stdout:(70/120): thrift-0.13.0-2.el8.x86_64.rpm 13 MB/s | 1.7 MB 00:00 2023-10-06T05:04:27.464 INFO:teuthology.orchestra.run.smithi174.stdout:(71/120): python3-waitress-1.4.3-1.el8.noarch.r 1.1 MB/s | 255 kB 00:00 2023-10-06T05:04:27.508 INFO:teuthology.orchestra.run.smithi174.stdout:(72/120): python3-jsonpatch-1.21-2.el8.noarch.r 618 kB/s | 27 kB 00:00 2023-10-06T05:04:27.511 INFO:teuthology.orchestra.run.smithi174.stdout:(73/120): python3-prettytable-0.7.2-14.el8.noar 904 kB/s | 44 kB 00:00 2023-10-06T05:04:27.536 INFO:teuthology.orchestra.run.smithi174.stdout:(74/120): jq-1.5-12.el8.x86_64.rpm 5.5 MB/s | 161 kB 00:00 2023-10-06T05:04:27.538 INFO:teuthology.orchestra.run.smithi174.stdout:(75/120): flex-2.6.1-9.el8.x86_64.rpm 12 MB/s | 320 kB 00:00 2023-10-06T05:04:27.640 INFO:teuthology.orchestra.run.smithi174.stdout:(76/120): python3-scipy-1.0.0-20.module+el8.1.0 47 MB/s | 15 MB 00:00 2023-10-06T05:04:27.647 INFO:teuthology.orchestra.run.smithi174.stdout:(77/120): python3-werkzeug-0.12.2-4.el8.noarch. 4.1 MB/s | 457 kB 00:00 2023-10-06T05:04:27.662 INFO:teuthology.orchestra.run.smithi174.stdout:(78/120): python3-markupsafe-0.23-19.el8.x86_64 1.8 MB/s | 39 kB 00:00 2023-10-06T05:04:27.676 INFO:teuthology.orchestra.run.smithi174.stdout:(79/120): bison-3.0.4-10.el8.x86_64.rpm 23 MB/s | 688 kB 00:00 2023-10-06T05:04:27.684 INFO:teuthology.orchestra.run.smithi174.stdout:(80/120): lttng-ust-2.8.1-11.el8.x86_64.rpm 12 MB/s | 259 kB 00:00 2023-10-06T05:04:27.701 INFO:teuthology.orchestra.run.smithi174.stdout:(81/120): python3-babel-2.5.1-5.el8.noarch.rpm 29 MB/s | 4.8 MB 00:00 2023-10-06T05:04:27.703 INFO:teuthology.orchestra.run.smithi174.stdout:(82/120): python3-pyasn1-0.3.7-6.el8.noarch.rpm 6.3 MB/s | 126 kB 00:00 2023-10-06T05:04:27.705 INFO:teuthology.orchestra.run.smithi174.stdout:(83/120): python3-numpy-f2py-1.14.3-9.el8.x86_6 7.7 MB/s | 225 kB 00:00 2023-10-06T05:04:27.741 INFO:teuthology.orchestra.run.smithi174.stdout:(84/120): python3-jinja2-2.10.1-2.el8_0.noarch. 14 MB/s | 538 kB 00:00 2023-10-06T05:04:27.748 INFO:teuthology.orchestra.run.smithi174.stdout:(85/120): oniguruma-6.8.2-2.el8.x86_64.rpm 4.1 MB/s | 187 kB 00:00 2023-10-06T05:04:27.764 INFO:teuthology.orchestra.run.smithi174.stdout:(86/120): python3-numpy-1.14.3-9.el8.x86_64.rpm 59 MB/s | 3.7 MB 00:00 2023-10-06T05:04:27.768 INFO:teuthology.orchestra.run.smithi174.stdout:(87/120): socat-1.7.3.3-2.el8.x86_64.rpm 11 MB/s | 302 kB 00:00 2023-10-06T05:04:27.769 INFO:teuthology.orchestra.run.smithi174.stdout:(88/120): python3-mako-1.0.6-13.el8.noarch.rpm 7.4 MB/s | 157 kB 00:00 2023-10-06T05:04:27.794 INFO:teuthology.orchestra.run.smithi174.stdout:(89/120): python3-pyasn1-modules-0.3.7-6.el8.no 3.7 MB/s | 110 kB 00:00 2023-10-06T05:04:27.814 INFO:teuthology.orchestra.run.smithi174.stdout:(90/120): python3-jsonpointer-1.10-11.el8.noarc 450 kB/s | 20 kB 00:00 2023-10-06T05:04:27.818 INFO:teuthology.orchestra.run.smithi174.stdout:(91/120): librdkafka-0.11.4-1.el8.x86_64.rpm 6.9 MB/s | 353 kB 00:00 2023-10-06T05:04:27.824 INFO:teuthology.orchestra.run.smithi174.stdout:(92/120): libtool-2.4.6-25.el8.x86_64.rpm 23 MB/s | 709 kB 00:00 2023-10-06T05:04:27.837 INFO:teuthology.orchestra.run.smithi174.stdout:(93/120): python3-pytz-2017.2-9.el8.noarch.rpm 2.3 MB/s | 54 kB 00:00 2023-10-06T05:04:27.882 INFO:teuthology.orchestra.run.smithi174.stdout:(94/120): automake-1.16.1-7.el8.noarch.rpm 16 MB/s | 713 kB 00:00 2023-10-06T05:04:27.901 INFO:teuthology.orchestra.run.smithi174.stdout:(95/120): openblas-threads-0.3.12-1.el8.x86_64. 58 MB/s | 4.7 MB 00:00 2023-10-06T05:04:27.901 INFO:teuthology.orchestra.run.smithi174.stdout:(96/120): libaio-devel-0.3.112-1.el8.x86_64.rpm 966 kB/s | 19 kB 00:00 2023-10-06T05:04:27.927 INFO:teuthology.orchestra.run.smithi174.stdout:(97/120): openblas-0.3.12-1.el8.x86_64.rpm 45 MB/s | 4.6 MB 00:00 2023-10-06T05:04:27.928 INFO:teuthology.orchestra.run.smithi174.stdout:(98/120): python3-jwt-1.6.1-2.el8.noarch.rpm 1.5 MB/s | 43 kB 00:00 2023-10-06T05:04:27.947 INFO:teuthology.orchestra.run.smithi174.stdout:(99/120): keyutils-libs-devel-1.5.10-6.el8.x86_ 2.5 MB/s | 48 kB 00:00 2023-10-06T05:04:27.949 INFO:teuthology.orchestra.run.smithi174.stdout:(100/120): python3-oauthlib-2.1.0-1.el8.noarch. 3.2 MB/s | 155 kB 00:00 2023-10-06T05:04:27.953 INFO:teuthology.orchestra.run.smithi174.stdout:(101/120): pcre2-utf32-10.32-2.el8.x86_64.rpm 8.5 MB/s | 220 kB 00:00 2023-10-06T05:04:27.975 INFO:teuthology.orchestra.run.smithi174.stdout:(102/120): pcre2-utf16-10.32-2.el8.x86_64.rpm 8.1 MB/s | 229 kB 00:00 2023-10-06T05:04:27.981 INFO:teuthology.orchestra.run.smithi174.stdout:(103/120): python3-requests-oauthlib-1.0.0-1.el 1.3 MB/s | 43 kB 00:00 2023-10-06T05:04:28.008 INFO:teuthology.orchestra.run.smithi174.stdout:(104/120): openssl-devel-1.1.1g-15.el8_3.x86_64 42 MB/s | 2.3 MB 00:00 2023-10-06T05:04:28.009 INFO:teuthology.orchestra.run.smithi174.stdout:(105/120): libverto-devel-0.3.0-5.el8.x86_64.rp 522 kB/s | 18 kB 00:00 2023-10-06T05:04:28.020 INFO:teuthology.orchestra.run.smithi174.stdout:(106/120): pcre2-devel-10.32-2.el8.x86_64.rpm 15 MB/s | 605 kB 00:00 2023-10-06T05:04:28.029 INFO:teuthology.orchestra.run.smithi174.stdout:(107/120): fuse-2.9.7-12.el8.x86_64.rpm 3.9 MB/s | 83 kB 00:00 2023-10-06T05:04:28.031 INFO:teuthology.orchestra.run.smithi174.stdout:(108/120): elfutils-libelf-devel-0.182-3.el8.x8 2.6 MB/s | 59 kB 00:00 2023-10-06T05:04:28.055 INFO:teuthology.orchestra.run.smithi174.stdout:(109/120): libsepol-devel-2.9-2.el8.x86_64.rpm 3.6 MB/s | 86 kB 00:00 2023-10-06T05:04:28.056 INFO:teuthology.orchestra.run.smithi174.stdout:(110/120): libquadmath-8.4.1-1.el8.x86_64.rpm 4.6 MB/s | 169 kB 00:00 2023-10-06T05:04:28.058 INFO:teuthology.orchestra.run.smithi174.stdout:(111/120): libcom_err-devel-1.45.6-1.el8.x86_64 1.3 MB/s | 38 kB 00:00 2023-10-06T05:04:28.074 INFO:teuthology.orchestra.run.smithi174.stdout:(112/120): zlib-devel-1.2.11-17.el8.x86_64.rpm 3.0 MB/s | 58 kB 00:00 2023-10-06T05:04:28.078 INFO:teuthology.orchestra.run.smithi174.stdout:(113/120): librabbitmq-0.9.0-3.el8.x86_64.rpm 2.1 MB/s | 47 kB 00:00 2023-10-06T05:04:28.099 INFO:teuthology.orchestra.run.smithi174.stdout:(114/120): libgfortran-8.4.1-1.el8.x86_64.rpm 24 MB/s | 639 kB 00:00 2023-10-06T05:04:28.106 INFO:teuthology.orchestra.run.smithi174.stdout:(115/120): krb5-devel-1.18.2-8.3.el8_4.x86_64.r 11 MB/s | 559 kB 00:00 2023-10-06T05:04:28.124 INFO:teuthology.orchestra.run.smithi174.stdout:(116/120): xfsprogs-devel-5.0.0-8.el8.x86_64.rp 1.5 MB/s | 70 kB 00:00 2023-10-06T05:04:28.125 INFO:teuthology.orchestra.run.smithi174.stdout:(117/120): libselinux-devel-2.9-5.el8.x86_64.rp 7.6 MB/s | 200 kB 00:00 2023-10-06T05:04:28.127 INFO:teuthology.orchestra.run.smithi174.stdout:(118/120): libuuid-devel-2.32.1-27.el8.x86_64.r 4.5 MB/s | 97 kB 00:00 2023-10-06T05:04:28.136 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2023-10-06T05:04:28.141 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2023-10-06T05:04:28.600 INFO:teuthology.orchestra.run.smithi174.stdout:(119/120): librados2-17.2.6-1274.g68e53a4a.el8. 7.2 MB/s | 3.4 MB 00:00 2023-10-06T05:04:28.645 INFO:teuthology.orchestra.run.smithi174.stdout:(120/120): librbd1-17.2.6-1274.g68e53a4a.el8.x8 6.8 MB/s | 3.5 MB 00:00 2023-10-06T05:04:28.648 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2023-10-06T05:04:28.648 INFO:teuthology.orchestra.run.smithi174.stdout:Total 12 MB/s | 201 MB 00:17 2023-10-06T05:04:28.695 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2023-10-06T05:04:28.809 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2023-10-06T05:04:28.809 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2023-10-06T05:04:30.960 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2023-10-06T05:04:31.121 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2023-10-06T05:04:32.988 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2023-10-06T05:04:33.197 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/122 2023-10-06T05:04:33.423 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 2/122 2023-10-06T05:04:33.639 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 3/122 2023-10-06T05:04:33.797 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 3/122 2023-10-06T05:04:33.972 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 4/122 2023-10-06T05:04:34.389 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 4/122 2023-10-06T05:04:34.664 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : librados2-2:17.2.6-1274.g68e53a4a.el8.x86_64 5/122 2023-10-06T05:04:35.053 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librados2-2:17.2.6-1274.g68e53a4a.el8.x86_64 5/122 2023-10-06T05:04:35.280 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : librbd1-2:17.2.6-1274.g68e53a4a.el8.x86_64 6/122 2023-10-06T05:04:35.409 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librbd1-2:17.2.6-1274.g68e53a4a.el8.x86_64 6/122 2023-10-06T05:04:35.638 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephfs2-2:17.2.6-1274.g68e53a4a.el8.x86_64 7/122 2023-10-06T05:04:35.719 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libcephfs2-2:17.2.6-1274.g68e53a4a.el8.x86_64 7/122 2023-10-06T05:04:35.856 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2023-10-06T05:04:36.140 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/122 2023-10-06T05:04:36.330 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 2/122 2023-10-06T05:04:36.528 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 3/122 2023-10-06T05:04:36.686 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rados-2:17.2.6-1274.g68e53a4a.el8.x86_64 8/122 2023-10-06T05:04:36.687 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 3/122 2023-10-06T05:04:36.961 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 4/122 2023-10-06T05:04:37.340 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 4/122 2023-10-06T05:04:37.393 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : thrift-0.13.0-2.el8.x86_64 9/122 2023-10-06T05:04:37.563 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librgw2-2:17.2.6-1274.g68e53a4a.el8.x86_64 10/122 2023-10-06T05:04:37.586 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : librados2-2:17.2.6-1274.g68e53a4a.el8.x86_64 5/122 2023-10-06T05:04:37.703 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librgw2-2:17.2.6-1274.g68e53a4a.el8.x86_64 10/122 2023-10-06T05:04:37.976 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: librados2-2:17.2.6-1274.g68e53a4a.el8.x86_64 5/122 2023-10-06T05:04:37.984 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/122 2023-10-06T05:04:38.153 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/122 2023-10-06T05:04:38.169 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : librbd1-2:17.2.6-1274.g68e53a4a.el8.x86_64 6/122 2023-10-06T05:04:38.322 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: librbd1-2:17.2.6-1274.g68e53a4a.el8.x86_64 6/122 2023-10-06T05:04:38.388 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephsqlite-2:17.2.6-1274.g68e53a4a.el8.x86_64 13/122 2023-10-06T05:04:38.477 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libcephsqlite-2:17.2.6-1274.g68e53a4a.el8.x86_64 13/122 2023-10-06T05:04:38.544 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libcephfs2-2:17.2.6-1274.g68e53a4a.el8.x86_64 7/122 2023-10-06T05:04:38.641 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: libcephfs2-2:17.2.6-1274.g68e53a4a.el8.x86_64 7/122 2023-10-06T05:04:38.663 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libradosstriper1-2:17.2.6-1274.g68e53a4a.el8.x86 14/122 2023-10-06T05:04:38.714 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libradosstriper1-2:17.2.6-1274.g68e53a4a.el8.x86 14/122 2023-10-06T05:04:38.910 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : zlib-devel-1.2.11-17.el8.x86_64 15/122 2023-10-06T05:04:39.009 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-rados-2:17.2.6-1274.g68e53a4a.el8.x86_64 8/122 2023-10-06T05:04:39.063 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libquadmath-8.4.1-1.el8.x86_64 16/122 2023-10-06T05:04:39.230 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libquadmath-8.4.1-1.el8.x86_64 16/122 2023-10-06T05:04:39.413 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libgfortran-8.4.1-1.el8.x86_64 17/122 2023-10-06T05:04:39.649 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : thrift-0.13.0-2.el8.x86_64 9/122 2023-10-06T05:04:39.852 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : librgw2-2:17.2.6-1274.g68e53a4a.el8.x86_64 10/122 2023-10-06T05:04:39.964 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libgfortran-8.4.1-1.el8.x86_64 17/122 2023-10-06T05:04:39.965 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: librgw2-2:17.2.6-1274.g68e53a4a.el8.x86_64 10/122 2023-10-06T05:04:40.138 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : openblas-threads-0.3.12-1.el8.x86_64 18/122 2023-10-06T05:04:40.247 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/122 2023-10-06T05:04:40.401 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/122 2023-10-06T05:04:40.619 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libcephsqlite-2:17.2.6-1274.g68e53a4a.el8.x86_64 13/122 2023-10-06T05:04:40.681 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: openblas-threads-0.3.12-1.el8.x86_64 18/122 2023-10-06T05:04:40.731 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: libcephsqlite-2:17.2.6-1274.g68e53a4a.el8.x86_64 13/122 2023-10-06T05:04:40.877 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libradosstriper1-2:17.2.6-1274.g68e53a4a.el8.x86 14/122 2023-10-06T05:04:40.913 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : openblas-0.3.12-1.el8.x86_64 19/122 2023-10-06T05:04:40.927 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: libradosstriper1-2:17.2.6-1274.g68e53a4a.el8.x86 14/122 2023-10-06T05:04:41.261 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : zlib-devel-1.2.11-17.el8.x86_64 15/122 2023-10-06T05:04:41.427 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libquadmath-8.4.1-1.el8.x86_64 16/122 2023-10-06T05:04:41.591 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: libquadmath-8.4.1-1.el8.x86_64 16/122 2023-10-06T05:04:41.651 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: openblas-0.3.12-1.el8.x86_64 19/122 2023-10-06T05:04:41.793 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libgfortran-8.4.1-1.el8.x86_64 17/122 2023-10-06T05:04:41.958 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 20/122 2023-10-06T05:04:42.178 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 21/122 2023-10-06T05:04:42.329 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: libgfortran-8.4.1-1.el8.x86_64 17/122 2023-10-06T05:04:42.358 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 22/122 2023-10-06T05:04:42.502 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : openblas-threads-0.3.12-1.el8.x86_64 18/122 2023-10-06T05:04:42.634 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 23/122 2023-10-06T05:04:42.956 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 24/122 2023-10-06T05:04:43.067 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: openblas-threads-0.3.12-1.el8.x86_64 18/122 2023-10-06T05:04:43.225 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/122 2023-10-06T05:04:43.310 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : openblas-0.3.12-1.el8.x86_64 19/122 2023-10-06T05:04:43.436 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-ceph-common-2:17.2.6-1274.g68e53a4a.el8. 26/122 2023-10-06T05:04:43.573 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-ceph-argparse-2:17.2.6-1274.g68e53a4a.el 27/122 2023-10-06T05:04:43.845 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cephfs-2:17.2.6-1274.g68e53a4a.el8.x86_6 28/122 2023-10-06T05:04:44.037 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: openblas-0.3.12-1.el8.x86_64 19/122 2023-10-06T05:04:44.305 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 20/122 2023-10-06T05:04:44.500 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 21/122 2023-10-06T05:04:44.689 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 22/122 2023-10-06T05:04:44.896 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 29/122 2023-10-06T05:04:44.964 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 23/122 2023-10-06T05:04:45.259 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 30/122 2023-10-06T05:04:45.277 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 24/122 2023-10-06T05:04:45.563 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/122 2023-10-06T05:04:45.587 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 31/122 2023-10-06T05:04:45.766 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-ceph-common-2:17.2.6-1274.g68e53a4a.el8. 26/122 2023-10-06T05:04:45.860 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 32/122 2023-10-06T05:04:45.946 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-ceph-argparse-2:17.2.6-1274.g68e53a4a.el 27/122 2023-10-06T05:04:46.163 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 33/122 2023-10-06T05:04:46.233 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cephfs-2:17.2.6-1274.g68e53a4a.el8.x86_6 28/122 2023-10-06T05:04:46.429 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 34/122 2023-10-06T05:04:47.260 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 29/122 2023-10-06T05:04:47.597 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 30/122 2023-10-06T05:04:47.908 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 31/122 2023-10-06T05:04:48.165 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 32/122 2023-10-06T05:04:48.475 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 33/122 2023-10-06T05:04:48.504 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 34/122 2023-10-06T05:04:48.734 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 34/122 2023-10-06T05:04:48.849 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 35/122 2023-10-06T05:04:49.118 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rsa-4.9-2.el8.noarch 36/122 2023-10-06T05:04:49.365 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 37/122 2023-10-06T05:04:49.597 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rgw-2:17.2.6-1274.g68e53a4a.el8.x86_64 38/122 2023-10-06T05:04:49.801 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rbd-2:17.2.6-1274.g68e53a4a.el8.x86_64 39/122 2023-10-06T05:04:50.022 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librados-devel-2:17.2.6-1274.g68e53a4a.el8.x86_6 40/122 2023-10-06T05:04:50.269 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libuuid-devel-2.32.1-27.el8.x86_64 41/122 2023-10-06T05:04:50.467 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libsepol-devel-2.9-2.el8.x86_64 42/122 2023-10-06T05:04:50.614 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcom_err-devel-1.45.6-1.el8.x86_64 43/122 2023-10-06T05:04:50.791 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 34/122 2023-10-06T05:04:50.798 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : fuse-2.9.7-12.el8.x86_64 44/122 2023-10-06T05:04:50.933 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 45/122 2023-10-06T05:04:51.137 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 35/122 2023-10-06T05:04:51.163 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 46/122 2023-10-06T05:04:51.373 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : keyutils-libs-devel-1.5.10-6.el8.x86_64 47/122 2023-10-06T05:04:51.398 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-rsa-4.9-2.el8.noarch 36/122 2023-10-06T05:04:51.628 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 37/122 2023-10-06T05:04:51.687 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 48/122 2023-10-06T05:04:51.868 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-rgw-2:17.2.6-1274.g68e53a4a.el8.x86_64 38/122 2023-10-06T05:04:51.991 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 49/122 2023-10-06T05:04:52.086 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-rbd-2:17.2.6-1274.g68e53a4a.el8.x86_64 39/122 2023-10-06T05:04:52.273 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 50/122 2023-10-06T05:04:52.278 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : librados-devel-2:17.2.6-1274.g68e53a4a.el8.x86_6 40/122 2023-10-06T05:04:52.491 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libuuid-devel-2.32.1-27.el8.x86_64 41/122 2023-10-06T05:04:52.621 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : krb5-devel-1.18.2-8.3.el8_4.x86_64 51/122 2023-10-06T05:04:52.671 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libsepol-devel-2.9-2.el8.x86_64 42/122 2023-10-06T05:04:52.845 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libcom_err-devel-1.45.6-1.el8.x86_64 43/122 2023-10-06T05:04:52.876 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : automake-1.16.1-7.el8.noarch 52/122 2023-10-06T05:04:53.019 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : fuse-2.9.7-12.el8.x86_64 44/122 2023-10-06T05:04:53.119 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 53/122 2023-10-06T05:04:53.186 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 45/122 2023-10-06T05:04:53.333 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 54/122 2023-10-06T05:04:53.351 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 46/122 2023-10-06T05:04:53.539 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 55/122 2023-10-06T05:04:53.553 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : keyutils-libs-devel-1.5.10-6.el8.x86_64 47/122 2023-10-06T05:04:53.720 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 56/122 2023-10-06T05:04:53.784 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 56/122 2023-10-06T05:04:53.856 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 48/122 2023-10-06T05:04:54.003 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : jq-1.5-12.el8.x86_64 57/122 2023-10-06T05:04:54.128 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 49/122 2023-10-06T05:04:54.193 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 58/122 2023-10-06T05:04:54.358 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 50/122 2023-10-06T05:04:54.396 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 59/122 2023-10-06T05:04:54.612 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 60/122 2023-10-06T05:04:54.775 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : krb5-devel-1.18.2-8.3.el8_4.x86_64 51/122 2023-10-06T05:04:54.901 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 61/122 2023-10-06T05:04:55.013 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : automake-1.16.1-7.el8.noarch 52/122 2023-10-06T05:04:55.154 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 62/122 2023-10-06T05:04:55.232 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 53/122 2023-10-06T05:04:55.378 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 63/122 2023-10-06T05:04:55.430 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 54/122 2023-10-06T05:04:55.572 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 64/122 2023-10-06T05:04:55.611 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 55/122 2023-10-06T05:04:55.775 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 65/122 2023-10-06T05:04:55.792 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 56/122 2023-10-06T05:04:55.875 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 56/122 2023-10-06T05:04:55.975 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 66/122 2023-10-06T05:04:56.058 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : jq-1.5-12.el8.x86_64 57/122 2023-10-06T05:04:56.236 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 67/122 2023-10-06T05:04:56.294 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 58/122 2023-10-06T05:04:56.475 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 59/122 2023-10-06T05:04:56.507 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 68/122 2023-10-06T05:04:56.629 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 60/122 2023-10-06T05:04:56.748 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 69/122 2023-10-06T05:04:56.904 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 61/122 2023-10-06T05:04:56.969 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 70/122 2023-10-06T05:04:57.150 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 62/122 2023-10-06T05:04:57.237 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 71/122 2023-10-06T05:04:57.373 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 63/122 2023-10-06T05:04:57.532 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 72/122 2023-10-06T05:04:57.535 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 64/122 2023-10-06T05:04:57.685 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 65/122 2023-10-06T05:04:57.784 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 73/122 2023-10-06T05:04:57.918 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 66/122 2023-10-06T05:04:58.090 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-portend-2.6-1.el8.noarch 74/122 2023-10-06T05:04:58.171 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 67/122 2023-10-06T05:04:58.386 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 75/122 2023-10-06T05:04:58.411 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 68/122 2023-10-06T05:04:58.579 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 76/122 2023-10-06T05:04:58.653 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 69/122 2023-10-06T05:04:58.856 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 70/122 2023-10-06T05:04:58.865 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 77/122 2023-10-06T05:04:59.108 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 71/122 2023-10-06T05:04:59.337 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 72/122 2023-10-06T05:04:59.572 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 73/122 2023-10-06T05:04:59.886 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-portend-2.6-1.el8.noarch 74/122 2023-10-06T05:05:00.026 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 78/122 2023-10-06T05:05:00.182 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 75/122 2023-10-06T05:05:00.366 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 79/122 2023-10-06T05:05:00.399 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 76/122 2023-10-06T05:05:00.629 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 80/122 2023-10-06T05:05:00.675 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 77/122 2023-10-06T05:05:00.991 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 81/122 2023-10-06T05:05:01.297 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 82/122 2023-10-06T05:05:01.632 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 83/122 2023-10-06T05:05:01.849 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 78/122 2023-10-06T05:05:01.870 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 84/122 2023-10-06T05:05:02.086 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 85/122 2023-10-06T05:05:02.188 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 79/122 2023-10-06T05:05:02.336 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 86/122 2023-10-06T05:05:02.442 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 80/122 2023-10-06T05:05:02.786 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 81/122 2023-10-06T05:05:03.098 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 82/122 2023-10-06T05:05:03.447 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 83/122 2023-10-06T05:05:03.694 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 84/122 2023-10-06T05:05:03.887 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 85/122 2023-10-06T05:05:04.082 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 86/122 2023-10-06T05:05:07.396 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-common-2:17.2.6-1274.g68e53a4a.el8.x86_64 87/122 2023-10-06T05:05:08.410 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-common-2:17.2.6-1274.g68e53a4a.el8.x86_64 87/122 2023-10-06T05:05:09.006 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-common-2:17.2.6-1274.g68e53a4a.el8.x86_64 87/122 2023-10-06T05:05:09.198 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-common-2:17.2.6-1274.g68e53a4a.el8.x86_64 87/122 2023-10-06T05:05:09.314 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-base-2:17.2.6-1274.g68e53a4a.el8.x86_64 88/122 2023-10-06T05:05:09.403 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-base-2:17.2.6-1274.g68e53a4a.el8.x86_64 88/122 2023-10-06T05:05:09.473 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-common-2:17.2.6-1274.g68e53a4a.el8.x86_64 87/122 2023-10-06T05:05:09.584 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-selinux-2:17.2.6-1274.g68e53a4a.el8.x86_64 89/122 2023-10-06T05:05:10.061 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-common-2:17.2.6-1274.g68e53a4a.el8.x86_64 87/122 2023-10-06T05:05:10.347 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-base-2:17.2.6-1274.g68e53a4a.el8.x86_64 88/122 2023-10-06T05:05:10.432 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-base-2:17.2.6-1274.g68e53a4a.el8.x86_64 88/122 2023-10-06T05:05:10.606 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-selinux-2:17.2.6-1274.g68e53a4a.el8.x86_64 89/122 2023-10-06T05:05:27.713 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-selinux-2:17.2.6-1274.g68e53a4a.el8.x86_64 89/122 2023-10-06T05:05:27.713 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /sys 2023-10-06T05:05:27.713 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /proc 2023-10-06T05:05:27.713 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /dev 2023-10-06T05:05:27.713 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /run 2023-10-06T05:05:27.713 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /mnt 2023-10-06T05:05:27.713 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /var/tmp 2023-10-06T05:05:27.713 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /home 2023-10-06T05:05:27.713 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /tmp 2023-10-06T05:05:27.714 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /dev 2023-10-06T05:05:27.714 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:05:28.124 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-volume-2:17.2.6-1274.g68e53a4a.el8.noarch 90/122 2023-10-06T05:05:28.399 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-selinux-2:17.2.6-1274.g68e53a4a.el8.x86_64 89/122 2023-10-06T05:05:28.399 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /sys 2023-10-06T05:05:28.399 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /proc 2023-10-06T05:05:28.399 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /dev 2023-10-06T05:05:28.399 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /run 2023-10-06T05:05:28.399 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /mnt 2023-10-06T05:05:28.399 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /var/tmp 2023-10-06T05:05:28.399 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /home 2023-10-06T05:05:28.400 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /tmp 2023-10-06T05:05:28.400 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /dev 2023-10-06T05:05:28.400 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-10-06T05:05:28.795 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-volume-2:17.2.6-1274.g68e53a4a.el8.noarch 90/122 2023-10-06T05:05:29.608 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-volume-2:17.2.6-1274.g68e53a4a.el8.noarch 90/122 2023-10-06T05:05:29.778 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-osd-2:17.2.6-1274.g68e53a4a.el8.x86_64 91/122 2023-10-06T05:05:29.999 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-osd-2:17.2.6-1274.g68e53a4a.el8.x86_64 91/122 2023-10-06T05:05:30.190 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mds-2:17.2.6-1274.g68e53a4a.el8.x86_64 92/122 2023-10-06T05:05:30.296 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-volume-2:17.2.6-1274.g68e53a4a.el8.noarch 90/122 2023-10-06T05:05:30.485 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-osd-2:17.2.6-1274.g68e53a4a.el8.x86_64 91/122 2023-10-06T05:05:30.585 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mds-2:17.2.6-1274.g68e53a4a.el8.x86_64 92/122 2023-10-06T05:05:30.706 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-osd-2:17.2.6-1274.g68e53a4a.el8.x86_64 91/122 2023-10-06T05:05:30.774 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mon-2:17.2.6-1274.g68e53a4a.el8.x86_64 93/122 2023-10-06T05:05:30.791 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mon-2:17.2.6-1274.g68e53a4a.el8.x86_64 93/122 2023-10-06T05:05:30.895 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mds-2:17.2.6-1274.g68e53a4a.el8.x86_64 92/122 2023-10-06T05:05:31.294 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mds-2:17.2.6-1274.g68e53a4a.el8.x86_64 92/122 2023-10-06T05:05:31.453 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mon-2:17.2.6-1274.g68e53a4a.el8.x86_64 93/122 2023-10-06T05:05:31.470 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mon-2:17.2.6-1274.g68e53a4a.el8.x86_64 93/122 2023-10-06T05:05:33.638 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: cephadm-2:17.2.6-1274.g68e53a4a.el8.noarch 94/122 2023-10-06T05:05:33.814 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : cephadm-2:17.2.6-1274.g68e53a4a.el8.noarch 94/122 2023-10-06T05:05:34.013 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-prometheus-alerts-2:17.2.6-1274.g68e53a4a.e 95/122 2023-10-06T05:05:34.197 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-grafana-dashboards-2:17.2.6-1274.g68e53a4a. 96/122 2023-10-06T05:05:34.422 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-cephadm-2:17.2.6-1274.g68e53a4a.el8.noa 97/122 2023-10-06T05:05:34.502 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: cephadm-2:17.2.6-1274.g68e53a4a.el8.noarch 94/122 2023-10-06T05:05:34.638 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : cephadm-2:17.2.6-1274.g68e53a4a.el8.noarch 94/122 2023-10-06T05:05:34.638 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.6-1274.g68e53a4a.el8.noa 97/122 2023-10-06T05:05:34.850 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-prometheus-alerts-2:17.2.6-1274.g68e53a4a.e 95/122 2023-10-06T05:05:34.868 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-dashboard-2:17.2.6-1274.g68e53a4a.el8.n 98/122 2023-10-06T05:05:35.063 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-grafana-dashboards-2:17.2.6-1274.g68e53a4a. 96/122 2023-10-06T05:05:35.260 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-cephadm-2:17.2.6-1274.g68e53a4a.el8.noa 97/122 2023-10-06T05:05:35.475 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.6-1274.g68e53a4a.el8.noa 97/122 2023-10-06T05:05:35.701 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-dashboard-2:17.2.6-1274.g68e53a4a.el8.n 98/122 2023-10-06T05:05:35.835 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.6-1274.g68e53a4a.el8.n 98/122 2023-10-06T05:05:36.022 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.6-1274.g68e 99/122 2023-10-06T05:05:36.120 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.6-1274.g68e 99/122 2023-10-06T05:05:36.509 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-modules-core-2:17.2.6-1274.g68e53a4a.el 100/122 2023-10-06T05:05:36.668 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.6-1274.g68e53a4a.el8.n 98/122 2023-10-06T05:05:36.710 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-2:17.2.6-1274.g68e53a4a.el8.x86_64 101/122 2023-10-06T05:05:36.747 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-2:17.2.6-1274.g68e53a4a.el8.x86_64 101/122 2023-10-06T05:05:36.869 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.6-1274.g68e 99/122 2023-10-06T05:05:36.931 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-rook-2:17.2.6-1274.g68e53a4a.el8.noarch 102/122 2023-10-06T05:05:36.942 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.6-1274.g68e53a4a.el8.noarch 102/122 2023-10-06T05:05:36.967 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.6-1274.g68e 99/122 2023-10-06T05:05:37.366 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-modules-core-2:17.2.6-1274.g68e53a4a.el 100/122 2023-10-06T05:05:37.584 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-2:17.2.6-1274.g68e53a4a.el8.x86_64 101/122 2023-10-06T05:05:37.621 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mgr-2:17.2.6-1274.g68e53a4a.el8.x86_64 101/122 2023-10-06T05:05:37.760 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-rook-2:17.2.6-1274.g68e53a4a.el8.noarch 102/122 2023-10-06T05:05:37.771 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.6-1274.g68e53a4a.el8.noarch 102/122 2023-10-06T05:05:38.532 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-2:17.2.6-1274.g68e53a4a.el8.x86_64 103/122 2023-10-06T05:05:38.802 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-radosgw-2:17.2.6-1274.g68e53a4a.el8.x86_64 104/122 2023-10-06T05:05:38.882 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-radosgw-2:17.2.6-1274.g68e53a4a.el8.x86_64 104/122 2023-10-06T05:05:39.055 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-immutable-object-cache-2:17.2.6-1274.g68e53 105/122 2023-10-06T05:05:39.101 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.6-1274.g68e53 105/122 2023-10-06T05:05:39.288 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : cephfs-mirror-2:17.2.6-1274.g68e53a4a.el8.x86_64 106/122 2023-10-06T05:05:39.377 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-2:17.2.6-1274.g68e53a4a.el8.x86_64 103/122 2023-10-06T05:05:39.625 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-radosgw-2:17.2.6-1274.g68e53a4a.el8.x86_64 104/122 2023-10-06T05:05:39.636 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: cephfs-mirror-2:17.2.6-1274.g68e53a4a.el8.x86_64 106/122 2023-10-06T05:05:39.716 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-radosgw-2:17.2.6-1274.g68e53a4a.el8.x86_64 104/122 2023-10-06T05:05:39.839 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-mirror-2:17.2.6-1274.g68e53a4a.el8.x86_64 107/122 2023-10-06T05:05:39.893 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-immutable-object-cache-2:17.2.6-1274.g68e53 105/122 2023-10-06T05:05:39.940 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.6-1274.g68e53 105/122 2023-10-06T05:05:40.116 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : cephfs-mirror-2:17.2.6-1274.g68e53a4a.el8.x86_64 106/122 2023-10-06T05:05:40.463 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: cephfs-mirror-2:17.2.6-1274.g68e53a4a.el8.x86_64 106/122 2023-10-06T05:05:40.668 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : rbd-mirror-2:17.2.6-1274.g68e53a4a.el8.x86_64 107/122 2023-10-06T05:05:44.818 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: rbd-mirror-2:17.2.6-1274.g68e53a4a.el8.x86_64 107/122 2023-10-06T05:05:45.159 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-test-2:17.2.6-1274.g68e53a4a.el8.x86_64 108/122 2023-10-06T05:05:45.313 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libtool-2.4.6-25.el8.x86_64 109/122 2023-10-06T05:05:45.632 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: rbd-mirror-2:17.2.6-1274.g68e53a4a.el8.x86_64 107/122 2023-10-06T05:05:45.965 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-test-2:17.2.6-1274.g68e53a4a.el8.x86_64 108/122 2023-10-06T05:05:46.018 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 109/122 2023-10-06T05:05:46.150 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libtool-2.4.6-25.el8.x86_64 109/122 2023-10-06T05:05:46.418 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : openssl-devel-1:1.1.1g-15.el8_3.x86_64 110/122 2023-10-06T05:05:46.656 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-fuse-2:17.2.6-1274.g68e53a4a.el8.x86_64 111/122 2023-10-06T05:05:46.856 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 109/122 2023-10-06T05:05:46.857 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : xfsprogs-devel-5.0.0-8.el8.x86_64 112/122 2023-10-06T05:05:47.025 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephfs-devel-2:17.2.6-1274.g68e53a4a.el8.x86_ 113/122 2023-10-06T05:05:47.226 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : elfutils-libelf-devel-0.182-3.el8.x86_64 114/122 2023-10-06T05:05:47.245 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : openssl-devel-1:1.1.1g-15.el8_3.x86_64 110/122 2023-10-06T05:05:47.412 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : cephfs-top-2:17.2.6-1274.g68e53a4a.el8.noarch 115/122 2023-10-06T05:05:47.467 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-fuse-2:17.2.6-1274.g68e53a4a.el8.x86_64 111/122 2023-10-06T05:05:47.604 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-fuse-2:17.2.6-1274.g68e53a4a.el8.x86_64 116/122 2023-10-06T05:05:47.637 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : xfsprogs-devel-5.0.0-8.el8.x86_64 112/122 2023-10-06T05:05:47.780 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libcephfs-devel-2:17.2.6-1274.g68e53a4a.el8.x86_ 113/122 2023-10-06T05:05:47.780 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-nbd-2:17.2.6-1274.g68e53a4a.el8.x86_64 117/122 2023-10-06T05:05:47.963 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : elfutils-libelf-devel-0.182-3.el8.x86_64 114/122 2023-10-06T05:05:48.014 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 118/122 2023-10-06T05:05:48.150 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : cephfs-top-2:17.2.6-1274.g68e53a4a.el8.noarch 115/122 2023-10-06T05:05:48.195 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : bison-3.0.4-10.el8.x86_64 119/122 2023-10-06T05:05:48.256 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 119/122 2023-10-06T05:05:48.399 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : rbd-fuse-2:17.2.6-1274.g68e53a4a.el8.x86_64 116/122 2023-10-06T05:05:48.471 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flex-2.6.1-9.el8.x86_64 120/122 2023-10-06T05:05:48.480 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 120/122 2023-10-06T05:05:48.480 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 121/122 2023-10-06T05:05:48.551 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : rbd-nbd-2:17.2.6-1274.g68e53a4a.el8.x86_64 117/122 2023-10-06T05:05:48.724 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 121/122 2023-10-06T05:05:48.724 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 122/122 2023-10-06T05:05:48.796 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 118/122 2023-10-06T05:05:49.016 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : bison-3.0.4-10.el8.x86_64 119/122 2023-10-06T05:05:49.067 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 119/122 2023-10-06T05:05:49.258 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : flex-2.6.1-9.el8.x86_64 120/122 2023-10-06T05:05:49.267 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 120/122 2023-10-06T05:05:49.267 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 121/122 2023-10-06T05:05:49.503 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 121/122 2023-10-06T05:05:49.503 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 122/122 2023-10-06T05:05:49.615 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 122/122 2023-10-06T05:05:49.615 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-2:17.2.6-1274.g68e53a4a.el8.x86_64 1/122 2023-10-06T05:05:49.615 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-base-2:17.2.6-1274.g68e53a4a.el8.x86_64 2/122 2023-10-06T05:05:49.615 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-common-2:17.2.6-1274.g68e53a4a.el8.x86_64 3/122 2023-10-06T05:05:49.615 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-fuse-2:17.2.6-1274.g68e53a4a.el8.x86_64 4/122 2023-10-06T05:05:49.616 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-immutable-object-cache-2:17.2.6-1274.g68e53 5/122 2023-10-06T05:05:49.616 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mds-2:17.2.6-1274.g68e53a4a.el8.x86_64 6/122 2023-10-06T05:05:49.616 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-2:17.2.6-1274.g68e53a4a.el8.x86_64 7/122 2023-10-06T05:05:49.616 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mon-2:17.2.6-1274.g68e53a4a.el8.x86_64 8/122 2023-10-06T05:05:49.616 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-osd-2:17.2.6-1274.g68e53a4a.el8.x86_64 9/122 2023-10-06T05:05:49.616 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-radosgw-2:17.2.6-1274.g68e53a4a.el8.x86_64 10/122 2023-10-06T05:05:49.616 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-selinux-2:17.2.6-1274.g68e53a4a.el8.x86_64 11/122 2023-10-06T05:05:49.616 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-test-2:17.2.6-1274.g68e53a4a.el8.x86_64 12/122 2023-10-06T05:05:49.616 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : cephfs-mirror-2:17.2.6-1274.g68e53a4a.el8.x86_64 13/122 2023-10-06T05:05:49.616 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephfs-devel-2:17.2.6-1274.g68e53a4a.el8.x86_ 14/122 2023-10-06T05:05:49.616 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephfs2-2:17.2.6-1274.g68e53a4a.el8.x86_64 15/122 2023-10-06T05:05:49.616 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephsqlite-2:17.2.6-1274.g68e53a4a.el8.x86_64 16/122 2023-10-06T05:05:49.617 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados-devel-2:17.2.6-1274.g68e53a4a.el8.x86_6 17/122 2023-10-06T05:05:49.617 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libradosstriper1-2:17.2.6-1274.g68e53a4a.el8.x86 18/122 2023-10-06T05:05:49.617 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librgw2-2:17.2.6-1274.g68e53a4a.el8.x86_64 19/122 2023-10-06T05:05:49.617 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-ceph-argparse-2:17.2.6-1274.g68e53a4a.el 20/122 2023-10-06T05:05:49.617 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-ceph-common-2:17.2.6-1274.g68e53a4a.el8. 21/122 2023-10-06T05:05:49.617 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cephfs-2:17.2.6-1274.g68e53a4a.el8.x86_6 22/122 2023-10-06T05:05:49.617 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rados-2:17.2.6-1274.g68e53a4a.el8.x86_64 23/122 2023-10-06T05:05:49.617 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rbd-2:17.2.6-1274.g68e53a4a.el8.x86_64 24/122 2023-10-06T05:05:49.618 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rgw-2:17.2.6-1274.g68e53a4a.el8.x86_64 25/122 2023-10-06T05:05:49.618 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-fuse-2:17.2.6-1274.g68e53a4a.el8.x86_64 26/122 2023-10-06T05:05:49.618 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-mirror-2:17.2.6-1274.g68e53a4a.el8.x86_64 27/122 2023-10-06T05:05:49.618 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-nbd-2:17.2.6-1274.g68e53a4a.el8.x86_64 28/122 2023-10-06T05:05:49.618 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-grafana-dashboards-2:17.2.6-1274.g68e53a4a. 29/122 2023-10-06T05:05:49.618 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-cephadm-2:17.2.6-1274.g68e53a4a.el8.noa 30/122 2023-10-06T05:05:49.618 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-dashboard-2:17.2.6-1274.g68e53a4a.el8.n 31/122 2023-10-06T05:05:49.618 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.6-1274.g68e 32/122 2023-10-06T05:05:49.618 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-modules-core-2:17.2.6-1274.g68e53a4a.el 33/122 2023-10-06T05:05:49.618 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-rook-2:17.2.6-1274.g68e53a4a.el8.noarch 34/122 2023-10-06T05:05:49.618 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-prometheus-alerts-2:17.2.6-1274.g68e53a4a.e 35/122 2023-10-06T05:05:49.619 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-volume-2:17.2.6-1274.g68e53a4a.el8.noarch 36/122 2023-10-06T05:05:49.619 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : cephadm-2:17.2.6-1274.g68e53a4a.el8.noarch 37/122 2023-10-06T05:05:49.619 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : cephfs-top-2:17.2.6-1274.g68e53a4a.el8.noarch 38/122 2023-10-06T05:05:49.619 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/122 2023-10-06T05:05:49.619 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/122 2023-10-06T05:05:49.619 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/122 2023-10-06T05:05:49.619 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/122 2023-10-06T05:05:49.619 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/122 2023-10-06T05:05:49.620 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/122 2023-10-06T05:05:49.620 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/122 2023-10-06T05:05:49.620 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/122 2023-10-06T05:05:49.620 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/122 2023-10-06T05:05:49.620 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/122 2023-10-06T05:05:49.620 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/122 2023-10-06T05:05:49.620 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/122 2023-10-06T05:05:49.620 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/122 2023-10-06T05:05:49.620 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/122 2023-10-06T05:05:49.621 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/122 2023-10-06T05:05:49.621 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/122 2023-10-06T05:05:49.621 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/122 2023-10-06T05:05:49.621 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/122 2023-10-06T05:05:49.621 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/122 2023-10-06T05:05:49.621 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/122 2023-10-06T05:05:49.621 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/122 2023-10-06T05:05:49.621 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/122 2023-10-06T05:05:49.621 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/122 2023-10-06T05:05:49.621 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/122 2023-10-06T05:05:49.622 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/122 2023-10-06T05:05:49.622 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/122 2023-10-06T05:05:49.622 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/122 2023-10-06T05:05:49.622 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/122 2023-10-06T05:05:49.622 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/122 2023-10-06T05:05:49.622 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/122 2023-10-06T05:05:49.622 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/122 2023-10-06T05:05:49.622 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/122 2023-10-06T05:05:49.622 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/122 2023-10-06T05:05:49.622 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 72/122 2023-10-06T05:05:49.623 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 73/122 2023-10-06T05:05:49.623 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 74/122 2023-10-06T05:05:49.623 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : jq-1.5-12.el8.x86_64 75/122 2023-10-06T05:05:49.623 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flex-2.6.1-9.el8.x86_64 76/122 2023-10-06T05:05:49.623 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 77/122 2023-10-06T05:05:49.623 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 78/122 2023-10-06T05:05:49.623 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 79/122 2023-10-06T05:05:49.623 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : bison-3.0.4-10.el8.x86_64 80/122 2023-10-06T05:05:49.623 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 81/122 2023-10-06T05:05:49.623 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 82/122 2023-10-06T05:05:49.623 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 83/122 2023-10-06T05:05:49.623 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 84/122 2023-10-06T05:05:49.624 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 85/122 2023-10-06T05:05:49.624 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 86/122 2023-10-06T05:05:49.624 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 87/122 2023-10-06T05:05:49.624 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 88/122 2023-10-06T05:05:49.624 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 89/122 2023-10-06T05:05:49.624 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 90/122 2023-10-06T05:05:49.624 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 91/122 2023-10-06T05:05:49.624 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 92/122 2023-10-06T05:05:49.624 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 93/122 2023-10-06T05:05:49.624 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openblas-threads-0.3.12-1.el8.x86_64 94/122 2023-10-06T05:05:49.624 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openblas-0.3.12-1.el8.x86_64 95/122 2023-10-06T05:05:49.625 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : automake-1.16.1-7.el8.noarch 96/122 2023-10-06T05:05:49.625 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 97/122 2023-10-06T05:05:49.625 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 98/122 2023-10-06T05:05:49.625 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 99/122 2023-10-06T05:05:49.625 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 100/122 2023-10-06T05:05:49.625 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : keyutils-libs-devel-1.5.10-6.el8.x86_64 101/122 2023-10-06T05:05:49.625 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 102/122 2023-10-06T05:05:49.625 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 103/122 2023-10-06T05:05:49.626 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openssl-devel-1:1.1.1g-15.el8_3.x86_64 104/122 2023-10-06T05:05:49.626 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 105/122 2023-10-06T05:05:49.626 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 106/122 2023-10-06T05:05:49.626 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 107/122 2023-10-06T05:05:49.626 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : elfutils-libelf-devel-0.182-3.el8.x86_64 108/122 2023-10-06T05:05:49.626 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libquadmath-8.4.1-1.el8.x86_64 109/122 2023-10-06T05:05:49.627 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcom_err-devel-1.45.6-1.el8.x86_64 110/122 2023-10-06T05:05:49.627 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libsepol-devel-2.9-2.el8.x86_64 111/122 2023-10-06T05:05:49.627 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : zlib-devel-1.2.11-17.el8.x86_64 112/122 2023-10-06T05:05:49.627 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 113/122 2023-10-06T05:05:49.627 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : krb5-devel-1.18.2-8.3.el8_4.x86_64 114/122 2023-10-06T05:05:49.627 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libgfortran-8.4.1-1.el8.x86_64 115/122 2023-10-06T05:05:49.628 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : xfsprogs-devel-5.0.0-8.el8.x86_64 116/122 2023-10-06T05:05:49.628 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 117/122 2023-10-06T05:05:49.628 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libuuid-devel-2.32.1-27.el8.x86_64 118/122 2023-10-06T05:05:49.628 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados2-2:17.2.6-1274.g68e53a4a.el8.x86_64 119/122 2023-10-06T05:05:49.628 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 120/122 2023-10-06T05:05:49.628 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librbd1-2:17.2.6-1274.g68e53a4a.el8.x86_64 121/122 2023-10-06T05:05:50.451 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 122/122 2023-10-06T05:05:50.451 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-2:17.2.6-1274.g68e53a4a.el8.x86_64 1/122 2023-10-06T05:05:50.451 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-base-2:17.2.6-1274.g68e53a4a.el8.x86_64 2/122 2023-10-06T05:05:50.451 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-common-2:17.2.6-1274.g68e53a4a.el8.x86_64 3/122 2023-10-06T05:05:50.451 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-fuse-2:17.2.6-1274.g68e53a4a.el8.x86_64 4/122 2023-10-06T05:05:50.451 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-immutable-object-cache-2:17.2.6-1274.g68e53 5/122 2023-10-06T05:05:50.451 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mds-2:17.2.6-1274.g68e53a4a.el8.x86_64 6/122 2023-10-06T05:05:50.451 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-2:17.2.6-1274.g68e53a4a.el8.x86_64 7/122 2023-10-06T05:05:50.451 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mon-2:17.2.6-1274.g68e53a4a.el8.x86_64 8/122 2023-10-06T05:05:50.451 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-osd-2:17.2.6-1274.g68e53a4a.el8.x86_64 9/122 2023-10-06T05:05:50.452 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-radosgw-2:17.2.6-1274.g68e53a4a.el8.x86_64 10/122 2023-10-06T05:05:50.452 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-selinux-2:17.2.6-1274.g68e53a4a.el8.x86_64 11/122 2023-10-06T05:05:50.452 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-test-2:17.2.6-1274.g68e53a4a.el8.x86_64 12/122 2023-10-06T05:05:50.452 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : cephfs-mirror-2:17.2.6-1274.g68e53a4a.el8.x86_64 13/122 2023-10-06T05:05:50.452 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libcephfs-devel-2:17.2.6-1274.g68e53a4a.el8.x86_ 14/122 2023-10-06T05:05:50.452 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libcephfs2-2:17.2.6-1274.g68e53a4a.el8.x86_64 15/122 2023-10-06T05:05:50.452 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libcephsqlite-2:17.2.6-1274.g68e53a4a.el8.x86_64 16/122 2023-10-06T05:05:50.452 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librados-devel-2:17.2.6-1274.g68e53a4a.el8.x86_6 17/122 2023-10-06T05:05:50.452 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libradosstriper1-2:17.2.6-1274.g68e53a4a.el8.x86 18/122 2023-10-06T05:05:50.452 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librgw2-2:17.2.6-1274.g68e53a4a.el8.x86_64 19/122 2023-10-06T05:05:50.452 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-ceph-argparse-2:17.2.6-1274.g68e53a4a.el 20/122 2023-10-06T05:05:50.452 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-ceph-common-2:17.2.6-1274.g68e53a4a.el8. 21/122 2023-10-06T05:05:50.453 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cephfs-2:17.2.6-1274.g68e53a4a.el8.x86_6 22/122 2023-10-06T05:05:50.453 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-rados-2:17.2.6-1274.g68e53a4a.el8.x86_64 23/122 2023-10-06T05:05:50.453 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-rbd-2:17.2.6-1274.g68e53a4a.el8.x86_64 24/122 2023-10-06T05:05:50.453 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-rgw-2:17.2.6-1274.g68e53a4a.el8.x86_64 25/122 2023-10-06T05:05:50.453 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : rbd-fuse-2:17.2.6-1274.g68e53a4a.el8.x86_64 26/122 2023-10-06T05:05:50.453 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : rbd-mirror-2:17.2.6-1274.g68e53a4a.el8.x86_64 27/122 2023-10-06T05:05:50.453 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : rbd-nbd-2:17.2.6-1274.g68e53a4a.el8.x86_64 28/122 2023-10-06T05:05:50.453 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-grafana-dashboards-2:17.2.6-1274.g68e53a4a. 29/122 2023-10-06T05:05:50.453 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-cephadm-2:17.2.6-1274.g68e53a4a.el8.noa 30/122 2023-10-06T05:05:50.453 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-dashboard-2:17.2.6-1274.g68e53a4a.el8.n 31/122 2023-10-06T05:05:50.453 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.6-1274.g68e 32/122 2023-10-06T05:05:50.453 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-modules-core-2:17.2.6-1274.g68e53a4a.el 33/122 2023-10-06T05:05:50.454 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-rook-2:17.2.6-1274.g68e53a4a.el8.noarch 34/122 2023-10-06T05:05:50.454 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-prometheus-alerts-2:17.2.6-1274.g68e53a4a.e 35/122 2023-10-06T05:05:50.454 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-volume-2:17.2.6-1274.g68e53a4a.el8.noarch 36/122 2023-10-06T05:05:50.454 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : cephadm-2:17.2.6-1274.g68e53a4a.el8.noarch 37/122 2023-10-06T05:05:50.454 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : cephfs-top-2:17.2.6-1274.g68e53a4a.el8.noarch 38/122 2023-10-06T05:05:50.454 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/122 2023-10-06T05:05:50.454 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/122 2023-10-06T05:05:50.454 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/122 2023-10-06T05:05:50.454 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/122 2023-10-06T05:05:50.454 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/122 2023-10-06T05:05:50.454 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/122 2023-10-06T05:05:50.455 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/122 2023-10-06T05:05:50.455 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/122 2023-10-06T05:05:50.455 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/122 2023-10-06T05:05:50.455 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/122 2023-10-06T05:05:50.455 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/122 2023-10-06T05:05:50.455 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/122 2023-10-06T05:05:50.455 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/122 2023-10-06T05:05:50.455 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/122 2023-10-06T05:05:50.455 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/122 2023-10-06T05:05:50.455 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/122 2023-10-06T05:05:50.455 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/122 2023-10-06T05:05:50.456 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/122 2023-10-06T05:05:50.456 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/122 2023-10-06T05:05:50.456 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/122 2023-10-06T05:05:50.456 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/122 2023-10-06T05:05:50.456 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/122 2023-10-06T05:05:50.456 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/122 2023-10-06T05:05:50.456 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/122 2023-10-06T05:05:50.456 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/122 2023-10-06T05:05:50.456 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/122 2023-10-06T05:05:50.456 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/122 2023-10-06T05:05:50.456 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/122 2023-10-06T05:05:50.457 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/122 2023-10-06T05:05:50.457 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/122 2023-10-06T05:05:50.457 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/122 2023-10-06T05:05:50.457 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/122 2023-10-06T05:05:50.457 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/122 2023-10-06T05:05:50.457 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 72/122 2023-10-06T05:05:50.457 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 73/122 2023-10-06T05:05:50.457 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 74/122 2023-10-06T05:05:50.457 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : jq-1.5-12.el8.x86_64 75/122 2023-10-06T05:05:50.457 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : flex-2.6.1-9.el8.x86_64 76/122 2023-10-06T05:05:50.457 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 77/122 2023-10-06T05:05:50.457 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 78/122 2023-10-06T05:05:50.458 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 79/122 2023-10-06T05:05:50.458 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : bison-3.0.4-10.el8.x86_64 80/122 2023-10-06T05:05:50.458 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 81/122 2023-10-06T05:05:50.458 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 82/122 2023-10-06T05:05:50.458 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 83/122 2023-10-06T05:05:50.458 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 84/122 2023-10-06T05:05:50.458 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 85/122 2023-10-06T05:05:50.458 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 86/122 2023-10-06T05:05:50.458 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 87/122 2023-10-06T05:05:50.458 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 88/122 2023-10-06T05:05:50.458 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 89/122 2023-10-06T05:05:50.459 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 90/122 2023-10-06T05:05:50.459 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 91/122 2023-10-06T05:05:50.459 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 92/122 2023-10-06T05:05:50.459 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 93/122 2023-10-06T05:05:50.459 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : openblas-threads-0.3.12-1.el8.x86_64 94/122 2023-10-06T05:05:50.459 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : openblas-0.3.12-1.el8.x86_64 95/122 2023-10-06T05:05:50.459 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : automake-1.16.1-7.el8.noarch 96/122 2023-10-06T05:05:50.459 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 97/122 2023-10-06T05:05:50.459 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 98/122 2023-10-06T05:05:50.459 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 99/122 2023-10-06T05:05:50.459 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 100/122 2023-10-06T05:05:50.459 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : keyutils-libs-devel-1.5.10-6.el8.x86_64 101/122 2023-10-06T05:05:50.460 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 102/122 2023-10-06T05:05:50.460 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 103/122 2023-10-06T05:05:50.460 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : openssl-devel-1:1.1.1g-15.el8_3.x86_64 104/122 2023-10-06T05:05:50.460 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 105/122 2023-10-06T05:05:50.460 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 106/122 2023-10-06T05:05:50.460 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 107/122 2023-10-06T05:05:50.460 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : elfutils-libelf-devel-0.182-3.el8.x86_64 108/122 2023-10-06T05:05:50.460 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libquadmath-8.4.1-1.el8.x86_64 109/122 2023-10-06T05:05:50.460 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libcom_err-devel-1.45.6-1.el8.x86_64 110/122 2023-10-06T05:05:50.460 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libsepol-devel-2.9-2.el8.x86_64 111/122 2023-10-06T05:05:50.460 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : zlib-devel-1.2.11-17.el8.x86_64 112/122 2023-10-06T05:05:50.461 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 113/122 2023-10-06T05:05:50.461 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : krb5-devel-1.18.2-8.3.el8_4.x86_64 114/122 2023-10-06T05:05:50.461 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libgfortran-8.4.1-1.el8.x86_64 115/122 2023-10-06T05:05:50.461 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : xfsprogs-devel-5.0.0-8.el8.x86_64 116/122 2023-10-06T05:05:50.461 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 117/122 2023-10-06T05:05:50.461 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libuuid-devel-2.32.1-27.el8.x86_64 118/122 2023-10-06T05:05:50.461 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librados2-2:17.2.6-1274.g68e53a4a.el8.x86_64 119/122 2023-10-06T05:05:50.461 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 120/122 2023-10-06T05:05:50.461 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librbd1-2:17.2.6-1274.g68e53a4a.el8.x86_64 121/122 2023-10-06T05:05:51.873 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 122/122 2023-10-06T05:05:51.874 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:49 ago on Fri 06 Oct 2023 05:04:02 AM UTC. 2023-10-06T05:05:52.774 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 122/122 2023-10-06T05:05:52.774 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:01:50 ago on Fri 06 Oct 2023 05:04:02 AM UTC. 2023-10-06T05:05:53.831 INFO:teuthology.orchestra.run.smithi098.stdout:Installed products updated. 2023-10-06T05:05:53.970 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:05:53.970 INFO:teuthology.orchestra.run.smithi098.stdout:Upgraded: 2023-10-06T05:05:53.970 INFO:teuthology.orchestra.run.smithi098.stdout: librados2-2:17.2.6-1274.g68e53a4a.el8.x86_64 2023-10-06T05:05:53.971 INFO:teuthology.orchestra.run.smithi098.stdout: librbd1-2:17.2.6-1274.g68e53a4a.el8.x86_64 2023-10-06T05:05:53.971 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2023-10-06T05:05:53.971 INFO:teuthology.orchestra.run.smithi098.stdout: automake-1.16.1-7.el8.noarch 2023-10-06T05:05:53.971 INFO:teuthology.orchestra.run.smithi098.stdout: bison-3.0.4-10.el8.x86_64 2023-10-06T05:05:53.971 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-2:17.2.6-1274.g68e53a4a.el8.x86_64 2023-10-06T05:05:53.972 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-base-2:17.2.6-1274.g68e53a4a.el8.x86_64 2023-10-06T05:05:53.972 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-common-2:17.2.6-1274.g68e53a4a.el8.x86_64 2023-10-06T05:05:53.972 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-fuse-2:17.2.6-1274.g68e53a4a.el8.x86_64 2023-10-06T05:05:53.972 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-grafana-dashboards-2:17.2.6-1274.g68e53a4a.el8.noarch 2023-10-06T05:05:53.972 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-immutable-object-cache-2:17.2.6-1274.g68e53a4a.el8.x86_64 2023-10-06T05:05:53.973 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mds-2:17.2.6-1274.g68e53a4a.el8.x86_64 2023-10-06T05:05:53.973 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-2:17.2.6-1274.g68e53a4a.el8.x86_64 2023-10-06T05:05:53.973 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-cephadm-2:17.2.6-1274.g68e53a4a.el8.noarch 2023-10-06T05:05:53.973 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-dashboard-2:17.2.6-1274.g68e53a4a.el8.noarch 2023-10-06T05:05:53.973 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-diskprediction-local-2:17.2.6-1274.g68e53a4a.el8.noarch 2023-10-06T05:05:53.973 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-modules-core-2:17.2.6-1274.g68e53a4a.el8.noarch 2023-10-06T05:05:53.974 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-rook-2:17.2.6-1274.g68e53a4a.el8.noarch 2023-10-06T05:05:53.974 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mon-2:17.2.6-1274.g68e53a4a.el8.x86_64 2023-10-06T05:05:53.974 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-osd-2:17.2.6-1274.g68e53a4a.el8.x86_64 2023-10-06T05:05:53.974 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-prometheus-alerts-2:17.2.6-1274.g68e53a4a.el8.noarch 2023-10-06T05:05:53.974 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-radosgw-2:17.2.6-1274.g68e53a4a.el8.x86_64 2023-10-06T05:05:53.975 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-selinux-2:17.2.6-1274.g68e53a4a.el8.x86_64 2023-10-06T05:05:53.975 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-test-2:17.2.6-1274.g68e53a4a.el8.x86_64 2023-10-06T05:05:53.975 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-volume-2:17.2.6-1274.g68e53a4a.el8.noarch 2023-10-06T05:05:53.975 INFO:teuthology.orchestra.run.smithi098.stdout: cephadm-2:17.2.6-1274.g68e53a4a.el8.noarch 2023-10-06T05:05:53.975 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-mirror-2:17.2.6-1274.g68e53a4a.el8.x86_64 2023-10-06T05:05:53.976 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-top-2:17.2.6-1274.g68e53a4a.el8.noarch 2023-10-06T05:05:53.976 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-libelf-devel-0.182-3.el8.x86_64 2023-10-06T05:05:53.976 INFO:teuthology.orchestra.run.smithi098.stdout: flex-2.6.1-9.el8.x86_64 2023-10-06T05:05:53.976 INFO:teuthology.orchestra.run.smithi098.stdout: fuse-2.9.7-12.el8.x86_64 2023-10-06T05:05:53.976 INFO:teuthology.orchestra.run.smithi098.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-10-06T05:05:53.976 INFO:teuthology.orchestra.run.smithi098.stdout: jq-1.5-12.el8.x86_64 2023-10-06T05:05:53.977 INFO:teuthology.orchestra.run.smithi098.stdout: keyutils-libs-devel-1.5.10-6.el8.x86_64 2023-10-06T05:05:53.977 INFO:teuthology.orchestra.run.smithi098.stdout: krb5-devel-1.18.2-8.3.el8_4.x86_64 2023-10-06T05:05:53.977 INFO:teuthology.orchestra.run.smithi098.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-10-06T05:05:53.977 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs-devel-2:17.2.6-1274.g68e53a4a.el8.x86_64 2023-10-06T05:05:53.977 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs2-2:17.2.6-1274.g68e53a4a.el8.x86_64 2023-10-06T05:05:53.977 INFO:teuthology.orchestra.run.smithi098.stdout: libcephsqlite-2:17.2.6-1274.g68e53a4a.el8.x86_64 2023-10-06T05:05:53.978 INFO:teuthology.orchestra.run.smithi098.stdout: libcom_err-devel-1.45.6-1.el8.x86_64 2023-10-06T05:05:53.978 INFO:teuthology.orchestra.run.smithi098.stdout: libgfortran-8.4.1-1.el8.x86_64 2023-10-06T05:05:53.978 INFO:teuthology.orchestra.run.smithi098.stdout: liboath-2.6.2-3.el8.x86_64 2023-10-06T05:05:53.978 INFO:teuthology.orchestra.run.smithi098.stdout: libquadmath-8.4.1-1.el8.x86_64 2023-10-06T05:05:53.978 INFO:teuthology.orchestra.run.smithi098.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-10-06T05:05:53.978 INFO:teuthology.orchestra.run.smithi098.stdout: librados-devel-2:17.2.6-1274.g68e53a4a.el8.x86_64 2023-10-06T05:05:53.979 INFO:teuthology.orchestra.run.smithi098.stdout: libradosstriper1-2:17.2.6-1274.g68e53a4a.el8.x86_64 2023-10-06T05:05:53.979 INFO:teuthology.orchestra.run.smithi098.stdout: librdkafka-0.11.4-1.el8.x86_64 2023-10-06T05:05:53.979 INFO:teuthology.orchestra.run.smithi098.stdout: librgw2-2:17.2.6-1274.g68e53a4a.el8.x86_64 2023-10-06T05:05:53.979 INFO:teuthology.orchestra.run.smithi098.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-10-06T05:05:53.979 INFO:teuthology.orchestra.run.smithi098.stdout: libsepol-devel-2.9-2.el8.x86_64 2023-10-06T05:05:53.980 INFO:teuthology.orchestra.run.smithi098.stdout: libtool-2.4.6-25.el8.x86_64 2023-10-06T05:05:53.980 INFO:teuthology.orchestra.run.smithi098.stdout: libunwind-1.3.1-3.el8.x86_64 2023-10-06T05:05:53.980 INFO:teuthology.orchestra.run.smithi098.stdout: libuuid-devel-2.32.1-27.el8.x86_64 2023-10-06T05:05:53.980 INFO:teuthology.orchestra.run.smithi098.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-10-06T05:05:53.980 INFO:teuthology.orchestra.run.smithi098.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-10-06T05:05:53.980 INFO:teuthology.orchestra.run.smithi098.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-10-06T05:05:53.980 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-0.3.12-1.el8.x86_64 2023-10-06T05:05:53.981 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-threads-0.3.12-1.el8.x86_64 2023-10-06T05:05:53.981 INFO:teuthology.orchestra.run.smithi098.stdout: openssl-devel-1:1.1.1g-15.el8_3.x86_64 2023-10-06T05:05:53.981 INFO:teuthology.orchestra.run.smithi098.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-10-06T05:05:53.981 INFO:teuthology.orchestra.run.smithi098.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-10-06T05:05:53.981 INFO:teuthology.orchestra.run.smithi098.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-10-06T05:05:53.981 INFO:teuthology.orchestra.run.smithi098.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-10-06T05:05:53.981 INFO:teuthology.orchestra.run.smithi098.stdout: python3-babel-2.5.1-5.el8.noarch 2023-10-06T05:05:53.981 INFO:teuthology.orchestra.run.smithi098.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-10-06T05:05:53.981 INFO:teuthology.orchestra.run.smithi098.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-10-06T05:05:53.982 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-10-06T05:05:53.982 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-argparse-2:17.2.6-1274.g68e53a4a.el8.x86_64 2023-10-06T05:05:53.982 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-common-2:17.2.6-1274.g68e53a4a.el8.x86_64 2023-10-06T05:05:53.982 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cephfs-2:17.2.6-1274.g68e53a4a.el8.x86_64 2023-10-06T05:05:53.982 INFO:teuthology.orchestra.run.smithi098.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-10-06T05:05:53.982 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-10-06T05:05:53.982 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-10-06T05:05:53.982 INFO:teuthology.orchestra.run.smithi098.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-10-06T05:05:53.983 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-6.2-6.el8.noarch 2023-10-06T05:05:53.983 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-10-06T05:05:53.983 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2023-10-06T05:05:53.983 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-10-06T05:05:53.983 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-10-06T05:05:53.983 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-10-06T05:05:53.983 INFO:teuthology.orchestra.run.smithi098.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-10-06T05:05:53.983 INFO:teuthology.orchestra.run.smithi098.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-10-06T05:05:53.984 INFO:teuthology.orchestra.run.smithi098.stdout: python3-mako-1.0.6-13.el8.noarch 2023-10-06T05:05:53.984 INFO:teuthology.orchestra.run.smithi098.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-10-06T05:05:53.984 INFO:teuthology.orchestra.run.smithi098.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-10-06T05:05:53.984 INFO:teuthology.orchestra.run.smithi098.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-10-06T05:05:53.984 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2023-10-06T05:05:53.984 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2023-10-06T05:05:53.984 INFO:teuthology.orchestra.run.smithi098.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-10-06T05:05:53.985 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-10-06T05:05:53.985 INFO:teuthology.orchestra.run.smithi098.stdout: python3-portend-2.6-1.el8.noarch 2023-10-06T05:05:53.985 INFO:teuthology.orchestra.run.smithi098.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-10-06T05:05:53.985 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-10-06T05:05:53.985 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-10-06T05:05:53.985 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pytz-2017.2-9.el8.noarch 2023-10-06T05:05:53.985 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rados-2:17.2.6-1274.g68e53a4a.el8.x86_64 2023-10-06T05:05:53.985 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rbd-2:17.2.6-1274.g68e53a4a.el8.x86_64 2023-10-06T05:05:53.985 INFO:teuthology.orchestra.run.smithi098.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-10-06T05:05:53.986 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-10-06T05:05:53.986 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rgw-2:17.2.6-1274.g68e53a4a.el8.x86_64 2023-10-06T05:05:53.986 INFO:teuthology.orchestra.run.smithi098.stdout: python3-routes-2.4.1-12.el8.noarch 2023-10-06T05:05:53.986 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rsa-4.9-2.el8.noarch 2023-10-06T05:05:53.986 INFO:teuthology.orchestra.run.smithi098.stdout: python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7.x86_64 2023-10-06T05:05:53.986 INFO:teuthology.orchestra.run.smithi098.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-10-06T05:05:53.987 INFO:teuthology.orchestra.run.smithi098.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-10-06T05:05:53.987 INFO:teuthology.orchestra.run.smithi098.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-10-06T05:05:53.987 INFO:teuthology.orchestra.run.smithi098.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-10-06T05:05:53.987 INFO:teuthology.orchestra.run.smithi098.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-10-06T05:05:53.987 INFO:teuthology.orchestra.run.smithi098.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-10-06T05:05:53.987 INFO:teuthology.orchestra.run.smithi098.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-10-06T05:05:53.987 INFO:teuthology.orchestra.run.smithi098.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-10-06T05:05:53.987 INFO:teuthology.orchestra.run.smithi098.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-10-06T05:05:53.987 INFO:teuthology.orchestra.run.smithi098.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-10-06T05:05:53.988 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-fuse-2:17.2.6-1274.g68e53a4a.el8.x86_64 2023-10-06T05:05:53.988 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-mirror-2:17.2.6-1274.g68e53a4a.el8.x86_64 2023-10-06T05:05:53.988 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-nbd-2:17.2.6-1274.g68e53a4a.el8.x86_64 2023-10-06T05:05:53.988 INFO:teuthology.orchestra.run.smithi098.stdout: socat-1.7.3.3-2.el8.x86_64 2023-10-06T05:05:53.989 INFO:teuthology.orchestra.run.smithi098.stdout: thrift-0.13.0-2.el8.x86_64 2023-10-06T05:05:53.989 INFO:teuthology.orchestra.run.smithi098.stdout: xfsprogs-devel-5.0.0-8.el8.x86_64 2023-10-06T05:05:53.989 INFO:teuthology.orchestra.run.smithi098.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-10-06T05:05:53.989 INFO:teuthology.orchestra.run.smithi098.stdout: zlib-devel-1.2.11-17.el8.x86_64 2023-10-06T05:05:53.990 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:05:53.990 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2023-10-06T05:05:54.313 DEBUG:teuthology.parallel:result is None 2023-10-06T05:05:54.835 INFO:teuthology.orchestra.run.smithi174.stdout:Installed products updated. 2023-10-06T05:05:54.995 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-10-06T05:05:54.995 INFO:teuthology.orchestra.run.smithi174.stdout:Upgraded: 2023-10-06T05:05:54.995 INFO:teuthology.orchestra.run.smithi174.stdout: librados2-2:17.2.6-1274.g68e53a4a.el8.x86_64 2023-10-06T05:05:54.995 INFO:teuthology.orchestra.run.smithi174.stdout: librbd1-2:17.2.6-1274.g68e53a4a.el8.x86_64 2023-10-06T05:05:54.995 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2023-10-06T05:05:54.995 INFO:teuthology.orchestra.run.smithi174.stdout: automake-1.16.1-7.el8.noarch 2023-10-06T05:05:54.995 INFO:teuthology.orchestra.run.smithi174.stdout: bison-3.0.4-10.el8.x86_64 2023-10-06T05:05:54.996 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-2:17.2.6-1274.g68e53a4a.el8.x86_64 2023-10-06T05:05:54.996 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-base-2:17.2.6-1274.g68e53a4a.el8.x86_64 2023-10-06T05:05:54.996 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-common-2:17.2.6-1274.g68e53a4a.el8.x86_64 2023-10-06T05:05:54.996 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-fuse-2:17.2.6-1274.g68e53a4a.el8.x86_64 2023-10-06T05:05:54.996 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-grafana-dashboards-2:17.2.6-1274.g68e53a4a.el8.noarch 2023-10-06T05:05:54.996 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-immutable-object-cache-2:17.2.6-1274.g68e53a4a.el8.x86_64 2023-10-06T05:05:54.996 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mds-2:17.2.6-1274.g68e53a4a.el8.x86_64 2023-10-06T05:05:54.996 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-2:17.2.6-1274.g68e53a4a.el8.x86_64 2023-10-06T05:05:54.996 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-cephadm-2:17.2.6-1274.g68e53a4a.el8.noarch 2023-10-06T05:05:54.996 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-dashboard-2:17.2.6-1274.g68e53a4a.el8.noarch 2023-10-06T05:05:54.996 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-diskprediction-local-2:17.2.6-1274.g68e53a4a.el8.noarch 2023-10-06T05:05:54.996 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-modules-core-2:17.2.6-1274.g68e53a4a.el8.noarch 2023-10-06T05:05:54.997 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-rook-2:17.2.6-1274.g68e53a4a.el8.noarch 2023-10-06T05:05:54.997 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mon-2:17.2.6-1274.g68e53a4a.el8.x86_64 2023-10-06T05:05:54.997 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-osd-2:17.2.6-1274.g68e53a4a.el8.x86_64 2023-10-06T05:05:54.997 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-prometheus-alerts-2:17.2.6-1274.g68e53a4a.el8.noarch 2023-10-06T05:05:54.997 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-radosgw-2:17.2.6-1274.g68e53a4a.el8.x86_64 2023-10-06T05:05:54.997 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-selinux-2:17.2.6-1274.g68e53a4a.el8.x86_64 2023-10-06T05:05:54.997 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-test-2:17.2.6-1274.g68e53a4a.el8.x86_64 2023-10-06T05:05:54.997 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-volume-2:17.2.6-1274.g68e53a4a.el8.noarch 2023-10-06T05:05:54.997 INFO:teuthology.orchestra.run.smithi174.stdout: cephadm-2:17.2.6-1274.g68e53a4a.el8.noarch 2023-10-06T05:05:54.997 INFO:teuthology.orchestra.run.smithi174.stdout: cephfs-mirror-2:17.2.6-1274.g68e53a4a.el8.x86_64 2023-10-06T05:05:54.997 INFO:teuthology.orchestra.run.smithi174.stdout: cephfs-top-2:17.2.6-1274.g68e53a4a.el8.noarch 2023-10-06T05:05:54.997 INFO:teuthology.orchestra.run.smithi174.stdout: elfutils-libelf-devel-0.182-3.el8.x86_64 2023-10-06T05:05:54.998 INFO:teuthology.orchestra.run.smithi174.stdout: flex-2.6.1-9.el8.x86_64 2023-10-06T05:05:54.998 INFO:teuthology.orchestra.run.smithi174.stdout: fuse-2.9.7-12.el8.x86_64 2023-10-06T05:05:54.998 INFO:teuthology.orchestra.run.smithi174.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-10-06T05:05:54.998 INFO:teuthology.orchestra.run.smithi174.stdout: jq-1.5-12.el8.x86_64 2023-10-06T05:05:54.998 INFO:teuthology.orchestra.run.smithi174.stdout: keyutils-libs-devel-1.5.10-6.el8.x86_64 2023-10-06T05:05:54.998 INFO:teuthology.orchestra.run.smithi174.stdout: krb5-devel-1.18.2-8.3.el8_4.x86_64 2023-10-06T05:05:54.998 INFO:teuthology.orchestra.run.smithi174.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-10-06T05:05:54.998 INFO:teuthology.orchestra.run.smithi174.stdout: libcephfs-devel-2:17.2.6-1274.g68e53a4a.el8.x86_64 2023-10-06T05:05:54.998 INFO:teuthology.orchestra.run.smithi174.stdout: libcephfs2-2:17.2.6-1274.g68e53a4a.el8.x86_64 2023-10-06T05:05:54.998 INFO:teuthology.orchestra.run.smithi174.stdout: libcephsqlite-2:17.2.6-1274.g68e53a4a.el8.x86_64 2023-10-06T05:05:54.998 INFO:teuthology.orchestra.run.smithi174.stdout: libcom_err-devel-1.45.6-1.el8.x86_64 2023-10-06T05:05:54.999 INFO:teuthology.orchestra.run.smithi174.stdout: libgfortran-8.4.1-1.el8.x86_64 2023-10-06T05:05:54.999 INFO:teuthology.orchestra.run.smithi174.stdout: liboath-2.6.2-3.el8.x86_64 2023-10-06T05:05:54.999 INFO:teuthology.orchestra.run.smithi174.stdout: libquadmath-8.4.1-1.el8.x86_64 2023-10-06T05:05:54.999 INFO:teuthology.orchestra.run.smithi174.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-10-06T05:05:54.999 INFO:teuthology.orchestra.run.smithi174.stdout: librados-devel-2:17.2.6-1274.g68e53a4a.el8.x86_64 2023-10-06T05:05:54.999 INFO:teuthology.orchestra.run.smithi174.stdout: libradosstriper1-2:17.2.6-1274.g68e53a4a.el8.x86_64 2023-10-06T05:05:54.999 INFO:teuthology.orchestra.run.smithi174.stdout: librdkafka-0.11.4-1.el8.x86_64 2023-10-06T05:05:54.999 INFO:teuthology.orchestra.run.smithi174.stdout: librgw2-2:17.2.6-1274.g68e53a4a.el8.x86_64 2023-10-06T05:05:54.999 INFO:teuthology.orchestra.run.smithi174.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-10-06T05:05:54.999 INFO:teuthology.orchestra.run.smithi174.stdout: libsepol-devel-2.9-2.el8.x86_64 2023-10-06T05:05:54.999 INFO:teuthology.orchestra.run.smithi174.stdout: libtool-2.4.6-25.el8.x86_64 2023-10-06T05:05:55.000 INFO:teuthology.orchestra.run.smithi174.stdout: libunwind-1.3.1-3.el8.x86_64 2023-10-06T05:05:55.000 INFO:teuthology.orchestra.run.smithi174.stdout: libuuid-devel-2.32.1-27.el8.x86_64 2023-10-06T05:05:55.000 INFO:teuthology.orchestra.run.smithi174.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-10-06T05:05:55.000 INFO:teuthology.orchestra.run.smithi174.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-10-06T05:05:55.000 INFO:teuthology.orchestra.run.smithi174.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-10-06T05:05:55.000 INFO:teuthology.orchestra.run.smithi174.stdout: openblas-0.3.12-1.el8.x86_64 2023-10-06T05:05:55.000 INFO:teuthology.orchestra.run.smithi174.stdout: openblas-threads-0.3.12-1.el8.x86_64 2023-10-06T05:05:55.000 INFO:teuthology.orchestra.run.smithi174.stdout: openssl-devel-1:1.1.1g-15.el8_3.x86_64 2023-10-06T05:05:55.000 INFO:teuthology.orchestra.run.smithi174.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-10-06T05:05:55.000 INFO:teuthology.orchestra.run.smithi174.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-10-06T05:05:55.000 INFO:teuthology.orchestra.run.smithi174.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-10-06T05:05:55.000 INFO:teuthology.orchestra.run.smithi174.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-10-06T05:05:55.001 INFO:teuthology.orchestra.run.smithi174.stdout: python3-babel-2.5.1-5.el8.noarch 2023-10-06T05:05:55.001 INFO:teuthology.orchestra.run.smithi174.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-10-06T05:05:55.001 INFO:teuthology.orchestra.run.smithi174.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-10-06T05:05:55.001 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-10-06T05:05:55.001 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ceph-argparse-2:17.2.6-1274.g68e53a4a.el8.x86_64 2023-10-06T05:05:55.001 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ceph-common-2:17.2.6-1274.g68e53a4a.el8.x86_64 2023-10-06T05:05:55.001 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cephfs-2:17.2.6-1274.g68e53a4a.el8.x86_64 2023-10-06T05:05:55.001 INFO:teuthology.orchestra.run.smithi174.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-10-06T05:05:55.001 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-10-06T05:05:55.001 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-10-06T05:05:55.001 INFO:teuthology.orchestra.run.smithi174.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-10-06T05:05:55.001 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-6.2-6.el8.noarch 2023-10-06T05:05:55.002 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-10-06T05:05:55.002 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2023-10-06T05:05:55.002 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-10-06T05:05:55.002 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-10-06T05:05:55.002 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-10-06T05:05:55.002 INFO:teuthology.orchestra.run.smithi174.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-10-06T05:05:55.002 INFO:teuthology.orchestra.run.smithi174.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-10-06T05:05:55.002 INFO:teuthology.orchestra.run.smithi174.stdout: python3-mako-1.0.6-13.el8.noarch 2023-10-06T05:05:55.002 INFO:teuthology.orchestra.run.smithi174.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-10-06T05:05:55.002 INFO:teuthology.orchestra.run.smithi174.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-10-06T05:05:55.002 INFO:teuthology.orchestra.run.smithi174.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-10-06T05:05:55.003 INFO:teuthology.orchestra.run.smithi174.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2023-10-06T05:05:55.003 INFO:teuthology.orchestra.run.smithi174.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2023-10-06T05:05:55.003 INFO:teuthology.orchestra.run.smithi174.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-10-06T05:05:55.003 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-10-06T05:05:55.003 INFO:teuthology.orchestra.run.smithi174.stdout: python3-portend-2.6-1.el8.noarch 2023-10-06T05:05:55.003 INFO:teuthology.orchestra.run.smithi174.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-10-06T05:05:55.003 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-10-06T05:05:55.003 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-10-06T05:05:55.003 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pytz-2017.2-9.el8.noarch 2023-10-06T05:05:55.003 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rados-2:17.2.6-1274.g68e53a4a.el8.x86_64 2023-10-06T05:05:55.003 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rbd-2:17.2.6-1274.g68e53a4a.el8.x86_64 2023-10-06T05:05:55.003 INFO:teuthology.orchestra.run.smithi174.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-10-06T05:05:55.004 INFO:teuthology.orchestra.run.smithi174.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-10-06T05:05:55.004 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rgw-2:17.2.6-1274.g68e53a4a.el8.x86_64 2023-10-06T05:05:55.004 INFO:teuthology.orchestra.run.smithi174.stdout: python3-routes-2.4.1-12.el8.noarch 2023-10-06T05:05:55.004 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rsa-4.9-2.el8.noarch 2023-10-06T05:05:55.004 INFO:teuthology.orchestra.run.smithi174.stdout: python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7.x86_64 2023-10-06T05:05:55.004 INFO:teuthology.orchestra.run.smithi174.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-10-06T05:05:55.004 INFO:teuthology.orchestra.run.smithi174.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-10-06T05:05:55.004 INFO:teuthology.orchestra.run.smithi174.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-10-06T05:05:55.004 INFO:teuthology.orchestra.run.smithi174.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-10-06T05:05:55.004 INFO:teuthology.orchestra.run.smithi174.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-10-06T05:05:55.004 INFO:teuthology.orchestra.run.smithi174.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-10-06T05:05:55.005 INFO:teuthology.orchestra.run.smithi174.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-10-06T05:05:55.005 INFO:teuthology.orchestra.run.smithi174.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-10-06T05:05:55.005 INFO:teuthology.orchestra.run.smithi174.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-10-06T05:05:55.005 INFO:teuthology.orchestra.run.smithi174.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-10-06T05:05:55.005 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-fuse-2:17.2.6-1274.g68e53a4a.el8.x86_64 2023-10-06T05:05:55.005 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-mirror-2:17.2.6-1274.g68e53a4a.el8.x86_64 2023-10-06T05:05:55.005 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-nbd-2:17.2.6-1274.g68e53a4a.el8.x86_64 2023-10-06T05:05:55.005 INFO:teuthology.orchestra.run.smithi174.stdout: socat-1.7.3.3-2.el8.x86_64 2023-10-06T05:05:55.005 INFO:teuthology.orchestra.run.smithi174.stdout: thrift-0.13.0-2.el8.x86_64 2023-10-06T05:05:55.005 INFO:teuthology.orchestra.run.smithi174.stdout: xfsprogs-devel-5.0.0-8.el8.x86_64 2023-10-06T05:05:55.005 INFO:teuthology.orchestra.run.smithi174.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-10-06T05:05:55.006 INFO:teuthology.orchestra.run.smithi174.stdout: zlib-devel-1.2.11-17.el8.x86_64 2023-10-06T05:05:55.006 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-10-06T05:05:55.006 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2023-10-06T05:05:55.269 DEBUG:teuthology.parallel:result is None 2023-10-06T05:05:55.269 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=68e53a4a7d4ae06d4fc1a0a7cfef840d1d942564 2023-10-06T05:05:55.409 DEBUG:teuthology.orchestra.run.smithi098:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-10-06T05:05:55.432 INFO:teuthology.orchestra.run.smithi098.stdout:17.2.6-1274.g68e53a4a.el8 2023-10-06T05:05:55.432 INFO:teuthology.packaging:The installed version of ceph is 17.2.6-1274.g68e53a4a.el8 2023-10-06T05:05:55.432 INFO:teuthology.task.install:The correct ceph version 17.2.6-1274.g68e53a4a is installed. 2023-10-06T05:05:55.433 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=68e53a4a7d4ae06d4fc1a0a7cfef840d1d942564 2023-10-06T05:05:55.568 DEBUG:teuthology.orchestra.run.smithi174:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-10-06T05:05:55.589 INFO:teuthology.orchestra.run.smithi174.stdout:17.2.6-1274.g68e53a4a.el8 2023-10-06T05:05:55.590 INFO:teuthology.packaging:The installed version of ceph is 17.2.6-1274.g68e53a4a.el8 2023-10-06T05:05:55.590 INFO:teuthology.task.install:The correct ceph version 17.2.6-1274.g68e53a4a is installed. 2023-10-06T05:05:55.590 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-10-06T05:05:55.591 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-10-06T05:05:55.591 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-06T05:05:55.632 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2023-10-06T05:05:55.632 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-06T05:05:55.671 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-10-06T05:05:55.671 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-10-06T05:05:55.671 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/daemon-helper 2023-10-06T05:05:55.698 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-06T05:05:55.763 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2023-10-06T05:05:55.763 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/usr/bin/daemon-helper 2023-10-06T05:05:55.787 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-06T05:05:55.853 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-10-06T05:05:55.853 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-10-06T05:05:55.853 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-06T05:05:55.879 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-06T05:05:55.943 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2023-10-06T05:05:55.943 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-06T05:05:55.967 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-06T05:05:56.031 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-10-06T05:05:56.032 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-10-06T05:05:56.032 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/stdin-killer 2023-10-06T05:05:56.056 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-06T05:05:56.120 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2023-10-06T05:05:56.120 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/usr/bin/stdin-killer 2023-10-06T05:05:56.144 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-06T05:05:56.208 INFO:teuthology.run_tasks:Running task ceph... 2023-10-06T05:05:56.283 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-10-06T05:05:56.283 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 777 /var/log/ceph 2023-10-06T05:05:56.286 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod 777 /var/log/ceph 2023-10-06T05:05:56.309 INFO:tasks.ceph:Disabling ceph logrotate... 2023-10-06T05:05:56.309 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-06T05:05:56.350 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-06T05:05:56.374 INFO:tasks.ceph:Creating extra log directories... 2023-10-06T05:05:56.374 DEBUG:teuthology.orchestra.run.smithi098:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-06T05:05:56.414 DEBUG:teuthology.orchestra.run.smithi174:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-06T05:05:56.444 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2023-10-06T05:05:56.445 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2023-10-06T05:05:56.445 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2023-10-06T05:05:56.445 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-10-06T05:05:56.445 DEBUG:teuthology.orchestra.run.smithi098:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-10-06T05:05:56.491 DEBUG:teuthology.orchestra.run.smithi098:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-10-06T05:05:56.604 DEBUG:teuthology.orchestra.run.smithi098:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-10-06T05:05:56.628 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2023-10-06T05:05:56.628 DEBUG:teuthology.orchestra.run.smithi174:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-10-06T05:05:56.643 DEBUG:teuthology.orchestra.run.smithi174:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-10-06T05:05:56.756 DEBUG:teuthology.orchestra.run.smithi174:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-10-06T05:05:56.787 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-10-06T05:05:56.788 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-10-06T05:05:56.788 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-10-05_23:54:57-fs-wip-yuri3-testing-2023-10-05-0943-quincy-distro-default-smithi/7412314', 'branch': 'wip-yuri3-testing-2023-10-05-0943-quincy', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-readahead}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7412314', 'kernel': {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'yuriw-2023-10-05_23:54:57-fs-wip-yuri3-testing-2023-10-05-0943-quincy-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'rhel', 'os_version': '8.4', 'overrides': {'admin_socket': {'branch': 'wip-yuri3-testing-2023-10-05-0943-quincy'}, 'ceph': {'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '68e53a4a7d4ae06d4fc1a0a7cfef840d1d942564'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '68e53a4a7d4ae06d4fc1a0a7cfef840d1d942564'}}, 'kclient': {'syntax': 'v2'}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri3-testing-2023-10-05-0943-quincy', 'sha1': '68e53a4a7d4ae06d4fc1a0a7cfef840d1d942564'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 4002, 'sha1': '68e53a4a7d4ae06d4fc1a0a7cfef840d1d942564', 'sleep_before_teardown': 0, 'subset': '111/12000', 'suite': 'fs', 'suite_branch': 'wip-yuri3-testing-2023-10-05-0943-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_68e53a4a7d4ae06d4fc1a0a7cfef840d1d942564/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '68e53a4a7d4ae06d4fc1a0a7cfef840d1d942564', 'targets': {'smithi098.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAdLN1jqITOyOs2/LG9xBqbeR1dU57UfESY/Js/od2mIfqcEEbswvtH7X0gj0zBZVDFXiTLyYoFCX8PqhzYepWI=', 'smithi174.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKTO+hRfafQJmPn7d061Qdv6KXPKph9+gBvigvYovlxzFKbU3ueuwcnmiJKd4IHyY+NWJGJPMku58GXgpv6R/oQ='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'flavor': 'default', 'sha1': '68e53a4a7d4ae06d4fc1a0a7cfef840d1d942564'}}, {'ceph': None}, {'kclient': None}, {'cephfs_test_runner': {'fail_on_skip': False, 'modules': ['tasks.cephfs.test_readahead']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': '54e62bcbac4e53d9685e08328b790d3b20d71cae', 'timestamp': '2023-10-05_23:54:57', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-10-06T05:05:56.789 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-06T05:05:56.791 DEBUG:teuthology.orchestra.run.smithi174:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-06T05:05:56.835 DEBUG:teuthology.orchestra.run.smithi098:> sudo install -d -m0777 -- /var/run/ceph 2023-10-06T05:05:56.845 DEBUG:teuthology.orchestra.run.smithi174:> sudo install -d -m0777 -- /var/run/ceph 2023-10-06T05:05:56.900 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-10-06T05:05:56.900 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/scratch_devs of=/dev/stdout 2023-10-06T05:05:56.921 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-06T05:05:56.922 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_1 2023-10-06T05:05:56.977 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-06T05:05:56.977 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-06T05:05:56.977 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 6h/6d Inode: 215710 Links: 1 2023-10-06T05:05:56.977 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-06T05:05:56.977 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2023-10-06T05:05:56.977 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2023-10-06 05:05:49.183541125 +0000 2023-10-06T05:05:56.977 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2023-10-06 05:03:15.226183120 +0000 2023-10-06T05:05:56.977 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2023-10-06 05:03:15.226183120 +0000 2023-10-06T05:05:56.977 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: - 2023-10-06T05:05:56.978 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-06T05:05:57.040 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2023-10-06T05:05:57.040 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2023-10-06T05:05:57.040 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.00027289 s, 1.9 MB/s 2023-10-06T05:05:57.041 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-06T05:05:57.097 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_2 2023-10-06T05:05:57.155 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-06T05:05:57.157 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-06T05:05:57.157 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 6h/6d Inode: 215802 Links: 1 2023-10-06T05:05:57.157 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-06T05:05:57.157 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2023-10-06T05:05:57.157 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2023-10-06 05:05:49.183541125 +0000 2023-10-06T05:05:57.158 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2023-10-06 05:03:15.701177126 +0000 2023-10-06T05:05:57.159 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2023-10-06 05:03:15.701177126 +0000 2023-10-06T05:05:57.159 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: - 2023-10-06T05:05:57.159 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-06T05:05:57.228 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2023-10-06T05:05:57.229 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2023-10-06T05:05:57.229 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000203721 s, 2.5 MB/s 2023-10-06T05:05:57.230 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-06T05:05:57.288 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_3 2023-10-06T05:05:57.347 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-06T05:05:57.347 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-06T05:05:57.347 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 6h/6d Inode: 219556 Links: 1 2023-10-06T05:05:57.347 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-06T05:05:57.347 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2023-10-06T05:05:57.347 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2023-10-06 05:05:49.183541125 +0000 2023-10-06T05:05:57.347 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2023-10-06 05:03:16.218170601 +0000 2023-10-06T05:05:57.347 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2023-10-06 05:03:16.218170601 +0000 2023-10-06T05:05:57.347 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: - 2023-10-06T05:05:57.348 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-06T05:05:57.415 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2023-10-06T05:05:57.415 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2023-10-06T05:05:57.415 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000223613 s, 2.3 MB/s 2023-10-06T05:05:57.415 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-06T05:05:57.474 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_4 2023-10-06T05:05:57.532 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-06T05:05:57.533 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-06T05:05:57.533 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 6h/6d Inode: 221107 Links: 1 2023-10-06T05:05:57.533 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-06T05:05:57.533 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2023-10-06T05:05:57.533 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2023-10-06 05:05:49.183541125 +0000 2023-10-06T05:05:57.533 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2023-10-06 05:03:16.694164594 +0000 2023-10-06T05:05:57.533 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2023-10-06 05:03:16.694164594 +0000 2023-10-06T05:05:57.533 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: - 2023-10-06T05:05:57.533 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-06T05:05:57.598 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2023-10-06T05:05:57.598 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2023-10-06T05:05:57.598 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000178199 s, 2.9 MB/s 2023-10-06T05:05:57.599 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-06T05:05:57.654 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-06T05:05:57.655 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2023-10-06T05:05:57.655 DEBUG:teuthology.orchestra.run.smithi174:> dd if=/scratch_devs of=/dev/stdout 2023-10-06T05:05:57.671 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-06T05:05:57.671 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/vg_nvme/lv_1 2023-10-06T05:05:57.727 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-06T05:05:57.727 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-06T05:05:57.728 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 6h/6d Inode: 215803 Links: 1 2023-10-06T05:05:57.728 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-06T05:05:57.728 INFO:teuthology.orchestra.run.smithi174.stdout:Context: system_u:object_r:device_t:s0 2023-10-06T05:05:57.728 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2023-10-06 05:05:49.974650055 +0000 2023-10-06T05:05:57.728 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2023-10-06 05:03:15.218873918 +0000 2023-10-06T05:05:57.728 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2023-10-06 05:03:15.218873918 +0000 2023-10-06T05:05:57.728 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: - 2023-10-06T05:05:57.728 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-06T05:05:57.793 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2023-10-06T05:05:57.793 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2023-10-06T05:05:57.793 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.000275437 s, 1.9 MB/s 2023-10-06T05:05:57.794 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-06T05:05:57.852 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/vg_nvme/lv_2 2023-10-06T05:05:57.908 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-06T05:05:57.909 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-06T05:05:57.909 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 6h/6d Inode: 219441 Links: 1 2023-10-06T05:05:57.909 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-06T05:05:57.909 INFO:teuthology.orchestra.run.smithi174.stdout:Context: system_u:object_r:device_t:s0 2023-10-06T05:05:57.909 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2023-10-06 05:05:49.974650055 +0000 2023-10-06T05:05:57.909 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2023-10-06 05:03:15.719867560 +0000 2023-10-06T05:05:57.909 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2023-10-06 05:03:15.719867560 +0000 2023-10-06T05:05:57.909 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: - 2023-10-06T05:05:57.910 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-06T05:05:57.972 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2023-10-06T05:05:57.972 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2023-10-06T05:05:57.973 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.000224936 s, 2.3 MB/s 2023-10-06T05:05:57.974 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-06T05:05:58.030 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/vg_nvme/lv_3 2023-10-06T05:05:58.085 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-06T05:05:58.085 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-06T05:05:58.086 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 6h/6d Inode: 219057 Links: 1 2023-10-06T05:05:58.086 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-06T05:05:58.086 INFO:teuthology.orchestra.run.smithi174.stdout:Context: system_u:object_r:device_t:s0 2023-10-06T05:05:58.086 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2023-10-06 05:05:49.975650043 +0000 2023-10-06T05:05:58.086 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2023-10-06 05:03:16.202861432 +0000 2023-10-06T05:05:58.086 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2023-10-06 05:03:16.202861432 +0000 2023-10-06T05:05:58.086 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: - 2023-10-06T05:05:58.087 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-06T05:05:58.149 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2023-10-06T05:05:58.149 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2023-10-06T05:05:58.149 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.000246173 s, 2.1 MB/s 2023-10-06T05:05:58.150 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-06T05:05:58.205 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/vg_nvme/lv_4 2023-10-06T05:05:58.260 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-06T05:05:58.260 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-06T05:05:58.260 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 6h/6d Inode: 219595 Links: 1 2023-10-06T05:05:58.260 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-06T05:05:58.260 INFO:teuthology.orchestra.run.smithi174.stdout:Context: system_u:object_r:device_t:s0 2023-10-06T05:05:58.261 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2023-10-06 05:05:49.975650043 +0000 2023-10-06T05:05:58.261 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2023-10-06 05:03:16.711854971 +0000 2023-10-06T05:05:58.261 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2023-10-06 05:03:16.711854971 +0000 2023-10-06T05:05:58.261 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: - 2023-10-06T05:05:58.261 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-06T05:05:58.322 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2023-10-06T05:05:58.322 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2023-10-06T05:05:58.322 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.000212965 s, 2.4 MB/s 2023-10-06T05:05:58.323 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-06T05:05:58.378 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-06T05:05:58.379 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi098.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi174.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-10-06T05:05:58.379 INFO:tasks.ceph:Generating config... 2023-10-06T05:05:58.380 INFO:tasks.ceph:[client] client mount timeout = 600 2023-10-06T05:05:58.380 INFO:tasks.ceph:[client] debug client = 20 2023-10-06T05:05:58.380 INFO:tasks.ceph:[client] debug ms = 1 2023-10-06T05:05:58.380 INFO:tasks.ceph:[client] pid file = 2023-10-06T05:05:58.380 INFO:tasks.ceph:[client] rados mon op timeout = 900 2023-10-06T05:05:58.380 INFO:tasks.ceph:[client] rados osd op timeout = 900 2023-10-06T05:05:58.380 INFO:tasks.ceph:[global] ms die on skipped message = False 2023-10-06T05:05:58.380 INFO:tasks.ceph:[mds] debug mds = 20 2023-10-06T05:05:58.381 INFO:tasks.ceph:[mds] debug ms = 1 2023-10-06T05:05:58.381 INFO:tasks.ceph:[mds] mds debug frag = True 2023-10-06T05:05:58.381 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2023-10-06T05:05:58.381 INFO:tasks.ceph:[mds] mds op complaint time = 180 2023-10-06T05:05:58.381 INFO:tasks.ceph:[mds] mds verify scatter = True 2023-10-06T05:05:58.381 INFO:tasks.ceph:[mds] osd op complaint time = 180 2023-10-06T05:05:58.381 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2023-10-06T05:05:58.381 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2023-10-06T05:05:58.381 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-10-06T05:05:58.381 INFO:tasks.ceph:[mgr] debug ms = 1 2023-10-06T05:05:58.381 INFO:tasks.ceph:[mon] debug mon = 20 2023-10-06T05:05:58.381 INFO:tasks.ceph:[mon] debug ms = 1 2023-10-06T05:05:58.382 INFO:tasks.ceph:[mon] debug paxos = 20 2023-10-06T05:05:58.382 INFO:tasks.ceph:[mon] mon op complaint time = 120 2023-10-06T05:05:58.382 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-10-06T05:05:58.382 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-10-06T05:05:58.382 INFO:tasks.ceph:[osd] debug bluefs = 20 2023-10-06T05:05:58.382 INFO:tasks.ceph:[osd] debug bluestore = 20 2023-10-06T05:05:58.382 INFO:tasks.ceph:[osd] debug ms = 1 2023-10-06T05:05:58.382 INFO:tasks.ceph:[osd] debug osd = 20 2023-10-06T05:05:58.382 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-10-06T05:05:58.382 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-10-06T05:05:58.382 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-10-06T05:05:58.382 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-10-06T05:05:58.383 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-10-06T05:05:58.383 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-10-06T05:05:58.383 INFO:tasks.ceph:[osd] osd op complaint time = 180 2023-10-06T05:05:58.383 INFO:tasks.ceph:Setting up mon.a... 2023-10-06T05:05:58.383 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-10-06T05:05:58.432 INFO:teuthology.orchestra.run.smithi098.stdout:creating /etc/ceph/ceph.keyring 2023-10-06T05:05:58.435 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-10-06T05:05:58.474 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-06T05:05:58.539 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.98'), ('mon.b', '172.21.15.174'), ('mon.c', '[v2:172.21.15.174:3301,v1:172.21.15.174:6790]')] 2023-10-06T05:05:58.539 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.98,172.21.15.174,[v2:172.21.15.174:3301,v1:172.21.15.174:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2023-10-06T05:05:58.540 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-10-06T05:05:58.540 DEBUG:teuthology.orchestra.run.smithi098:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-10-06T05:05:58.593 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.98 --add b 172.21.15.174 --addv c '[v2:172.21.15.174:3301,v1:172.21.15.174:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2023-10-06T05:05:58.675 INFO:teuthology.orchestra.run.smithi098.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-10-06T05:05:58.675 INFO:teuthology.orchestra.run.smithi098.stdout:monmaptool: generated fsid f50c5df5-cad5-47a4-9bb1-8c791d4311ec 2023-10-06T05:05:58.675 INFO:teuthology.orchestra.run.smithi098.stdout:setting min_mon_release = octopus 2023-10-06T05:05:58.675 INFO:teuthology.orchestra.run.smithi098.stdout:epoch 0 2023-10-06T05:05:58.676 INFO:teuthology.orchestra.run.smithi098.stdout:fsid f50c5df5-cad5-47a4-9bb1-8c791d4311ec 2023-10-06T05:05:58.676 INFO:teuthology.orchestra.run.smithi098.stdout:last_changed 2023-10-06T05:05:58.674689+0000 2023-10-06T05:05:58.676 INFO:teuthology.orchestra.run.smithi098.stdout:created 2023-10-06T05:05:58.674689+0000 2023-10-06T05:05:58.676 INFO:teuthology.orchestra.run.smithi098.stdout:min_mon_release 15 (octopus) 2023-10-06T05:05:58.676 INFO:teuthology.orchestra.run.smithi098.stdout:election_strategy: 1 2023-10-06T05:05:58.676 INFO:teuthology.orchestra.run.smithi098.stdout:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2023-10-06T05:05:58.676 INFO:teuthology.orchestra.run.smithi098.stdout:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2023-10-06T05:05:58.676 INFO:teuthology.orchestra.run.smithi098.stdout:2: [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] mon.c 2023-10-06T05:05:58.677 INFO:teuthology.orchestra.run.smithi098.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-10-06T05:05:58.678 DEBUG:teuthology.orchestra.run.smithi098:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-10-06T05:05:58.732 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID f50c5df5-cad5-47a4-9bb1-8c791d4311ec... 2023-10-06T05:05:58.734 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-06T05:05:58.775 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-06T05:05:58.815 INFO:teuthology.orchestra.run.smithi098.stdout:[global] 2023-10-06T05:05:58.816 INFO:teuthology.orchestra.run.smithi098.stdout: chdir = "" 2023-10-06T05:05:58.816 INFO:teuthology.orchestra.run.smithi098.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-06T05:05:58.816 INFO:teuthology.orchestra.run.smithi098.stdout: auth supported = cephx 2023-10-06T05:05:58.816 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:05:58.816 INFO:teuthology.orchestra.run.smithi098.stdout: filestore xattr use omap = true 2023-10-06T05:05:58.816 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:05:58.817 INFO:teuthology.orchestra.run.smithi098.stdout: mon clock drift allowed = 1.000 2023-10-06T05:05:58.817 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:05:58.817 INFO:teuthology.orchestra.run.smithi098.stdout: osd crush chooseleaf type = 0 2023-10-06T05:05:58.818 INFO:teuthology.orchestra.run.smithi098.stdout: auth debug = true 2023-10-06T05:05:58.818 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:05:58.819 INFO:teuthology.orchestra.run.smithi098.stdout: ms die on old message = true 2023-10-06T05:05:58.819 INFO:teuthology.orchestra.run.smithi098.stdout: ms die on bug = true 2023-10-06T05:05:58.819 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:05:58.819 INFO:teuthology.orchestra.run.smithi098.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-06T05:05:58.819 INFO:teuthology.orchestra.run.smithi098.stdout: mon pg warn max object skew = 0 2023-10-06T05:05:58.819 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:05:58.819 INFO:teuthology.orchestra.run.smithi098.stdout: # disable pg_autoscaler by default for new pools 2023-10-06T05:05:58.819 INFO:teuthology.orchestra.run.smithi098.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-06T05:05:58.820 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:05:58.820 INFO:teuthology.orchestra.run.smithi098.stdout: osd pool default size = 2 2023-10-06T05:05:58.820 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:05:58.820 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd allow primary affinity = true 2023-10-06T05:05:58.820 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd allow pg remap = true 2023-10-06T05:05:58.820 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on legacy crush tunables = false 2023-10-06T05:05:58.820 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on crush straw calc version zero = false 2023-10-06T05:05:58.820 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on no sortbitwise = false 2023-10-06T05:05:58.820 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on osd down out interval zero = false 2023-10-06T05:05:58.820 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on too few osds = false 2023-10-06T05:05:58.821 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-06T05:05:58.821 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-06T05:05:58.821 INFO:teuthology.orchestra.run.smithi098.stdout: mon_allow_pool_size_one = true 2023-10-06T05:05:58.821 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:05:58.821 INFO:teuthology.orchestra.run.smithi098.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-06T05:05:58.821 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:05:58.821 INFO:teuthology.orchestra.run.smithi098.stdout: osd default data pool replay window = 5 2023-10-06T05:05:58.821 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:05:58.821 INFO:teuthology.orchestra.run.smithi098.stdout: mon allow pool delete = true 2023-10-06T05:05:58.822 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:05:58.822 INFO:teuthology.orchestra.run.smithi098.stdout: mon cluster log file level = debug 2023-10-06T05:05:58.822 INFO:teuthology.orchestra.run.smithi098.stdout: debug asserts on shutdown = true 2023-10-06T05:05:58.822 INFO:teuthology.orchestra.run.smithi098.stdout: mon health detail to clog = false 2023-10-06T05:05:58.822 INFO:teuthology.orchestra.run.smithi098.stdout: mon host = "172.21.15.98,172.21.15.174,[v2:172.21.15.174:3301,v1:172.21.15.174:6790]" 2023-10-06T05:05:58.822 INFO:teuthology.orchestra.run.smithi098.stdout: ms die on skipped message = False 2023-10-06T05:05:58.822 INFO:teuthology.orchestra.run.smithi098.stdout: fsid = f50c5df5-cad5-47a4-9bb1-8c791d4311ec 2023-10-06T05:05:58.822 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:05:58.822 INFO:teuthology.orchestra.run.smithi098.stdout:[osd] 2023-10-06T05:05:58.823 INFO:teuthology.orchestra.run.smithi098.stdout: osd journal size = 100 2023-10-06T05:05:58.823 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:05:58.823 INFO:teuthology.orchestra.run.smithi098.stdout: osd scrub load threshold = 5.0 2023-10-06T05:05:58.823 INFO:teuthology.orchestra.run.smithi098.stdout: osd scrub max interval = 600 2023-10-06T05:05:58.823 INFO:teuthology.orchestra.run.smithi098.stdout: osd mclock profile = high_recovery_ops 2023-10-06T05:05:58.823 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:05:58.823 INFO:teuthology.orchestra.run.smithi098.stdout: osd recover clone overlap = true 2023-10-06T05:05:58.823 INFO:teuthology.orchestra.run.smithi098.stdout: osd recovery max chunk = 1048576 2023-10-06T05:05:58.823 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:05:58.823 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug shutdown = true 2023-10-06T05:05:58.824 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug op order = true 2023-10-06T05:05:58.824 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug verify stray on activate = true 2023-10-06T05:05:58.824 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:05:58.824 INFO:teuthology.orchestra.run.smithi098.stdout: osd open classes on start = true 2023-10-06T05:05:58.824 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug pg log writeout = true 2023-10-06T05:05:58.824 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:05:58.824 INFO:teuthology.orchestra.run.smithi098.stdout: osd deep scrub update digest min age = 30 2023-10-06T05:05:58.824 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:05:58.825 INFO:teuthology.orchestra.run.smithi098.stdout: osd map max advance = 10 2023-10-06T05:05:58.825 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:05:58.825 INFO:teuthology.orchestra.run.smithi098.stdout: journal zero on create = true 2023-10-06T05:05:58.825 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:05:58.825 INFO:teuthology.orchestra.run.smithi098.stdout: filestore ondisk finisher threads = 3 2023-10-06T05:05:58.825 INFO:teuthology.orchestra.run.smithi098.stdout: filestore apply finisher threads = 3 2023-10-06T05:05:58.825 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:05:58.825 INFO:teuthology.orchestra.run.smithi098.stdout: bdev debug aio = true 2023-10-06T05:05:58.825 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug misdirected ops = true 2023-10-06T05:05:58.825 INFO:teuthology.orchestra.run.smithi098.stdout: bluestore block size = 96636764160 2023-10-06T05:05:58.826 INFO:teuthology.orchestra.run.smithi098.stdout: bluestore fsck on mount = True 2023-10-06T05:05:58.826 INFO:teuthology.orchestra.run.smithi098.stdout: debug bluefs = 20 2023-10-06T05:05:58.826 INFO:teuthology.orchestra.run.smithi098.stdout: debug bluestore = 20 2023-10-06T05:05:58.826 INFO:teuthology.orchestra.run.smithi098.stdout: debug ms = 1 2023-10-06T05:05:58.826 INFO:teuthology.orchestra.run.smithi098.stdout: debug osd = 20 2023-10-06T05:05:58.826 INFO:teuthology.orchestra.run.smithi098.stdout: debug rocksdb = 10 2023-10-06T05:05:58.826 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd backfillfull_ratio = 0.85 2023-10-06T05:05:58.826 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd full ratio = 0.9 2023-10-06T05:05:58.826 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd nearfull ratio = 0.8 2023-10-06T05:05:58.826 INFO:teuthology.orchestra.run.smithi098.stdout: osd failsafe full ratio = 0.95 2023-10-06T05:05:58.827 INFO:teuthology.orchestra.run.smithi098.stdout: osd objectstore = bluestore 2023-10-06T05:05:58.827 INFO:teuthology.orchestra.run.smithi098.stdout: osd op complaint time = 180 2023-10-06T05:05:58.827 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:05:58.827 INFO:teuthology.orchestra.run.smithi098.stdout:[mgr] 2023-10-06T05:05:58.827 INFO:teuthology.orchestra.run.smithi098.stdout: debug ms = 1 2023-10-06T05:05:58.827 INFO:teuthology.orchestra.run.smithi098.stdout: debug mgr = 20 2023-10-06T05:05:58.827 INFO:teuthology.orchestra.run.smithi098.stdout: debug mon = 20 2023-10-06T05:05:58.827 INFO:teuthology.orchestra.run.smithi098.stdout: debug auth = 20 2023-10-06T05:05:58.827 INFO:teuthology.orchestra.run.smithi098.stdout: mon reweight min pgs per osd = 4 2023-10-06T05:05:58.827 INFO:teuthology.orchestra.run.smithi098.stdout: mon reweight min bytes per osd = 10 2023-10-06T05:05:58.827 INFO:teuthology.orchestra.run.smithi098.stdout: mgr/telemetry/nag = false 2023-10-06T05:05:58.827 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:05:58.828 INFO:teuthology.orchestra.run.smithi098.stdout:[mon] 2023-10-06T05:05:58.828 INFO:teuthology.orchestra.run.smithi098.stdout: debug ms = 1 2023-10-06T05:05:58.829 INFO:teuthology.orchestra.run.smithi098.stdout: debug mon = 20 2023-10-06T05:05:58.830 INFO:teuthology.orchestra.run.smithi098.stdout: debug paxos = 20 2023-10-06T05:05:58.830 INFO:teuthology.orchestra.run.smithi098.stdout: debug auth = 20 2023-10-06T05:05:58.830 INFO:teuthology.orchestra.run.smithi098.stdout: mon data avail warn = 5 2023-10-06T05:05:58.830 INFO:teuthology.orchestra.run.smithi098.stdout: mon mgr mkfs grace = 240 2023-10-06T05:05:58.830 INFO:teuthology.orchestra.run.smithi098.stdout: mon reweight min pgs per osd = 4 2023-10-06T05:05:58.830 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd reporter subtree level = osd 2023-10-06T05:05:58.831 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd prime pg temp = true 2023-10-06T05:05:58.831 INFO:teuthology.orchestra.run.smithi098.stdout: mon reweight min bytes per osd = 10 2023-10-06T05:05:58.831 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:05:58.831 INFO:teuthology.orchestra.run.smithi098.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-06T05:05:58.831 INFO:teuthology.orchestra.run.smithi098.stdout: auth mon ticket ttl = 660 # 11m 2023-10-06T05:05:58.831 INFO:teuthology.orchestra.run.smithi098.stdout: auth service ticket ttl = 240 # 4m 2023-10-06T05:05:58.831 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:05:58.831 INFO:teuthology.orchestra.run.smithi098.stdout: # don't complain about insecure global_id in the test suite 2023-10-06T05:05:58.831 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-06T05:05:58.831 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-06T05:05:58.831 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:05:58.831 INFO:teuthology.orchestra.run.smithi098.stdout: # 1m isn't quite enough 2023-10-06T05:05:58.832 INFO:teuthology.orchestra.run.smithi098.stdout: mon_down_mkfs_grace = 2m 2023-10-06T05:05:58.832 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:05:58.832 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_filestore_osds = false 2023-10-06T05:05:58.832 INFO:teuthology.orchestra.run.smithi098.stdout: mon op complaint time = 120 2023-10-06T05:05:58.832 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:05:58.832 INFO:teuthology.orchestra.run.smithi098.stdout:[client] 2023-10-06T05:05:58.832 INFO:teuthology.orchestra.run.smithi098.stdout: rgw cache enabled = true 2023-10-06T05:05:58.832 INFO:teuthology.orchestra.run.smithi098.stdout: rgw enable ops log = true 2023-10-06T05:05:58.832 INFO:teuthology.orchestra.run.smithi098.stdout: rgw enable usage log = true 2023-10-06T05:05:58.832 INFO:teuthology.orchestra.run.smithi098.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-06T05:05:58.833 INFO:teuthology.orchestra.run.smithi098.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-06T05:05:58.833 INFO:teuthology.orchestra.run.smithi098.stdout: client mount timeout = 600 2023-10-06T05:05:58.833 INFO:teuthology.orchestra.run.smithi098.stdout: debug client = 20 2023-10-06T05:05:58.833 INFO:teuthology.orchestra.run.smithi098.stdout: debug ms = 1 2023-10-06T05:05:58.833 INFO:teuthology.orchestra.run.smithi098.stdout: pid file = "" 2023-10-06T05:05:58.833 INFO:teuthology.orchestra.run.smithi098.stdout: rados mon op timeout = 900 2023-10-06T05:05:58.833 INFO:teuthology.orchestra.run.smithi098.stdout: rados osd op timeout = 900 2023-10-06T05:05:58.833 INFO:teuthology.orchestra.run.smithi098.stdout:[mon.a] 2023-10-06T05:05:58.833 INFO:teuthology.orchestra.run.smithi098.stdout:[mon.b] 2023-10-06T05:05:58.833 INFO:teuthology.orchestra.run.smithi098.stdout:[mon.c] 2023-10-06T05:05:58.833 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.a] 2023-10-06T05:05:58.833 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.c] 2023-10-06T05:05:58.833 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.b] 2023-10-06T05:05:58.834 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.d] 2023-10-06T05:05:58.834 INFO:teuthology.orchestra.run.smithi098.stdout:[mds] 2023-10-06T05:05:58.834 INFO:teuthology.orchestra.run.smithi098.stdout: debug mds = 20 2023-10-06T05:05:58.834 INFO:teuthology.orchestra.run.smithi098.stdout: debug ms = 1 2023-10-06T05:05:58.834 INFO:teuthology.orchestra.run.smithi098.stdout: mds debug frag = True 2023-10-06T05:05:58.834 INFO:teuthology.orchestra.run.smithi098.stdout: mds debug scatterstat = True 2023-10-06T05:05:58.834 INFO:teuthology.orchestra.run.smithi098.stdout: mds op complaint time = 180 2023-10-06T05:05:58.834 INFO:teuthology.orchestra.run.smithi098.stdout: mds verify scatter = True 2023-10-06T05:05:58.834 INFO:teuthology.orchestra.run.smithi098.stdout: osd op complaint time = 180 2023-10-06T05:05:58.834 INFO:teuthology.orchestra.run.smithi098.stdout: rados mon op timeout = 900 2023-10-06T05:05:58.834 INFO:teuthology.orchestra.run.smithi098.stdout: rados osd op timeout = 900 2023-10-06T05:05:58.836 INFO:teuthology.orchestra.run.smithi174.stdout:[global] 2023-10-06T05:05:58.836 INFO:teuthology.orchestra.run.smithi174.stdout: chdir = "" 2023-10-06T05:05:58.836 INFO:teuthology.orchestra.run.smithi174.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-06T05:05:58.836 INFO:teuthology.orchestra.run.smithi174.stdout: auth supported = cephx 2023-10-06T05:05:58.836 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-10-06T05:05:58.836 INFO:teuthology.orchestra.run.smithi174.stdout: filestore xattr use omap = true 2023-10-06T05:05:58.836 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-10-06T05:05:58.837 INFO:teuthology.orchestra.run.smithi174.stdout: mon clock drift allowed = 1.000 2023-10-06T05:05:58.837 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-10-06T05:05:58.837 INFO:teuthology.orchestra.run.smithi174.stdout: osd crush chooseleaf type = 0 2023-10-06T05:05:58.837 INFO:teuthology.orchestra.run.smithi174.stdout: auth debug = true 2023-10-06T05:05:58.837 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-10-06T05:05:58.837 INFO:teuthology.orchestra.run.smithi174.stdout: ms die on old message = true 2023-10-06T05:05:58.837 INFO:teuthology.orchestra.run.smithi174.stdout: ms die on bug = true 2023-10-06T05:05:58.837 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-10-06T05:05:58.837 INFO:teuthology.orchestra.run.smithi174.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-06T05:05:58.837 INFO:teuthology.orchestra.run.smithi174.stdout: mon pg warn max object skew = 0 2023-10-06T05:05:58.837 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-10-06T05:05:58.837 INFO:teuthology.orchestra.run.smithi174.stdout: # disable pg_autoscaler by default for new pools 2023-10-06T05:05:58.838 INFO:teuthology.orchestra.run.smithi174.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-06T05:05:58.838 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-10-06T05:05:58.838 INFO:teuthology.orchestra.run.smithi174.stdout: osd pool default size = 2 2023-10-06T05:05:58.838 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-10-06T05:05:58.838 INFO:teuthology.orchestra.run.smithi174.stdout: mon osd allow primary affinity = true 2023-10-06T05:05:58.838 INFO:teuthology.orchestra.run.smithi174.stdout: mon osd allow pg remap = true 2023-10-06T05:05:58.838 INFO:teuthology.orchestra.run.smithi174.stdout: mon warn on legacy crush tunables = false 2023-10-06T05:05:58.838 INFO:teuthology.orchestra.run.smithi174.stdout: mon warn on crush straw calc version zero = false 2023-10-06T05:05:58.838 INFO:teuthology.orchestra.run.smithi174.stdout: mon warn on no sortbitwise = false 2023-10-06T05:05:58.838 INFO:teuthology.orchestra.run.smithi174.stdout: mon warn on osd down out interval zero = false 2023-10-06T05:05:58.838 INFO:teuthology.orchestra.run.smithi174.stdout: mon warn on too few osds = false 2023-10-06T05:05:58.838 INFO:teuthology.orchestra.run.smithi174.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-06T05:05:58.839 INFO:teuthology.orchestra.run.smithi174.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-06T05:05:58.839 INFO:teuthology.orchestra.run.smithi174.stdout: mon_allow_pool_size_one = true 2023-10-06T05:05:58.839 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-10-06T05:05:58.839 INFO:teuthology.orchestra.run.smithi174.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-06T05:05:58.839 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-10-06T05:05:58.839 INFO:teuthology.orchestra.run.smithi174.stdout: osd default data pool replay window = 5 2023-10-06T05:05:58.839 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-10-06T05:05:58.839 INFO:teuthology.orchestra.run.smithi174.stdout: mon allow pool delete = true 2023-10-06T05:05:58.839 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-10-06T05:05:58.839 INFO:teuthology.orchestra.run.smithi174.stdout: mon cluster log file level = debug 2023-10-06T05:05:58.839 INFO:teuthology.orchestra.run.smithi174.stdout: debug asserts on shutdown = true 2023-10-06T05:05:58.839 INFO:teuthology.orchestra.run.smithi174.stdout: mon health detail to clog = false 2023-10-06T05:05:58.840 INFO:teuthology.orchestra.run.smithi174.stdout: mon host = "172.21.15.98,172.21.15.174,[v2:172.21.15.174:3301,v1:172.21.15.174:6790]" 2023-10-06T05:05:58.840 INFO:teuthology.orchestra.run.smithi174.stdout: ms die on skipped message = False 2023-10-06T05:05:58.840 INFO:teuthology.orchestra.run.smithi174.stdout: fsid = f50c5df5-cad5-47a4-9bb1-8c791d4311ec 2023-10-06T05:05:58.840 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-10-06T05:05:58.840 INFO:teuthology.orchestra.run.smithi174.stdout:[osd] 2023-10-06T05:05:58.840 INFO:teuthology.orchestra.run.smithi174.stdout: osd journal size = 100 2023-10-06T05:05:58.840 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-10-06T05:05:58.840 INFO:teuthology.orchestra.run.smithi174.stdout: osd scrub load threshold = 5.0 2023-10-06T05:05:58.840 INFO:teuthology.orchestra.run.smithi174.stdout: osd scrub max interval = 600 2023-10-06T05:05:58.840 INFO:teuthology.orchestra.run.smithi174.stdout: osd mclock profile = high_recovery_ops 2023-10-06T05:05:58.840 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-10-06T05:05:58.840 INFO:teuthology.orchestra.run.smithi174.stdout: osd recover clone overlap = true 2023-10-06T05:05:58.840 INFO:teuthology.orchestra.run.smithi174.stdout: osd recovery max chunk = 1048576 2023-10-06T05:05:58.841 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-10-06T05:05:58.841 INFO:teuthology.orchestra.run.smithi174.stdout: osd debug shutdown = true 2023-10-06T05:05:58.841 INFO:teuthology.orchestra.run.smithi174.stdout: osd debug op order = true 2023-10-06T05:05:58.841 INFO:teuthology.orchestra.run.smithi174.stdout: osd debug verify stray on activate = true 2023-10-06T05:05:58.841 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-10-06T05:05:58.841 INFO:teuthology.orchestra.run.smithi174.stdout: osd open classes on start = true 2023-10-06T05:05:58.841 INFO:teuthology.orchestra.run.smithi174.stdout: osd debug pg log writeout = true 2023-10-06T05:05:58.841 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-10-06T05:05:58.841 INFO:teuthology.orchestra.run.smithi174.stdout: osd deep scrub update digest min age = 30 2023-10-06T05:05:58.841 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-10-06T05:05:58.841 INFO:teuthology.orchestra.run.smithi174.stdout: osd map max advance = 10 2023-10-06T05:05:58.841 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-10-06T05:05:58.842 INFO:teuthology.orchestra.run.smithi174.stdout: journal zero on create = true 2023-10-06T05:05:58.842 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-10-06T05:05:58.842 INFO:teuthology.orchestra.run.smithi174.stdout: filestore ondisk finisher threads = 3 2023-10-06T05:05:58.842 INFO:teuthology.orchestra.run.smithi174.stdout: filestore apply finisher threads = 3 2023-10-06T05:05:58.842 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-10-06T05:05:58.842 INFO:teuthology.orchestra.run.smithi174.stdout: bdev debug aio = true 2023-10-06T05:05:58.842 INFO:teuthology.orchestra.run.smithi174.stdout: osd debug misdirected ops = true 2023-10-06T05:05:58.842 INFO:teuthology.orchestra.run.smithi174.stdout: bluestore block size = 96636764160 2023-10-06T05:05:58.843 INFO:teuthology.orchestra.run.smithi174.stdout: bluestore fsck on mount = True 2023-10-06T05:05:58.843 INFO:teuthology.orchestra.run.smithi174.stdout: debug bluefs = 20 2023-10-06T05:05:58.843 INFO:teuthology.orchestra.run.smithi174.stdout: debug bluestore = 20 2023-10-06T05:05:58.843 INFO:teuthology.orchestra.run.smithi174.stdout: debug ms = 1 2023-10-06T05:05:58.843 INFO:teuthology.orchestra.run.smithi174.stdout: debug osd = 20 2023-10-06T05:05:58.843 INFO:teuthology.orchestra.run.smithi174.stdout: debug rocksdb = 10 2023-10-06T05:05:58.843 INFO:teuthology.orchestra.run.smithi174.stdout: mon osd backfillfull_ratio = 0.85 2023-10-06T05:05:58.843 INFO:teuthology.orchestra.run.smithi174.stdout: mon osd full ratio = 0.9 2023-10-06T05:05:58.843 INFO:teuthology.orchestra.run.smithi174.stdout: mon osd nearfull ratio = 0.8 2023-10-06T05:05:58.843 INFO:teuthology.orchestra.run.smithi174.stdout: osd failsafe full ratio = 0.95 2023-10-06T05:05:58.843 INFO:teuthology.orchestra.run.smithi174.stdout: osd objectstore = bluestore 2023-10-06T05:05:58.843 INFO:teuthology.orchestra.run.smithi174.stdout: osd op complaint time = 180 2023-10-06T05:05:58.843 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-10-06T05:05:58.844 INFO:teuthology.orchestra.run.smithi174.stdout:[mgr] 2023-10-06T05:05:58.844 INFO:teuthology.orchestra.run.smithi174.stdout: debug ms = 1 2023-10-06T05:05:58.844 INFO:teuthology.orchestra.run.smithi174.stdout: debug mgr = 20 2023-10-06T05:05:58.844 INFO:teuthology.orchestra.run.smithi174.stdout: debug mon = 20 2023-10-06T05:05:58.844 INFO:teuthology.orchestra.run.smithi174.stdout: debug auth = 20 2023-10-06T05:05:58.844 INFO:teuthology.orchestra.run.smithi174.stdout: mon reweight min pgs per osd = 4 2023-10-06T05:05:58.844 INFO:teuthology.orchestra.run.smithi174.stdout: mon reweight min bytes per osd = 10 2023-10-06T05:05:58.844 INFO:teuthology.orchestra.run.smithi174.stdout: mgr/telemetry/nag = false 2023-10-06T05:05:58.845 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-10-06T05:05:58.845 INFO:teuthology.orchestra.run.smithi174.stdout:[mon] 2023-10-06T05:05:58.845 INFO:teuthology.orchestra.run.smithi174.stdout: debug ms = 1 2023-10-06T05:05:58.845 INFO:teuthology.orchestra.run.smithi174.stdout: debug mon = 20 2023-10-06T05:05:58.845 INFO:teuthology.orchestra.run.smithi174.stdout: debug paxos = 20 2023-10-06T05:05:58.845 INFO:teuthology.orchestra.run.smithi174.stdout: debug auth = 20 2023-10-06T05:05:58.845 INFO:teuthology.orchestra.run.smithi174.stdout: mon data avail warn = 5 2023-10-06T05:05:58.845 INFO:teuthology.orchestra.run.smithi174.stdout: mon mgr mkfs grace = 240 2023-10-06T05:05:58.845 INFO:teuthology.orchestra.run.smithi174.stdout: mon reweight min pgs per osd = 4 2023-10-06T05:05:58.845 INFO:teuthology.orchestra.run.smithi174.stdout: mon osd reporter subtree level = osd 2023-10-06T05:05:58.845 INFO:teuthology.orchestra.run.smithi174.stdout: mon osd prime pg temp = true 2023-10-06T05:05:58.846 INFO:teuthology.orchestra.run.smithi174.stdout: mon reweight min bytes per osd = 10 2023-10-06T05:05:58.846 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-10-06T05:05:58.846 INFO:teuthology.orchestra.run.smithi174.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-06T05:05:58.846 INFO:teuthology.orchestra.run.smithi174.stdout: auth mon ticket ttl = 660 # 11m 2023-10-06T05:05:58.846 INFO:teuthology.orchestra.run.smithi174.stdout: auth service ticket ttl = 240 # 4m 2023-10-06T05:05:58.846 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-10-06T05:05:58.846 INFO:teuthology.orchestra.run.smithi174.stdout: # don't complain about insecure global_id in the test suite 2023-10-06T05:05:58.846 INFO:teuthology.orchestra.run.smithi174.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-06T05:05:58.846 INFO:teuthology.orchestra.run.smithi174.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-06T05:05:58.846 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-10-06T05:05:58.847 INFO:teuthology.orchestra.run.smithi174.stdout: # 1m isn't quite enough 2023-10-06T05:05:58.847 INFO:teuthology.orchestra.run.smithi174.stdout: mon_down_mkfs_grace = 2m 2023-10-06T05:05:58.847 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-10-06T05:05:58.847 INFO:teuthology.orchestra.run.smithi174.stdout: mon_warn_on_filestore_osds = false 2023-10-06T05:05:58.847 INFO:teuthology.orchestra.run.smithi174.stdout: mon op complaint time = 120 2023-10-06T05:05:58.847 INFO:teuthology.orchestra.run.smithi174.stdout: 2023-10-06T05:05:58.847 INFO:teuthology.orchestra.run.smithi174.stdout:[client] 2023-10-06T05:05:58.847 INFO:teuthology.orchestra.run.smithi174.stdout: rgw cache enabled = true 2023-10-06T05:05:58.847 INFO:teuthology.orchestra.run.smithi174.stdout: rgw enable ops log = true 2023-10-06T05:05:58.847 INFO:teuthology.orchestra.run.smithi174.stdout: rgw enable usage log = true 2023-10-06T05:05:58.847 INFO:teuthology.orchestra.run.smithi174.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-06T05:05:58.847 INFO:teuthology.orchestra.run.smithi174.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-06T05:05:58.847 INFO:teuthology.orchestra.run.smithi174.stdout: client mount timeout = 600 2023-10-06T05:05:58.848 INFO:teuthology.orchestra.run.smithi174.stdout: debug client = 20 2023-10-06T05:05:58.848 INFO:teuthology.orchestra.run.smithi174.stdout: debug ms = 1 2023-10-06T05:05:58.849 INFO:teuthology.orchestra.run.smithi174.stdout: pid file = "" 2023-10-06T05:05:58.849 INFO:teuthology.orchestra.run.smithi174.stdout: rados mon op timeout = 900 2023-10-06T05:05:58.849 INFO:teuthology.orchestra.run.smithi174.stdout: rados osd op timeout = 900 2023-10-06T05:05:58.849 INFO:teuthology.orchestra.run.smithi174.stdout:[mon.a] 2023-10-06T05:05:58.849 INFO:teuthology.orchestra.run.smithi174.stdout:[mon.b] 2023-10-06T05:05:58.849 INFO:teuthology.orchestra.run.smithi174.stdout:[mon.c] 2023-10-06T05:05:58.849 INFO:teuthology.orchestra.run.smithi174.stdout:[mds.a] 2023-10-06T05:05:58.849 INFO:teuthology.orchestra.run.smithi174.stdout:[mds.c] 2023-10-06T05:05:58.849 INFO:teuthology.orchestra.run.smithi174.stdout:[mds.b] 2023-10-06T05:05:58.850 INFO:teuthology.orchestra.run.smithi174.stdout:[mds.d] 2023-10-06T05:05:58.850 INFO:teuthology.orchestra.run.smithi174.stdout:[mds] 2023-10-06T05:05:58.850 INFO:teuthology.orchestra.run.smithi174.stdout: debug mds = 20 2023-10-06T05:05:58.850 INFO:teuthology.orchestra.run.smithi174.stdout: debug ms = 1 2023-10-06T05:05:58.850 INFO:teuthology.orchestra.run.smithi174.stdout: mds debug frag = True 2023-10-06T05:05:58.850 INFO:teuthology.orchestra.run.smithi174.stdout: mds debug scatterstat = True 2023-10-06T05:05:58.850 INFO:teuthology.orchestra.run.smithi174.stdout: mds op complaint time = 180 2023-10-06T05:05:58.850 INFO:teuthology.orchestra.run.smithi174.stdout: mds verify scatter = True 2023-10-06T05:05:58.850 INFO:teuthology.orchestra.run.smithi174.stdout: osd op complaint time = 180 2023-10-06T05:05:58.851 INFO:teuthology.orchestra.run.smithi174.stdout: rados mon op timeout = 900 2023-10-06T05:05:58.851 INFO:teuthology.orchestra.run.smithi174.stdout: rados osd op timeout = 900 2023-10-06T05:05:58.851 INFO:tasks.ceph:Creating admin key on mon.a... 2023-10-06T05:05:58.851 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-10-06T05:05:58.927 INFO:tasks.ceph:Copying monmap to all nodes... 2023-10-06T05:05:58.928 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-10-06T05:05:58.928 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-10-06T05:05:58.948 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-10-06T05:05:58.949 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-10-06T05:05:59.003 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi098.front.sepia.ceph.com 2023-10-06T05:05:59.004 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-10-06T05:05:59.004 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-06T05:05:59.004 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-06T05:05:59.077 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-10-06T05:05:59.077 DEBUG:teuthology.orchestra.run.smithi098:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-06T05:05:59.131 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi174.front.sepia.ceph.com 2023-10-06T05:05:59.131 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2023-10-06T05:05:59.132 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-06T05:05:59.132 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-06T05:05:59.167 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2023-10-06T05:05:59.167 DEBUG:teuthology.orchestra.run.smithi174:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-06T05:05:59.221 INFO:tasks.ceph:Setting up mon nodes... 2023-10-06T05:05:59.222 INFO:tasks.ceph:Setting up mgr nodes... 2023-10-06T05:05:59.222 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-10-06T05:05:59.270 INFO:teuthology.orchestra.run.smithi098.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-10-06T05:05:59.272 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-10-06T05:05:59.323 INFO:teuthology.orchestra.run.smithi174.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-10-06T05:05:59.326 INFO:tasks.ceph:Setting up mds nodes... 2023-10-06T05:05:59.327 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2023-10-06T05:05:59.374 INFO:teuthology.orchestra.run.smithi098.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2023-10-06T05:05:59.377 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2023-10-06T05:05:59.401 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2023-10-06T05:05:59.488 INFO:teuthology.orchestra.run.smithi098.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2023-10-06T05:05:59.490 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2023-10-06T05:05:59.514 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2023-10-06T05:05:59.562 INFO:teuthology.orchestra.run.smithi174.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2023-10-06T05:05:59.564 DEBUG:teuthology.orchestra.run.smithi174:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2023-10-06T05:05:59.588 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2023-10-06T05:05:59.675 INFO:teuthology.orchestra.run.smithi174.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2023-10-06T05:05:59.678 DEBUG:teuthology.orchestra.run.smithi174:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2023-10-06T05:05:59.702 INFO:tasks.ceph_client:Setting up client nodes... 2023-10-06T05:05:59.702 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-10-06T05:05:59.739 INFO:teuthology.orchestra.run.smithi098.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-10-06T05:05:59.752 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2023-10-06T05:05:59.829 INFO:teuthology.orchestra.run.smithi098.stdout:creating /etc/ceph/ceph.client.2.keyring 2023-10-06T05:05:59.842 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-10-06T05:05:59.879 INFO:teuthology.orchestra.run.smithi174.stdout:creating /etc/ceph/ceph.client.1.keyring 2023-10-06T05:05:59.892 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2023-10-06T05:05:59.969 INFO:teuthology.orchestra.run.smithi174.stdout:creating /etc/ceph/ceph.client.3.keyring 2023-10-06T05:05:59.982 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-10-06T05:05:59.982 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi098.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi174.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-10-06T05:05:59.982 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-10-06T05:06:00.006 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-06T05:06:00.007 INFO:tasks.ceph:role: osd.0 2023-10-06T05:06:00.007 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi098.front.sepia.ceph.com 2023-10-06T05:06:00.007 DEBUG:teuthology.orchestra.run.smithi098:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-06T05:06:00.078 INFO:teuthology.orchestra.run.smithi098.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-06T05:06:00.078 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-06T05:06:00.078 INFO:teuthology.orchestra.run.smithi098.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-06T05:06:00.078 INFO:teuthology.orchestra.run.smithi098.stdout: = reflink=1 2023-10-06T05:06:00.078 INFO:teuthology.orchestra.run.smithi098.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-06T05:06:00.078 INFO:teuthology.orchestra.run.smithi098.stdout: = sunit=0 swidth=0 blks 2023-10-06T05:06:00.079 INFO:teuthology.orchestra.run.smithi098.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-06T05:06:00.079 INFO:teuthology.orchestra.run.smithi098.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-06T05:06:00.079 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-06T05:06:00.079 INFO:teuthology.orchestra.run.smithi098.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-06T05:06:00.502 INFO:teuthology.orchestra.run.smithi098.stdout:Discarding blocks...Done. 2023-10-06T05:06:00.504 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi098.front.sepia.ceph.com -o noatime 2023-10-06T05:06:00.504 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-10-06T05:06:00.672 DEBUG:teuthology.orchestra.run.smithi098:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-10-06T05:06:00.702 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-10-06T05:06:00.766 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-06T05:06:00.766 INFO:tasks.ceph:role: osd.1 2023-10-06T05:06:00.767 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi098.front.sepia.ceph.com 2023-10-06T05:06:00.767 DEBUG:teuthology.orchestra.run.smithi098:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-06T05:06:00.838 INFO:teuthology.orchestra.run.smithi098.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-06T05:06:00.838 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-06T05:06:00.838 INFO:teuthology.orchestra.run.smithi098.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-06T05:06:00.838 INFO:teuthology.orchestra.run.smithi098.stdout: = reflink=1 2023-10-06T05:06:00.838 INFO:teuthology.orchestra.run.smithi098.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-06T05:06:00.839 INFO:teuthology.orchestra.run.smithi098.stdout: = sunit=0 swidth=0 blks 2023-10-06T05:06:00.839 INFO:teuthology.orchestra.run.smithi098.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-06T05:06:00.839 INFO:teuthology.orchestra.run.smithi098.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-06T05:06:00.839 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-06T05:06:00.839 INFO:teuthology.orchestra.run.smithi098.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-06T05:06:01.275 INFO:teuthology.orchestra.run.smithi098.stdout:Discarding blocks...Done. 2023-10-06T05:06:01.277 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi098.front.sepia.ceph.com -o noatime 2023-10-06T05:06:01.277 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-10-06T05:06:01.373 DEBUG:teuthology.orchestra.run.smithi098:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-10-06T05:06:01.401 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-10-06T05:06:01.466 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-06T05:06:01.466 INFO:tasks.ceph:role: osd.2 2023-10-06T05:06:01.466 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi098.front.sepia.ceph.com 2023-10-06T05:06:01.466 DEBUG:teuthology.orchestra.run.smithi098:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-06T05:06:01.537 INFO:teuthology.orchestra.run.smithi098.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-06T05:06:01.538 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-06T05:06:01.538 INFO:teuthology.orchestra.run.smithi098.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-06T05:06:01.538 INFO:teuthology.orchestra.run.smithi098.stdout: = reflink=1 2023-10-06T05:06:01.538 INFO:teuthology.orchestra.run.smithi098.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-06T05:06:01.538 INFO:teuthology.orchestra.run.smithi098.stdout: = sunit=0 swidth=0 blks 2023-10-06T05:06:01.538 INFO:teuthology.orchestra.run.smithi098.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-06T05:06:01.538 INFO:teuthology.orchestra.run.smithi098.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-06T05:06:01.538 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-06T05:06:01.538 INFO:teuthology.orchestra.run.smithi098.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-06T05:06:01.984 INFO:teuthology.orchestra.run.smithi098.stdout:Discarding blocks...Done. 2023-10-06T05:06:01.986 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi098.front.sepia.ceph.com -o noatime 2023-10-06T05:06:01.986 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-10-06T05:06:02.152 DEBUG:teuthology.orchestra.run.smithi098:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-10-06T05:06:02.182 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-10-06T05:06:02.247 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-06T05:06:02.303 INFO:tasks.ceph:role: osd.3 2023-10-06T05:06:02.303 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi098.front.sepia.ceph.com 2023-10-06T05:06:02.303 DEBUG:teuthology.orchestra.run.smithi098:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-10-06T05:06:02.335 INFO:teuthology.orchestra.run.smithi098.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-10-06T05:06:02.335 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-06T05:06:02.335 INFO:teuthology.orchestra.run.smithi098.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-06T05:06:02.335 INFO:teuthology.orchestra.run.smithi098.stdout: = reflink=1 2023-10-06T05:06:02.335 INFO:teuthology.orchestra.run.smithi098.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-06T05:06:02.335 INFO:teuthology.orchestra.run.smithi098.stdout: = sunit=0 swidth=0 blks 2023-10-06T05:06:02.336 INFO:teuthology.orchestra.run.smithi098.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-06T05:06:02.336 INFO:teuthology.orchestra.run.smithi098.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-06T05:06:02.336 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-06T05:06:02.336 INFO:teuthology.orchestra.run.smithi098.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-06T05:06:02.779 INFO:teuthology.orchestra.run.smithi098.stdout:Discarding blocks...Done. 2023-10-06T05:06:02.785 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi098.front.sepia.ceph.com -o noatime 2023-10-06T05:06:02.786 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-10-06T05:06:02.955 DEBUG:teuthology.orchestra.run.smithi098:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-10-06T05:06:02.983 DEBUG:teuthology.orchestra.run.smithi098:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-06T05:06:03.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:03.274+0000 7f9df7f93540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-10-06T05:06:03.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:03.274+0000 7f9df7f93540 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-10-06T05:06:03.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:03.274+0000 7f9df7f93540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-06T05:06:03.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:03.274+0000 7f9df7f93540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-06T05:06:03.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:03.274+0000 7f9df7f93540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-06T05:06:03.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:03.274+0000 7f9df7f93540 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-10-06T05:06:03.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:03.799+0000 7f9df7f93540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-06T05:06:04.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:04.563+0000 7f9df7f93540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-06T05:06:05.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:05.325+0000 7f9df7f93540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-06T05:06:06.124 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-10-06T05:06:06.149 DEBUG:teuthology.orchestra.run.smithi098:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-06T05:06:06.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:06.227+0000 7f3eee202540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-10-06T05:06:06.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:06.228+0000 7f3eee202540 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-10-06T05:06:06.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:06.228+0000 7f3eee202540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-06T05:06:06.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:06.228+0000 7f3eee202540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-06T05:06:06.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:06.228+0000 7f3eee202540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-06T05:06:06.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:06.228+0000 7f3eee202540 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-10-06T05:06:06.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:06.753+0000 7f3eee202540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-06T05:06:07.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:07.521+0000 7f3eee202540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-06T05:06:08.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:08.295+0000 7f3eee202540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-06T05:06:09.088 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-10-06T05:06:09.159 DEBUG:teuthology.orchestra.run.smithi098:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-06T05:06:09.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:09.289+0000 7f2af2446540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-10-06T05:06:09.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:09.289+0000 7f2af2446540 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-10-06T05:06:09.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:09.289+0000 7f2af2446540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-06T05:06:09.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:09.289+0000 7f2af2446540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-06T05:06:09.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:09.289+0000 7f2af2446540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-06T05:06:09.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:09.289+0000 7f2af2446540 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-10-06T05:06:09.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:09.816+0000 7f2af2446540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-06T05:06:10.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:10.584+0000 7f2af2446540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-06T05:06:11.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:11.347+0000 7f2af2446540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-06T05:06:12.144 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-10-06T05:06:12.169 DEBUG:teuthology.orchestra.run.smithi098:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-06T05:06:12.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:12.248+0000 7f7efc8e5540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-10-06T05:06:12.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:12.249+0000 7f7efc8e5540 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-10-06T05:06:12.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:12.249+0000 7f7efc8e5540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-06T05:06:12.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:12.249+0000 7f7efc8e5540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-06T05:06:12.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:12.249+0000 7f7efc8e5540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-06T05:06:12.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:12.249+0000 7f7efc8e5540 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-10-06T05:06:12.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:12.771+0000 7f7efc8e5540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-06T05:06:13.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:13.533+0000 7f7efc8e5540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-06T05:06:14.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:14.296+0000 7f7efc8e5540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-06T05:06:15.095 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-10-06T05:06:15.163 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-10-06T05:06:15.191 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-06T05:06:15.192 INFO:tasks.ceph:role: osd.4 2023-10-06T05:06:15.192 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi174.front.sepia.ceph.com 2023-10-06T05:06:15.192 DEBUG:teuthology.orchestra.run.smithi174:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-06T05:06:15.266 INFO:teuthology.orchestra.run.smithi174.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-06T05:06:15.266 INFO:teuthology.orchestra.run.smithi174.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-06T05:06:15.266 INFO:teuthology.orchestra.run.smithi174.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-06T05:06:15.266 INFO:teuthology.orchestra.run.smithi174.stdout: = reflink=1 2023-10-06T05:06:15.267 INFO:teuthology.orchestra.run.smithi174.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-06T05:06:15.267 INFO:teuthology.orchestra.run.smithi174.stdout: = sunit=0 swidth=0 blks 2023-10-06T05:06:15.267 INFO:teuthology.orchestra.run.smithi174.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-06T05:06:15.267 INFO:teuthology.orchestra.run.smithi174.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-06T05:06:15.267 INFO:teuthology.orchestra.run.smithi174.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-06T05:06:15.267 INFO:teuthology.orchestra.run.smithi174.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-06T05:06:15.705 INFO:teuthology.orchestra.run.smithi174.stdout:Discarding blocks...Done. 2023-10-06T05:06:15.707 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi174.front.sepia.ceph.com -o noatime 2023-10-06T05:06:15.708 DEBUG:teuthology.orchestra.run.smithi174:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-10-06T05:06:15.890 DEBUG:teuthology.orchestra.run.smithi174:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-10-06T05:06:15.926 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-10-06T05:06:15.991 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-06T05:06:15.991 INFO:tasks.ceph:role: osd.5 2023-10-06T05:06:15.992 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi174.front.sepia.ceph.com 2023-10-06T05:06:15.992 DEBUG:teuthology.orchestra.run.smithi174:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-06T05:06:16.065 INFO:teuthology.orchestra.run.smithi174.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-06T05:06:16.065 INFO:teuthology.orchestra.run.smithi174.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-06T05:06:16.065 INFO:teuthology.orchestra.run.smithi174.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-06T05:06:16.065 INFO:teuthology.orchestra.run.smithi174.stdout: = reflink=1 2023-10-06T05:06:16.065 INFO:teuthology.orchestra.run.smithi174.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-06T05:06:16.065 INFO:teuthology.orchestra.run.smithi174.stdout: = sunit=0 swidth=0 blks 2023-10-06T05:06:16.065 INFO:teuthology.orchestra.run.smithi174.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-06T05:06:16.065 INFO:teuthology.orchestra.run.smithi174.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-06T05:06:16.065 INFO:teuthology.orchestra.run.smithi174.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-06T05:06:16.066 INFO:teuthology.orchestra.run.smithi174.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-06T05:06:16.484 INFO:teuthology.orchestra.run.smithi174.stdout:Discarding blocks...Done. 2023-10-06T05:06:16.486 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi174.front.sepia.ceph.com -o noatime 2023-10-06T05:06:16.487 DEBUG:teuthology.orchestra.run.smithi174:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-10-06T05:06:16.598 DEBUG:teuthology.orchestra.run.smithi174:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-10-06T05:06:16.629 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-10-06T05:06:16.695 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-06T05:06:16.696 INFO:tasks.ceph:role: osd.6 2023-10-06T05:06:16.696 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi174.front.sepia.ceph.com 2023-10-06T05:06:16.696 DEBUG:teuthology.orchestra.run.smithi174:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-06T05:06:16.770 INFO:teuthology.orchestra.run.smithi174.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-06T05:06:16.770 INFO:teuthology.orchestra.run.smithi174.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-06T05:06:16.771 INFO:teuthology.orchestra.run.smithi174.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-06T05:06:16.771 INFO:teuthology.orchestra.run.smithi174.stdout: = reflink=1 2023-10-06T05:06:16.771 INFO:teuthology.orchestra.run.smithi174.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-06T05:06:16.771 INFO:teuthology.orchestra.run.smithi174.stdout: = sunit=0 swidth=0 blks 2023-10-06T05:06:16.771 INFO:teuthology.orchestra.run.smithi174.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-06T05:06:16.771 INFO:teuthology.orchestra.run.smithi174.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-06T05:06:16.771 INFO:teuthology.orchestra.run.smithi174.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-06T05:06:16.771 INFO:teuthology.orchestra.run.smithi174.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-06T05:06:17.223 INFO:teuthology.orchestra.run.smithi174.stdout:Discarding blocks...Done. 2023-10-06T05:06:17.225 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi174.front.sepia.ceph.com -o noatime 2023-10-06T05:06:17.225 DEBUG:teuthology.orchestra.run.smithi174:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-10-06T05:06:17.357 DEBUG:teuthology.orchestra.run.smithi174:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-10-06T05:06:17.388 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-10-06T05:06:17.455 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-06T05:06:17.455 INFO:tasks.ceph:role: osd.7 2023-10-06T05:06:17.455 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi174.front.sepia.ceph.com 2023-10-06T05:06:17.455 DEBUG:teuthology.orchestra.run.smithi174:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-10-06T05:06:17.528 INFO:teuthology.orchestra.run.smithi174.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-10-06T05:06:17.528 INFO:teuthology.orchestra.run.smithi174.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-06T05:06:17.528 INFO:teuthology.orchestra.run.smithi174.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-06T05:06:17.528 INFO:teuthology.orchestra.run.smithi174.stdout: = reflink=1 2023-10-06T05:06:17.528 INFO:teuthology.orchestra.run.smithi174.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-06T05:06:17.528 INFO:teuthology.orchestra.run.smithi174.stdout: = sunit=0 swidth=0 blks 2023-10-06T05:06:17.528 INFO:teuthology.orchestra.run.smithi174.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-06T05:06:17.529 INFO:teuthology.orchestra.run.smithi174.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-06T05:06:17.529 INFO:teuthology.orchestra.run.smithi174.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-06T05:06:17.529 INFO:teuthology.orchestra.run.smithi174.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-06T05:06:17.977 INFO:teuthology.orchestra.run.smithi174.stdout:Discarding blocks...Done. 2023-10-06T05:06:17.979 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi174.front.sepia.ceph.com -o noatime 2023-10-06T05:06:17.980 DEBUG:teuthology.orchestra.run.smithi174:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-10-06T05:06:18.147 DEBUG:teuthology.orchestra.run.smithi174:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-10-06T05:06:18.181 DEBUG:teuthology.orchestra.run.smithi174:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-06T05:06:18.310 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:18.309+0000 7f082806f540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-10-06T05:06:18.311 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:18.310+0000 7f082806f540 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-10-06T05:06:18.311 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:18.310+0000 7f082806f540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-06T05:06:18.311 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:18.310+0000 7f082806f540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-06T05:06:18.311 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:18.310+0000 7f082806f540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-06T05:06:18.311 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:18.310+0000 7f082806f540 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-10-06T05:06:18.835 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:18.834+0000 7f082806f540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-06T05:06:19.600 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:19.598+0000 7f082806f540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-06T05:06:20.366 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:20.365+0000 7f082806f540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-06T05:06:21.164 DEBUG:teuthology.orchestra.run.smithi174:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-10-06T05:06:21.194 DEBUG:teuthology.orchestra.run.smithi174:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-06T05:06:21.279 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:21.278+0000 7f09b517d540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-10-06T05:06:21.280 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:21.278+0000 7f09b517d540 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-10-06T05:06:21.280 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:21.278+0000 7f09b517d540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-06T05:06:21.280 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:21.278+0000 7f09b517d540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-06T05:06:21.280 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:21.278+0000 7f09b517d540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-06T05:06:21.280 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:21.279+0000 7f09b517d540 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-10-06T05:06:21.803 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:21.801+0000 7f09b517d540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-06T05:06:22.578 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:22.577+0000 7f09b517d540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-06T05:06:23.342 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:23.341+0000 7f09b517d540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-06T05:06:24.137 DEBUG:teuthology.orchestra.run.smithi174:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-10-06T05:06:24.164 DEBUG:teuthology.orchestra.run.smithi174:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-06T05:06:24.249 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:24.247+0000 7f7ca8a40540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-10-06T05:06:24.249 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:24.248+0000 7f7ca8a40540 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-10-06T05:06:24.250 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:24.248+0000 7f7ca8a40540 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-06T05:06:24.250 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:24.248+0000 7f7ca8a40540 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-06T05:06:24.250 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:24.248+0000 7f7ca8a40540 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-06T05:06:24.250 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:24.248+0000 7f7ca8a40540 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-10-06T05:06:24.774 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:24.773+0000 7f7ca8a40540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-06T05:06:25.537 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:25.536+0000 7f7ca8a40540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-06T05:06:26.299 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:26.298+0000 7f7ca8a40540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-06T05:06:26.790 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-06T05:06:26.793 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-06T05:06:27.094 DEBUG:teuthology.orchestra.run.smithi174:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-10-06T05:06:27.119 DEBUG:teuthology.orchestra.run.smithi174:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-06T05:06:27.200 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:27.198+0000 7fc439d82540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-10-06T05:06:27.200 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:27.198+0000 7fc439d82540 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-10-06T05:06:27.200 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:27.198+0000 7fc439d82540 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-06T05:06:27.200 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:27.198+0000 7fc439d82540 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-06T05:06:27.200 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:27.198+0000 7fc439d82540 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-06T05:06:27.200 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:27.199+0000 7fc439d82540 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-10-06T05:06:27.723 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:27.722+0000 7fc439d82540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-06T05:06:28.496 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:28.495+0000 7fc439d82540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-06T05:06:29.264 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:29.262+0000 7fc439d82540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-06T05:06:30.057 DEBUG:teuthology.orchestra.run.smithi174:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-10-06T05:06:30.123 INFO:tasks.ceph:Reading keys from all nodes... 2023-10-06T05:06:30.123 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-10-06T05:06:30.124 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-10-06T05:06:30.153 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-10-06T05:06:30.153 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2023-10-06T05:06:30.220 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-10-06T05:06:30.220 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2023-10-06T05:06:30.286 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-10-06T05:06:30.286 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-10-06T05:06:30.353 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-10-06T05:06:30.353 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-10-06T05:06:30.419 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-10-06T05:06:30.419 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-10-06T05:06:30.484 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-10-06T05:06:30.484 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-10-06T05:06:30.548 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2023-10-06T05:06:30.549 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-10-06T05:06:30.578 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2023-10-06T05:06:30.578 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2023-10-06T05:06:30.643 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2023-10-06T05:06:30.643 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2023-10-06T05:06:30.708 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2023-10-06T05:06:30.708 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-10-06T05:06:30.773 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2023-10-06T05:06:30.773 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-10-06T05:06:30.838 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2023-10-06T05:06:30.838 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-10-06T05:06:30.903 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2023-10-06T05:06:30.903 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-10-06T05:06:30.967 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-10-06T05:06:30.967 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-10-06T05:06:30.983 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-10-06T05:06:30.983 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2023-10-06T05:06:31.038 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2023-10-06T05:06:31.038 DEBUG:teuthology.orchestra.run.smithi174:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2023-10-06T05:06:31.053 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2023-10-06T05:06:31.053 DEBUG:teuthology.orchestra.run.smithi174:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2023-10-06T05:06:31.107 INFO:tasks.ceph:Adding keys to all mons... 2023-10-06T05:06:31.107 DEBUG:teuthology.orchestra.run.smithi098:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-06T05:06:31.109 DEBUG:teuthology.orchestra.run.smithi174:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-06T05:06:31.150 INFO:teuthology.orchestra.run.smithi098.stdout:[mgr.y] 2023-10-06T05:06:31.150 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQC3lR9lop4MEBAAaic3jAzQKe2Yz+BX80cK4Q== 2023-10-06T05:06:31.150 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.a] 2023-10-06T05:06:31.151 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQC3lR9l8/dMFhAANvojBT5KMbxAA/zmYdKwdQ== 2023-10-06T05:06:31.151 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.c] 2023-10-06T05:06:31.151 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQC3lR9lstsTHRAAFofaYDUECIUfeln+MEyJ9w== 2023-10-06T05:06:31.151 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.0] 2023-10-06T05:06:31.151 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQC7lR9lWNtrEBAAT2EktzoBT2dsJmiioQ4ySg== 2023-10-06T05:06:31.152 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.1] 2023-10-06T05:06:31.152 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQC+lR9l/JepDRAA8FyqlCk/OfnLWtD5Ilnjzw== 2023-10-06T05:06:31.152 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.2] 2023-10-06T05:06:31.152 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQDBlR9ljslNERAAqnf/WmMPMgNrW4T5pK+59w== 2023-10-06T05:06:31.152 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.3] 2023-10-06T05:06:31.153 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQDElR9lMzfaDhAAhtX4bPxsE/x9EldgsnfWdA== 2023-10-06T05:06:31.153 INFO:teuthology.orchestra.run.smithi098.stdout:[mgr.x] 2023-10-06T05:06:31.153 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQC3lR9leW8+ExAAGn+V3TntGoC+CdBjnyWBuw== 2023-10-06T05:06:31.153 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.b] 2023-10-06T05:06:31.153 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQC3lR9lvbV3IRAAqPaSuCsrRugVa4cioHdaeQ== 2023-10-06T05:06:31.153 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.d] 2023-10-06T05:06:31.153 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQC3lR9lLGE6KBAAIYl9F6qfRBCRgk1I78L4wA== 2023-10-06T05:06:31.153 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.4] 2023-10-06T05:06:31.153 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQDKlR9leHV3EhAAwOyH0FPXKgz7jRmckY7F4A== 2023-10-06T05:06:31.153 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.5] 2023-10-06T05:06:31.153 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQDNlR9l/OGeEBAAc/1Dn4zBLTGZDHOV+DpyCA== 2023-10-06T05:06:31.154 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.6] 2023-10-06T05:06:31.154 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQDQlR9ldKHMDhAAzSvXhA/oE0CN/06HES/Vtg== 2023-10-06T05:06:31.154 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.7] 2023-10-06T05:06:31.154 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQDTlR9lKsrYCxAAarQ0keRxrs5848xZQhBl0Q== 2023-10-06T05:06:31.154 INFO:teuthology.orchestra.run.smithi098.stdout:[client.0] 2023-10-06T05:06:31.154 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQC3lR9lpjoPLBAAAYJorK9BQnAyELQt3AGqDw== 2023-10-06T05:06:31.154 INFO:teuthology.orchestra.run.smithi098.stdout:[client.2] 2023-10-06T05:06:31.154 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQC3lR9lbPdqMRAAB0QpBNi/jn+CQupZAWupMw== 2023-10-06T05:06:31.154 INFO:teuthology.orchestra.run.smithi098.stdout:[client.1] 2023-10-06T05:06:31.154 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQC3lR9lDCdkNBAANLue0SCLNp7ySmlhD97ARQ== 2023-10-06T05:06:31.154 INFO:teuthology.orchestra.run.smithi098.stdout:[client.3] 2023-10-06T05:06:31.154 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQC3lR9lAPzEORAAD6oTL9GOgguazIoZRPMpAA== 2023-10-06T05:06:31.169 INFO:teuthology.orchestra.run.smithi174.stdout:[mgr.y] 2023-10-06T05:06:31.169 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQC3lR9lop4MEBAAaic3jAzQKe2Yz+BX80cK4Q== 2023-10-06T05:06:31.170 INFO:teuthology.orchestra.run.smithi174.stdout:[mds.a] 2023-10-06T05:06:31.170 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQC3lR9l8/dMFhAANvojBT5KMbxAA/zmYdKwdQ== 2023-10-06T05:06:31.170 INFO:teuthology.orchestra.run.smithi174.stdout:[mds.c] 2023-10-06T05:06:31.170 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQC3lR9lstsTHRAAFofaYDUECIUfeln+MEyJ9w== 2023-10-06T05:06:31.170 INFO:teuthology.orchestra.run.smithi174.stdout:[osd.0] 2023-10-06T05:06:31.170 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQC7lR9lWNtrEBAAT2EktzoBT2dsJmiioQ4ySg== 2023-10-06T05:06:31.170 INFO:teuthology.orchestra.run.smithi174.stdout:[osd.1] 2023-10-06T05:06:31.170 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQC+lR9l/JepDRAA8FyqlCk/OfnLWtD5Ilnjzw== 2023-10-06T05:06:31.170 INFO:teuthology.orchestra.run.smithi174.stdout:[osd.2] 2023-10-06T05:06:31.170 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQDBlR9ljslNERAAqnf/WmMPMgNrW4T5pK+59w== 2023-10-06T05:06:31.170 INFO:teuthology.orchestra.run.smithi174.stdout:[osd.3] 2023-10-06T05:06:31.171 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQDElR9lMzfaDhAAhtX4bPxsE/x9EldgsnfWdA== 2023-10-06T05:06:31.171 INFO:teuthology.orchestra.run.smithi174.stdout:[mgr.x] 2023-10-06T05:06:31.171 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQC3lR9leW8+ExAAGn+V3TntGoC+CdBjnyWBuw== 2023-10-06T05:06:31.171 INFO:teuthology.orchestra.run.smithi174.stdout:[mds.b] 2023-10-06T05:06:31.171 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQC3lR9lvbV3IRAAqPaSuCsrRugVa4cioHdaeQ== 2023-10-06T05:06:31.171 INFO:teuthology.orchestra.run.smithi174.stdout:[mds.d] 2023-10-06T05:06:31.171 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQC3lR9lLGE6KBAAIYl9F6qfRBCRgk1I78L4wA== 2023-10-06T05:06:31.171 INFO:teuthology.orchestra.run.smithi174.stdout:[osd.4] 2023-10-06T05:06:31.171 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQDKlR9leHV3EhAAwOyH0FPXKgz7jRmckY7F4A== 2023-10-06T05:06:31.172 INFO:teuthology.orchestra.run.smithi174.stdout:[osd.5] 2023-10-06T05:06:31.172 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQDNlR9l/OGeEBAAc/1Dn4zBLTGZDHOV+DpyCA== 2023-10-06T05:06:31.172 INFO:teuthology.orchestra.run.smithi174.stdout:[osd.6] 2023-10-06T05:06:31.172 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQDQlR9ldKHMDhAAzSvXhA/oE0CN/06HES/Vtg== 2023-10-06T05:06:31.172 INFO:teuthology.orchestra.run.smithi174.stdout:[osd.7] 2023-10-06T05:06:31.172 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQDTlR9lKsrYCxAAarQ0keRxrs5848xZQhBl0Q== 2023-10-06T05:06:31.172 INFO:teuthology.orchestra.run.smithi174.stdout:[client.0] 2023-10-06T05:06:31.172 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQC3lR9lpjoPLBAAAYJorK9BQnAyELQt3AGqDw== 2023-10-06T05:06:31.173 INFO:teuthology.orchestra.run.smithi174.stdout:[client.2] 2023-10-06T05:06:31.173 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQC3lR9lbPdqMRAAB0QpBNi/jn+CQupZAWupMw== 2023-10-06T05:06:31.173 INFO:teuthology.orchestra.run.smithi174.stdout:[client.1] 2023-10-06T05:06:31.173 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQC3lR9lDCdkNBAANLue0SCLNp7ySmlhD97ARQ== 2023-10-06T05:06:31.173 INFO:teuthology.orchestra.run.smithi174.stdout:[client.3] 2023-10-06T05:06:31.173 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQC3lR9lAPzEORAAD6oTL9GOgguazIoZRPMpAA== 2023-10-06T05:06:31.174 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-06T05:06:31.198 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-06T05:06:31.260 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-06T05:06:31.263 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-06T05:06:31.312 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-06T05:06:31.315 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-06T05:06:31.365 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-06T05:06:31.368 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-06T05:06:31.415 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-06T05:06:31.417 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-06T05:06:31.464 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-06T05:06:31.467 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-06T05:06:31.514 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-06T05:06:31.516 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-06T05:06:31.563 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-06T05:06:31.565 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-06T05:06:31.613 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-06T05:06:31.615 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-06T05:06:31.665 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-06T05:06:31.667 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-06T05:06:31.717 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-06T05:06:31.719 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-06T05:06:31.767 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-06T05:06:31.769 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-06T05:06:31.816 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-06T05:06:31.818 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-06T05:06:31.865 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-06T05:06:31.868 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-06T05:06:31.915 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-06T05:06:31.917 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-06T05:06:31.967 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-06T05:06:31.970 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-06T05:06:32.019 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-06T05:06:32.022 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-06T05:06:32.072 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-06T05:06:32.074 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-06T05:06:32.125 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-10-06T05:06:32.126 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-10-06T05:06:32.150 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-06T05:06:32.241 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-10-06T05:06:32.265 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-10-06T05:06:32.289 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-06T05:06:32.381 DEBUG:teuthology.orchestra.run.smithi174:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-10-06T05:06:32.405 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-10-06T05:06:32.469 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-06T05:06:32.560 DEBUG:teuthology.orchestra.run.smithi174:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-10-06T05:06:32.584 DEBUG:teuthology.orchestra.run.smithi098:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-06T05:06:32.586 DEBUG:teuthology.orchestra.run.smithi174:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-06T05:06:32.639 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-10-06T05:06:32.639 INFO:tasks.ceph.mon.a:Restarting daemon 2023-10-06T05:06:32.639 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-10-06T05:06:32.641 INFO:tasks.ceph.mon.a:Started 2023-10-06T05:06:32.641 INFO:tasks.ceph.mon.b:Restarting daemon 2023-10-06T05:06:32.642 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-10-06T05:06:32.681 INFO:tasks.ceph.mon.b:Started 2023-10-06T05:06:32.681 INFO:tasks.ceph.mon.c:Restarting daemon 2023-10-06T05:06:32.682 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-10-06T05:06:32.683 INFO:tasks.ceph.mon.c:Started 2023-10-06T05:06:32.683 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-10-06T05:06:32.684 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-10-06T05:06:32.684 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2023-10-06T05:06:32.686 INFO:tasks.ceph.mgr.y:Started 2023-10-06T05:06:32.686 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-10-06T05:06:32.686 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-10-06T05:06:32.688 INFO:tasks.ceph.mgr.x:Started 2023-10-06T05:06:32.688 INFO:tasks.ceph:Setting crush tunables to default 2023-10-06T05:06:32.688 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --cluster ceph osd crush tunables default 2023-10-06T05:06:32.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:32.801+0000 7f72c950e700 1 Processor -- start 2023-10-06T05:06:32.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:32.801+0000 7f72c950e700 1 -- start start 2023-10-06T05:06:32.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:32.802+0000 7f72c950e700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f72c4150940 0x7f72c4159c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:32.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:32.802+0000 7f72c950e700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f72c415a160 0x7f72c40ccdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:32.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:32.802+0000 7f72c950e700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f72c40c5250 0x7f72c40cd300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:32.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:32.802+0000 7f72c950e700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72c415ac30 con 0x7f72c40c5250 2023-10-06T05:06:32.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:32.802+0000 7f72c950e700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72c40cb6a0 con 0x7f72c4150940 2023-10-06T05:06:32.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:32.802+0000 7f72c950e700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72c40cb090 con 0x7f72c415a160 2023-10-06T05:06:32.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:32.802+0000 7f72c37fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f72c40c5250 0x7f72c40cd300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:32.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:32.802+0000 7f72c37fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f72c40c5250 0x7f72c40cd300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39086/0 (socket says 172.21.15.98:39086) 2023-10-06T05:06:32.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:32.802+0000 7f72c37fe700 1 -- 172.21.15.98:0/3965240208 learned_addr learned my addr 172.21.15.98:0/3965240208 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:06:32.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:32.802+0000 7f72c37fe700 1 -- 172.21.15.98:0/3965240208 >> v2:172.21.15.98:3300/0 conn(0x7f72c40c5250 msgr2=0x7f72c40cd300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-06T05:06:32.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:32.802+0000 7f72c37fe700 1 -- 172.21.15.98:0/3965240208 >> v2:172.21.15.98:3300/0 conn(0x7f72c40c5250 msgr2=0x7f72c40cd300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-06T05:06:32.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:32.802+0000 7f72c37fe700 1 --2- 172.21.15.98:0/3965240208 >> v2:172.21.15.98:3300/0 conn(0x7f72c40c5250 0x7f72c40cd300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-10-06T05:06:32.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:32.802+0000 7f72c27fc700 1 --2- 172.21.15.98:0/3965240208 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f72c415a160 0x7f72c40ccdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:32.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:32.802+0000 7f72c2ffd700 1 --2- 172.21.15.98:0/3965240208 >> v2:172.21.15.174:3300/0 conn(0x7f72c4150940 0x7f72c4159c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:32.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:32.802+0000 7f72c37fe700 1 --2- 172.21.15.98:0/3965240208 >> v2:172.21.15.98:3300/0 conn(0x7f72c40c5250 0x7f72c40cd300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-06T05:06:32.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:32.802+0000 7f72c27fc700 1 -- 172.21.15.98:0/3965240208 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f72c415a160 msgr2=0x7f72c40ccdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-06T05:06:32.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:32.802+0000 7f72c27fc700 1 -- 172.21.15.98:0/3965240208 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f72c415a160 msgr2=0x7f72c40ccdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-06T05:06:32.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:32.802+0000 7f72c27fc700 1 --2- 172.21.15.98:0/3965240208 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f72c415a160 0x7f72c40ccdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-10-06T05:06:32.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:32.802+0000 7f72c2ffd700 1 -- 172.21.15.98:0/3965240208 >> v2:172.21.15.174:3300/0 conn(0x7f72c4150940 msgr2=0x7f72c4159c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-06T05:06:32.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:32.802+0000 7f72c27fc700 1 --2- 172.21.15.98:0/3965240208 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f72c415a160 0x7f72c40ccdc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-06T05:06:32.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:32.802+0000 7f72c2ffd700 1 -- 172.21.15.98:0/3965240208 >> v2:172.21.15.174:3300/0 conn(0x7f72c4150940 msgr2=0x7f72c4159c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-06T05:06:32.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:32.802+0000 7f72c2ffd700 1 --2- 172.21.15.98:0/3965240208 >> v2:172.21.15.174:3300/0 conn(0x7f72c4150940 0x7f72c4159c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-10-06T05:06:32.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:32.802+0000 7f72c2ffd700 1 --2- 172.21.15.98:0/3965240208 >> v2:172.21.15.174:3300/0 conn(0x7f72c4150940 0x7f72c4159c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-06T05:06:33.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:33.003+0000 7f72c37fe700 1 --2- 172.21.15.98:0/3965240208 >> v2:172.21.15.98:3300/0 conn(0x7f72c40c5250 0x7f72c40cd300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:33.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:33.003+0000 7f72c27fc700 1 --2- 172.21.15.98:0/3965240208 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f72c415a160 0x7f72c40ccdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:33.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:33.003+0000 7f72c2ffd700 1 --2- 172.21.15.98:0/3965240208 >> v2:172.21.15.174:3300/0 conn(0x7f72c4150940 0x7f72c4159c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:33.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:33.003+0000 7f72c37fe700 1 -- 172.21.15.98:0/3965240208 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f72c415a160 msgr2=0x7f72c40ccdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:06:33.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:33.003+0000 7f72c37fe700 1 --2- 172.21.15.98:0/3965240208 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f72c415a160 0x7f72c40ccdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:33.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:33.003+0000 7f72c37fe700 1 -- 172.21.15.98:0/3965240208 >> v2:172.21.15.174:3300/0 conn(0x7f72c4150940 msgr2=0x7f72c4159c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:06:33.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:33.003+0000 7f72c37fe700 1 --2- 172.21.15.98:0/3965240208 >> v2:172.21.15.174:3300/0 conn(0x7f72c4150940 0x7f72c4159c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:33.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:33.003+0000 7f72c37fe700 1 -- 172.21.15.98:0/3965240208 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72c415a670 con 0x7f72c40c5250 2023-10-06T05:06:33.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:33.003+0000 7f72c37fe700 1 --2- 172.21.15.98:0/3965240208 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72c40c5250 0x7f72c40cd300 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f72c408ab60 tx=0x7f72b40091c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f64301d0f9504827 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:06:33.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:33.003+0000 7f72c1ffb700 1 -- 172.21.15.98:0/3965240208 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72b4007ad0 con 0x7f72c40c5250 2023-10-06T05:06:33.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:33.003+0000 7f72c1ffb700 1 -- 172.21.15.98:0/3965240208 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72b400f040 con 0x7f72c40c5250 2023-10-06T05:06:33.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:33.003+0000 7f72c1ffb700 1 -- 172.21.15.98:0/3965240208 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72b4007e80 con 0x7f72c40c5250 2023-10-06T05:06:33.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:33.004+0000 7f72c950e700 1 -- 172.21.15.98:0/3965240208 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72c40c5250 msgr2=0x7f72c40cd300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:33.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:33.004+0000 7f72c950e700 1 --2- 172.21.15.98:0/3965240208 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72c40c5250 0x7f72c40cd300 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f72c408ab60 tx=0x7f72b40091c0 comp rx=0 tx=0).stop 2023-10-06T05:06:33.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:33.004+0000 7f72c950e700 1 -- 172.21.15.98:0/3965240208 shutdown_connections 2023-10-06T05:06:33.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:33.004+0000 7f72c950e700 1 --2- 172.21.15.98:0/3965240208 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72c40c5250 0x7f72c40cd300 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:33.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:33.004+0000 7f72c950e700 1 --2- 172.21.15.98:0/3965240208 >> v2:172.21.15.174:3300/0 conn(0x7f72c4150940 0x7f72c4159c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:33.008 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:33.004+0000 7f72c950e700 1 --2- 172.21.15.98:0/3965240208 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f72c415a160 0x7f72c40ccdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:33.008 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:33.004+0000 7f72c950e700 1 -- 172.21.15.98:0/3965240208 >> 172.21.15.98:0/3965240208 conn(0x7f72c4099700 msgr2=0x7f72c406f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:06:33.008 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:33.004+0000 7f72c950e700 1 -- 172.21.15.98:0/3965240208 shutdown_connections 2023-10-06T05:06:33.008 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:33.004+0000 7f72c950e700 1 -- 172.21.15.98:0/3965240208 wait complete. 2023-10-06T05:06:33.010 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:33.005+0000 7f72c950e700 1 Processor -- start 2023-10-06T05:06:33.010 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:33.005+0000 7f72c950e700 1 -- start start 2023-10-06T05:06:33.010 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:33.006+0000 7f72c950e700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f72c40c5250 0x7f72c415d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:33.010 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:33.006+0000 7f72c950e700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f72c415a160 0x7f72c40b3c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:33.010 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:33.006+0000 7f72c950e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72c40b5f20 0x7f72c414fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:33.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:33.006+0000 7f72c950e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72c415b310 con 0x7f72c40b5f20 2023-10-06T05:06:33.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:33.006+0000 7f72c950e700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72c415b5f0 con 0x7f72c40c5250 2023-10-06T05:06:33.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:33.006+0000 7f72c950e700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72c415b8d0 con 0x7f72c415a160 2023-10-06T05:06:33.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:33.006+0000 7f72c37fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72c40b5f20 0x7f72c414fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:33.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:33.006+0000 7f72c37fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72c40b5f20 0x7f72c414fd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39118/0 (socket says 172.21.15.98:39118) 2023-10-06T05:06:33.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:33.006+0000 7f72c37fe700 1 -- 172.21.15.98:0/1888232354 learned_addr learned my addr 172.21.15.98:0/1888232354 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:06:33.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:33.006+0000 7f72c27fc700 1 --2- 172.21.15.98:0/1888232354 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f72c415a160 0x7f72c40b3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:33.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:33.006+0000 7f72c2ffd700 1 --2- 172.21.15.98:0/1888232354 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f72c40c5250 0x7f72c415d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:33.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:33.006+0000 7f72c37fe700 1 -- 172.21.15.98:0/1888232354 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f72c415a160 msgr2=0x7f72c40b3c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:33.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:33.006+0000 7f72c37fe700 1 --2- 172.21.15.98:0/1888232354 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f72c415a160 0x7f72c40b3c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:33.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:33.006+0000 7f72c37fe700 1 -- 172.21.15.98:0/1888232354 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f72c40c5250 msgr2=0x7f72c415d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:33.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:33.006+0000 7f72c37fe700 1 --2- 172.21.15.98:0/1888232354 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f72c40c5250 0x7f72c415d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:33.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:33.006+0000 7f72c37fe700 1 -- 172.21.15.98:0/1888232354 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72b4009760 con 0x7f72c40b5f20 2023-10-06T05:06:33.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:33.007+0000 7f72c37fe700 1 --2- 172.21.15.98:0/1888232354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72c40b5f20 0x7f72c414fd90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f72c40d14c0 tx=0x7f72b4013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:06:33.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:33.007+0000 7f72aaffd700 1 -- 172.21.15.98:0/1888232354 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72b4007e30 con 0x7f72c40b5f20 2023-10-06T05:06:33.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:33.007+0000 7f72aaffd700 1 -- 172.21.15.98:0/1888232354 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72b4013db0 con 0x7f72c40b5f20 2023-10-06T05:06:33.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:33.007+0000 7f72c950e700 1 -- 172.21.15.98:0/1888232354 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72c415bbb0 con 0x7f72c40b5f20 2023-10-06T05:06:33.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:33.007+0000 7f72aaffd700 1 -- 172.21.15.98:0/1888232354 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72b401e650 con 0x7f72c40b5f20 2023-10-06T05:06:33.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:33.007+0000 7f72c950e700 1 -- 172.21.15.98:0/1888232354 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72c40b4310 con 0x7f72c40b5f20 2023-10-06T05:06:33.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:33.007+0000 7f72aaffd700 1 -- 172.21.15.98:0/1888232354 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f72b401ea40 con 0x7f72c40b5f20 2023-10-06T05:06:33.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:33.007+0000 7f72aaffd700 1 -- 172.21.15.98:0/1888232354 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f72b4027980 con 0x7f72c40b5f20 2023-10-06T05:06:33.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:33.008+0000 7f72c950e700 1 -- 172.21.15.98:0/1888232354 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72c4064a90 con 0x7f72c40b5f20 2023-10-06T05:06:33.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:33.010+0000 7f72aaffd700 1 -- 172.21.15.98:0/1888232354 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f72b402c060 con 0x7f72c40b5f20 2023-10-06T05:06:33.102 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:33.100+0000 7f72c950e700 1 -- 172.21.15.98:0/1888232354 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f72c404bb40 con 0x7f72c40b5f20 2023-10-06T05:06:33.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:33.107+0000 7f72aaffd700 1 -- 172.21.15.98:0/1888232354 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f72b4035090 con 0x7f72c40b5f20 2023-10-06T05:06:33.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:33.108+0000 7f72c950e700 1 -- 172.21.15.98:0/1888232354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72c40b5f20 msgr2=0x7f72c414fd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:33.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:33.108+0000 7f72c950e700 1 --2- 172.21.15.98:0/1888232354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72c40b5f20 0x7f72c414fd90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f72c40d14c0 tx=0x7f72b4013000 comp rx=0 tx=0).stop 2023-10-06T05:06:33.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:33.108+0000 7f72c950e700 1 -- 172.21.15.98:0/1888232354 shutdown_connections 2023-10-06T05:06:33.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:33.108+0000 7f72c950e700 1 --2- 172.21.15.98:0/1888232354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72c40b5f20 0x7f72c414fd90 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:33.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:33.108+0000 7f72c950e700 1 --2- 172.21.15.98:0/1888232354 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f72c40c5250 0x7f72c415d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:33.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:33.108+0000 7f72c950e700 1 --2- 172.21.15.98:0/1888232354 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f72c415a160 0x7f72c40b3c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:33.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:33.108+0000 7f72c950e700 1 -- 172.21.15.98:0/1888232354 >> 172.21.15.98:0/1888232354 conn(0x7f72c4099700 msgr2=0x7f72c4159330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:06:33.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:33.108+0000 7f72c950e700 1 -- 172.21.15.98:0/1888232354 shutdown_connections 2023-10-06T05:06:33.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:33.108+0000 7f72c950e700 1 -- 172.21.15.98:0/1888232354 wait complete. 2023-10-06T05:06:33.111 INFO:teuthology.orchestra.run.smithi098.stderr:adjusted tunables profile to default 2023-10-06T05:06:33.122 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-10-06T05:06:33.122 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-10-06T05:06:33.123 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-10-06T05:06:33.147 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-10-06T05:06:33.147 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-10-06T05:06:33.211 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-10-06T05:06:33.211 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-10-06T05:06:33.277 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-10-06T05:06:33.277 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-10-06T05:06:33.341 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2023-10-06T05:06:33.341 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-10-06T05:06:33.366 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2023-10-06T05:06:33.367 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-10-06T05:06:33.431 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2023-10-06T05:06:33.431 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-10-06T05:06:33.495 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2023-10-06T05:06:33.495 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-10-06T05:06:33.559 DEBUG:teuthology.orchestra.run.smithi174:> sudo ceph --cluster ceph osd new d9495d95-da75-4c3b-8927-e9bae6a8b093 0 2023-10-06T05:06:33.716 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.714+0000 7f92349f2700 1 Processor -- start 2023-10-06T05:06:33.717 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.714+0000 7f92349f2700 1 -- start start 2023-10-06T05:06:33.722 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.715+0000 7f92349f2700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9230157290 0x7f923015a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:33.722 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.715+0000 7f92349f2700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f92300675b0 0x7f9230067a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:33.723 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.715+0000 7f92349f2700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f923015da20 con 0x7f9230156ec0 2023-10-06T05:06:33.723 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.715+0000 7f92349f2700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9230067f70 con 0x7f92300675b0 2023-10-06T05:06:33.723 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.715+0000 7f92349f2700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f923006c9a0 con 0x7f9230157290 2023-10-06T05:06:33.723 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.716+0000 7f922dd9b700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9230157290 0x7f923015a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:33.723 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.716+0000 7f922ed9d700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f92300675b0 0x7f9230067a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:33.723 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.716+0000 7f922dd9b700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9230157290 0x7f923015a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.174:38314/0 (socket says 172.21.15.174:38314) 2023-10-06T05:06:33.723 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.716+0000 7f922ed9d700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f92300675b0 0x7f9230067a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.174:36198/0 (socket says 172.21.15.174:36198) 2023-10-06T05:06:33.723 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.716+0000 7f922dd9b700 1 -- 172.21.15.174:0/2814503479 learned_addr learned my addr 172.21.15.174:0/2814503479 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-10-06T05:06:33.724 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.716+0000 7f922ed9d700 1 -- 172.21.15.174:0/2814503479 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9230157290 msgr2=0x7f923015a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:06:33.724 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.716+0000 7f922ed9d700 1 --2- 172.21.15.174:0/2814503479 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9230157290 0x7f923015a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:33.724 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.716+0000 7f922ed9d700 1 -- 172.21.15.174:0/2814503479 >> v1:172.21.15.98:6789/0 conn(0x7f9230156ec0 legacy=0x7f92300cad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:33.724 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.716+0000 7f922ed9d700 1 -- 172.21.15.174:0/2814503479 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92300cb650 con 0x7f92300675b0 2023-10-06T05:06:33.724 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.716+0000 7f922dd9b700 1 --2- 172.21.15.174:0/2814503479 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9230157290 0x7f923015a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T05:06:33.724 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.716+0000 7f922ed9d700 1 --2- 172.21.15.174:0/2814503479 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f92300675b0 0x7f9230067a30 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9228009e60 tx=0x7f9228009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7320f7bb264f53b server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:06:33.724 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.716+0000 7f922d59a700 1 -- 172.21.15.174:0/2814503479 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f922800e070 con 0x7f92300675b0 2023-10-06T05:06:33.724 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.716+0000 7f922d59a700 1 -- 172.21.15.174:0/2814503479 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f922800bb60 con 0x7f92300675b0 2023-10-06T05:06:33.724 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.716+0000 7f922d59a700 1 -- 172.21.15.174:0/2814503479 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9228012430 con 0x7f92300675b0 2023-10-06T05:06:33.725 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.717+0000 7f92349f2700 1 -- 172.21.15.174:0/2814503479 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f92300675b0 msgr2=0x7f9230067a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:33.725 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.717+0000 7f92349f2700 1 --2- 172.21.15.174:0/2814503479 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f92300675b0 0x7f9230067a30 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9228009e60 tx=0x7f9228009780 comp rx=0 tx=0).stop 2023-10-06T05:06:33.725 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.717+0000 7f92349f2700 1 -- 172.21.15.174:0/2814503479 shutdown_connections 2023-10-06T05:06:33.725 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.717+0000 7f92349f2700 1 --2- 172.21.15.174:0/2814503479 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f92300675b0 0x7f9230067a30 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:33.725 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.717+0000 7f92349f2700 1 --2- 172.21.15.174:0/2814503479 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9230157290 0x7f923015a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:33.725 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.717+0000 7f92349f2700 1 -- 172.21.15.174:0/2814503479 >> 172.21.15.174:0/2814503479 conn(0x7f92300cd1a0 msgr2=0x7f92300cd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:06:33.725 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.717+0000 7f92349f2700 1 -- 172.21.15.174:0/2814503479 shutdown_connections 2023-10-06T05:06:33.725 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.717+0000 7f92349f2700 1 -- 172.21.15.174:0/2814503479 wait complete. 2023-10-06T05:06:33.726 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.718+0000 7f92349f2700 1 Processor -- start 2023-10-06T05:06:33.726 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.718+0000 7f92349f2700 1 -- start start 2023-10-06T05:06:33.726 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.719+0000 7f92349f2700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f92300675b0 0x7f9230155540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:33.726 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.719+0000 7f92349f2700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9230156ec0 0x7f9230158640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:33.726 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.719+0000 7f92349f2700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9230158b80 0x7f9230169b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:33.726 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.719+0000 7f92349f2700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9230159020 con 0x7f9230156ec0 2023-10-06T05:06:33.726 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.719+0000 7f92349f2700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9230155a80 con 0x7f92300675b0 2023-10-06T05:06:33.727 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.719+0000 7f92349f2700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f923016a950 con 0x7f9230158b80 2023-10-06T05:06:33.727 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.719+0000 7f922e59c700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f92300675b0 0x7f9230155540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:33.727 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.719+0000 7f922ed9d700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9230158b80 0x7f9230169b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:33.727 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.719+0000 7f922e59c700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f92300675b0 0x7f9230155540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.174:36208/0 (socket says 172.21.15.174:36208) 2023-10-06T05:06:33.727 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.719+0000 7f922e59c700 1 -- 172.21.15.174:0/3725362515 learned_addr learned my addr 172.21.15.174:0/3725362515 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-10-06T05:06:33.727 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.719+0000 7f922dd9b700 1 --2- 172.21.15.174:0/3725362515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9230156ec0 0x7f9230158640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:33.727 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.719+0000 7f922e59c700 1 -- 172.21.15.174:0/3725362515 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9230158b80 msgr2=0x7f9230169b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:33.727 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.719+0000 7f922e59c700 1 --2- 172.21.15.174:0/3725362515 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9230158b80 0x7f9230169b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:33.727 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.719+0000 7f922e59c700 1 -- 172.21.15.174:0/3725362515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9230156ec0 msgr2=0x7f9230158640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:33.728 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.719+0000 7f922e59c700 1 --2- 172.21.15.174:0/3725362515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9230156ec0 0x7f9230158640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:33.728 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.719+0000 7f922e59c700 1 -- 172.21.15.174:0/3725362515 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92240094f0 con 0x7f92300675b0 2023-10-06T05:06:33.728 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.719+0000 7f922ed9d700 1 --2- 172.21.15.174:0/3725362515 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9230158b80 0x7f9230169b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T05:06:33.728 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.719+0000 7f922e59c700 1 --2- 172.21.15.174:0/3725362515 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f92300675b0 0x7f9230155540 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f922000c590 tx=0x7f922000c8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:06:33.728 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.719+0000 7f921e7fc700 1 -- 172.21.15.174:0/3725362515 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9220011590 con 0x7f92300675b0 2023-10-06T05:06:33.728 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.719+0000 7f921e7fc700 1 -- 172.21.15.174:0/3725362515 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9220011710 con 0x7f92300675b0 2023-10-06T05:06:33.728 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.719+0000 7f92349f2700 1 -- 172.21.15.174:0/3725362515 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9228009a20 con 0x7f92300675b0 2023-10-06T05:06:33.728 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.720+0000 7f921e7fc700 1 -- 172.21.15.174:0/3725362515 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9220011ad0 con 0x7f92300675b0 2023-10-06T05:06:33.728 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.720+0000 7f92349f2700 1 -- 172.21.15.174:0/3725362515 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f923016b1e0 con 0x7f92300675b0 2023-10-06T05:06:33.854 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.720+0000 7f921e7fc700 1 -- 172.21.15.174:0/3725362515 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f922001c6c0 con 0x7f92300675b0 2023-10-06T05:06:33.854 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.720+0000 7f921e7fc700 1 -- 172.21.15.174:0/3725362515 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 721+0+0 (secure 0 0 0) 0x7f9220012770 con 0x7f92300675b0 2023-10-06T05:06:33.854 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.720+0000 7f92349f2700 1 -- 172.21.15.174:0/3725362515 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9230064a90 con 0x7f92300675b0 2023-10-06T05:06:33.855 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.722+0000 7f921e7fc700 1 -- 172.21.15.174:0/3725362515 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f9220017070 con 0x7f92300675b0 2023-10-06T05:06:33.856 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.804+0000 7f92349f2700 1 -- 172.21.15.174:0/3725362515 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd new", "uuid": "d9495d95-da75-4c3b-8927-e9bae6a8b093", "id": 0} v 0) v1 -- 0x7f923006cb70 con 0x7f92300675b0 2023-10-06T05:06:33.856 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.810+0000 7f921e7fc700 1 -- 172.21.15.174:0/3725362515 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d9495d95-da75-4c3b-8927-e9bae6a8b093", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f922001cb50 con 0x7f92300675b0 2023-10-06T05:06:33.856 INFO:teuthology.orchestra.run.smithi174.stdout:0 2023-10-06T05:06:33.858 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.811+0000 7f92349f2700 1 -- 172.21.15.174:0/3725362515 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f92300675b0 msgr2=0x7f9230155540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:33.858 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.811+0000 7f92349f2700 1 --2- 172.21.15.174:0/3725362515 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f92300675b0 0x7f9230155540 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f922000c590 tx=0x7f922000c8a0 comp rx=0 tx=0).stop 2023-10-06T05:06:33.858 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.811+0000 7f92349f2700 1 -- 172.21.15.174:0/3725362515 shutdown_connections 2023-10-06T05:06:33.858 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.811+0000 7f92349f2700 1 --2- 172.21.15.174:0/3725362515 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9230158b80 0x7f9230169b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:33.859 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.811+0000 7f92349f2700 1 --2- 172.21.15.174:0/3725362515 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f92300675b0 0x7f9230155540 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:33.859 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.811+0000 7f92349f2700 1 --2- 172.21.15.174:0/3725362515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9230156ec0 0x7f9230158640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:33.859 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.811+0000 7f92349f2700 1 -- 172.21.15.174:0/3725362515 >> 172.21.15.174:0/3725362515 conn(0x7f92300cd1a0 msgr2=0x7f92300c1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:06:33.859 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.811+0000 7f92349f2700 1 -- 172.21.15.174:0/3725362515 shutdown_connections 2023-10-06T05:06:33.859 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.811+0000 7f92349f2700 1 -- 172.21.15.174:0/3725362515 wait complete. 2023-10-06T05:06:33.859 DEBUG:teuthology.orchestra.run.smithi174:> sudo ceph --cluster ceph osd new f25ba5be-d65e-4707-a134-154092deb754 1 2023-10-06T05:06:33.985 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.980+0000 7f42438af700 1 Processor -- start 2023-10-06T05:06:33.985 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.980+0000 7f42438af700 1 -- start start 2023-10-06T05:06:33.992 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.981+0000 7f42438af700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f423c150940 0x7f423c159c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:33.992 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.981+0000 7f42438af700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f423c15a160 0x7f423c0ccdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:33.992 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.981+0000 7f42438af700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f423c15ac30 con 0x7f423c0c5250 2023-10-06T05:06:33.992 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.981+0000 7f42438af700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f423c0cb6a0 con 0x7f423c15a160 2023-10-06T05:06:33.993 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.981+0000 7f42438af700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f423c0cb090 con 0x7f423c150940 2023-10-06T05:06:33.993 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.981+0000 7f424164b700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f423c150940 0x7f423c159c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:33.993 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.981+0000 7f4241e4c700 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f423c0c5250 0x7f423c0b2050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.174:52724/0 (socket says 172.21.15.174:52724) 2023-10-06T05:06:33.993 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.982+0000 7f4241e4c700 1 -- 172.21.15.174:0/1623181723 learned_addr learned my addr 172.21.15.174:0/1623181723 (peer_addr_for_me v1:172.21.15.174:0/0) 2023-10-06T05:06:33.993 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.982+0000 7f4240e4a700 1 --2- 172.21.15.174:0/1623181723 >> v2:172.21.15.98:3300/0 conn(0x7f423c15a160 0x7f423c0ccdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:33.993 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.982+0000 7f4233fff700 1 -- 172.21.15.174:0/1623181723 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2206097103 0 0) 0x7f4234001490 con 0x7f423c0c5250 2023-10-06T05:06:33.994 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.982+0000 7f424164b700 1 -- 172.21.15.174:0/1623181723 >> v2:172.21.15.98:3300/0 conn(0x7f423c15a160 msgr2=0x7f423c0ccdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:06:33.994 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.982+0000 7f424164b700 1 --2- 172.21.15.174:0/1623181723 >> v2:172.21.15.98:3300/0 conn(0x7f423c15a160 0x7f423c0ccdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:33.994 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.982+0000 7f424164b700 1 -- 172.21.15.174:0/1623181723 >> v1:172.21.15.174:6789/0 conn(0x7f423c0c5250 legacy=0x7f423c0b2050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:33.994 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.982+0000 7f424164b700 1 -- 172.21.15.174:0/1623181723 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f423c15a670 con 0x7f423c150940 2023-10-06T05:06:33.994 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.982+0000 7f424164b700 1 --2- 172.21.15.174:0/1623181723 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f423c150940 0x7f423c159c20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f42380098d0 tx=0x7f4238009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe14c7cf944508e3 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:06:33.995 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.982+0000 7f4233fff700 1 -- 172.21.15.174:0/1623181723 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4238005c10 con 0x7f423c150940 2023-10-06T05:06:33.995 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.982+0000 7f4233fff700 1 -- 172.21.15.174:0/1623181723 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4238005d70 con 0x7f423c150940 2023-10-06T05:06:33.995 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.982+0000 7f4233fff700 1 -- 172.21.15.174:0/1623181723 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4238006280 con 0x7f423c150940 2023-10-06T05:06:33.995 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.983+0000 7f42438af700 1 -- 172.21.15.174:0/1623181723 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f423c150940 msgr2=0x7f423c159c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:33.996 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.983+0000 7f42438af700 1 --2- 172.21.15.174:0/1623181723 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f423c150940 0x7f423c159c20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f42380098d0 tx=0x7f4238009120 comp rx=0 tx=0).stop 2023-10-06T05:06:33.996 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.983+0000 7f42438af700 1 -- 172.21.15.174:0/1623181723 shutdown_connections 2023-10-06T05:06:33.996 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.983+0000 7f42438af700 1 --2- 172.21.15.174:0/1623181723 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f423c150940 0x7f423c159c20 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:33.996 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.983+0000 7f42438af700 1 --2- 172.21.15.174:0/1623181723 >> v2:172.21.15.98:3300/0 conn(0x7f423c15a160 0x7f423c0ccdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:33.996 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.983+0000 7f42438af700 1 -- 172.21.15.174:0/1623181723 >> 172.21.15.174:0/1623181723 conn(0x7f423c099700 msgr2=0x7f423c06f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:06:33.997 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.983+0000 7f42438af700 1 -- 172.21.15.174:0/1623181723 shutdown_connections 2023-10-06T05:06:33.997 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.983+0000 7f42438af700 1 -- 172.21.15.174:0/1623181723 wait complete. 2023-10-06T05:06:33.997 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.985+0000 7f42438af700 1 Processor -- start 2023-10-06T05:06:33.997 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.985+0000 7f42438af700 1 -- start start 2023-10-06T05:06:33.997 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.986+0000 7f42438af700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f423c0c5250 0x7f423c16aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:33.998 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.986+0000 7f42438af700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f423c15a160 0x7f423c0c9dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:33.998 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.986+0000 7f42438af700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f423c156a10 0x7f423c0cbb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:33.998 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.986+0000 7f42438af700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f423c169870 con 0x7f423c0c5250 2023-10-06T05:06:33.998 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.986+0000 7f42438af700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f423c0c76e0 con 0x7f423c156a10 2023-10-06T05:06:33.999 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.986+0000 7f42438af700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f423c0c79c0 con 0x7f423c15a160 2023-10-06T05:06:33.999 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.986+0000 7f4240e4a700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f423c15a160 0x7f423c0c9dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:33.999 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.986+0000 7f4241e4c700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f423c156a10 0x7f423c0cbb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:33.999 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.986+0000 7f4240e4a700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f423c15a160 0x7f423c0c9dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.174:38322/0 (socket says 172.21.15.174:38322) 2023-10-06T05:06:33.999 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.986+0000 7f424164b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f423c0c5250 0x7f423c16aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:33.999 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.986+0000 7f4240e4a700 1 -- 172.21.15.174:0/4269318353 learned_addr learned my addr 172.21.15.174:0/4269318353 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-10-06T05:06:33.999 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.987+0000 7f4240e4a700 1 -- 172.21.15.174:0/4269318353 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f423c156a10 msgr2=0x7f423c0cbb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:34.000 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.987+0000 7f4240e4a700 1 --2- 172.21.15.174:0/4269318353 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f423c156a10 0x7f423c0cbb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:34.000 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.987+0000 7f4240e4a700 1 -- 172.21.15.174:0/4269318353 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f423c0c5250 msgr2=0x7f423c16aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:34.000 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.987+0000 7f4240e4a700 1 --2- 172.21.15.174:0/4269318353 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f423c0c5250 0x7f423c16aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:34.000 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.987+0000 7f4240e4a700 1 -- 172.21.15.174:0/4269318353 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42380093c0 con 0x7f423c15a160 2023-10-06T05:06:34.000 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.987+0000 7f4241e4c700 1 --2- 172.21.15.174:0/4269318353 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f423c156a10 0x7f423c0cbb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T05:06:34.000 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.987+0000 7f424164b700 1 --2- 172.21.15.174:0/4269318353 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f423c0c5250 0x7f423c16aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T05:06:34.001 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.987+0000 7f4240e4a700 1 --2- 172.21.15.174:0/4269318353 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f423c15a160 0x7f423c0c9dc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f422c00e900 tx=0x7f422c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:06:34.001 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.987+0000 7f42317fa700 1 -- 172.21.15.174:0/4269318353 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f422c013680 con 0x7f423c15a160 2023-10-06T05:06:34.001 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.987+0000 7f42317fa700 1 -- 172.21.15.174:0/4269318353 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f422c0137e0 con 0x7f423c15a160 2023-10-06T05:06:34.001 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.987+0000 7f42438af700 1 -- 172.21.15.174:0/4269318353 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f423c0c7ca0 con 0x7f423c15a160 2023-10-06T05:06:34.001 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.987+0000 7f42317fa700 1 -- 172.21.15.174:0/4269318353 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f422c013ba0 con 0x7f423c15a160 2023-10-06T05:06:34.002 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.987+0000 7f42438af700 1 -- 172.21.15.174:0/4269318353 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f423c16af80 con 0x7f423c15a160 2023-10-06T05:06:34.002 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.987+0000 7f42317fa700 1 -- 172.21.15.174:0/4269318353 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f422c005c80 con 0x7f423c15a160 2023-10-06T05:06:34.002 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.987+0000 7f42317fa700 1 -- 172.21.15.174:0/4269318353 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 836+0+0 (secure 0 0 0) 0x7f422c01c450 con 0x7f423c15a160 2023-10-06T05:06:34.002 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.988+0000 7f42438af700 1 -- 172.21.15.174:0/4269318353 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f423c1699e0 con 0x7f423c15a160 2023-10-06T05:06:34.003 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:33.992+0000 7f42317fa700 1 -- 172.21.15.174:0/4269318353 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f423c1699e0 con 0x7f423c15a160 2023-10-06T05:06:34.076 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.074+0000 7f42438af700 1 -- 172.21.15.174:0/4269318353 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "osd new", "uuid": "f25ba5be-d65e-4707-a134-154092deb754", "id": 1} v 0) v1 -- 0x7f423c064a90 con 0x7f423c15a160 2023-10-06T05:06:34.081 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.079+0000 7f42317fa700 1 -- 172.21.15.174:0/4269318353 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f25ba5be-d65e-4707-a134-154092deb754", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f422c021060 con 0x7f423c15a160 2023-10-06T05:06:34.081 INFO:teuthology.orchestra.run.smithi174.stdout:1 2023-10-06T05:06:34.083 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.080+0000 7f42438af700 1 -- 172.21.15.174:0/4269318353 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f423c15a160 msgr2=0x7f423c0c9dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:34.083 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.080+0000 7f42438af700 1 --2- 172.21.15.174:0/4269318353 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f423c15a160 0x7f423c0c9dc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f422c00e900 tx=0x7f422c0091a0 comp rx=0 tx=0).stop 2023-10-06T05:06:34.083 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.080+0000 7f42438af700 1 -- 172.21.15.174:0/4269318353 shutdown_connections 2023-10-06T05:06:34.083 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.080+0000 7f42438af700 1 --2- 172.21.15.174:0/4269318353 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f423c156a10 0x7f423c0cbb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:34.083 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.080+0000 7f42438af700 1 --2- 172.21.15.174:0/4269318353 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f423c0c5250 0x7f423c16aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:34.083 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.080+0000 7f42438af700 1 --2- 172.21.15.174:0/4269318353 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f423c15a160 0x7f423c0c9dc0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:34.083 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.080+0000 7f42438af700 1 -- 172.21.15.174:0/4269318353 >> 172.21.15.174:0/4269318353 conn(0x7f423c099700 msgr2=0x7f423c159850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:06:34.083 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.080+0000 7f42438af700 1 -- 172.21.15.174:0/4269318353 shutdown_connections 2023-10-06T05:06:34.083 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.080+0000 7f42438af700 1 -- 172.21.15.174:0/4269318353 wait complete. 2023-10-06T05:06:34.094 DEBUG:teuthology.orchestra.run.smithi174:> sudo ceph --cluster ceph osd new 5ab9e028-945c-43e2-8c95-3b0b4cf38e39 2 2023-10-06T05:06:34.179 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-10-06T05:06:34.175+0000 7f983cd62180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-06T05:06:34.196 INFO:tasks.ceph.mgr.x.smithi174.stderr:2023-10-06T05:06:34.188+0000 7fcbfec4d180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-06T05:06:34.226 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.219+0000 7fce3523d700 1 Processor -- start 2023-10-06T05:06:34.226 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.219+0000 7fce3523d700 1 -- start start 2023-10-06T05:06:34.227 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.220+0000 7fce3523d700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fce30157290 0x7fce3015a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:34.227 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.220+0000 7fce3523d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce3015da20 con 0x7fce300675b0 2023-10-06T05:06:34.227 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.220+0000 7fce3523d700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce300cb650 con 0x7fce30156ec0 2023-10-06T05:06:34.227 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.220+0000 7fce3523d700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce3015d0f0 con 0x7fce30157290 2023-10-06T05:06:34.227 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.220+0000 7fce2e59c700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fce30157290 0x7fce3015a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:34.227 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.220+0000 7fce2ed9d700 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fce30156ec0 0x7fce300cad70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.174:52728/0 (socket says 172.21.15.174:52728) 2023-10-06T05:06:34.227 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.220+0000 7fce2ed9d700 1 -- 172.21.15.174:0/1376320604 learned_addr learned my addr 172.21.15.174:0/1376320604 (peer_addr_for_me v1:172.21.15.174:0/0) 2023-10-06T05:06:34.227 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.221+0000 7fce2dd9b700 1 -- 172.21.15.174:0/1376320604 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3128609079 0 0) 0x7fce300cb650 con 0x7fce30156ec0 2023-10-06T05:06:34.227 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.221+0000 7fce2dd9b700 1 -- 172.21.15.174:0/1376320604 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce18003570 con 0x7fce30156ec0 2023-10-06T05:06:34.228 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.221+0000 7fce2e59c700 1 -- 172.21.15.174:0/1376320604 >> v1:172.21.15.174:6789/0 conn(0x7fce30156ec0 legacy=0x7fce300cad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:34.228 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.221+0000 7fce2e59c700 1 -- 172.21.15.174:0/1376320604 >> v1:172.21.15.98:6789/0 conn(0x7fce300675b0 legacy=0x7fce3006b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:34.228 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.221+0000 7fce2dd9b700 1 -- 172.21.15.174:0/1376320604 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2039942980 0 0) 0x7fce3015da20 con 0x7fce300675b0 2023-10-06T05:06:34.228 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.221+0000 7fce2e59c700 1 -- 172.21.15.174:0/1376320604 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce3015d3d0 con 0x7fce30157290 2023-10-06T05:06:34.228 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.221+0000 7fce2e59c700 1 --2- 172.21.15.174:0/1376320604 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fce30157290 0x7fce3015a6e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fce24009910 tx=0x7fce240091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=751ecc6d1d849ee3 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:06:34.228 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.221+0000 7fce2dd9b700 1 -- 172.21.15.174:0/1376320604 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce2400b680 con 0x7fce30157290 2023-10-06T05:06:34.228 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.221+0000 7fce2dd9b700 1 -- 172.21.15.174:0/1376320604 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce2400b7e0 con 0x7fce30157290 2023-10-06T05:06:34.228 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.221+0000 7fce2dd9b700 1 -- 172.21.15.174:0/1376320604 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce2400bba0 con 0x7fce30157290 2023-10-06T05:06:34.228 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.222+0000 7fce3523d700 1 -- 172.21.15.174:0/1376320604 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fce30157290 msgr2=0x7fce3015a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:34.228 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.222+0000 7fce3523d700 1 --2- 172.21.15.174:0/1376320604 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fce30157290 0x7fce3015a6e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fce24009910 tx=0x7fce240091a0 comp rx=0 tx=0).stop 2023-10-06T05:06:34.233 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.222+0000 7fce3523d700 1 -- 172.21.15.174:0/1376320604 shutdown_connections 2023-10-06T05:06:34.233 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.222+0000 7fce3523d700 1 --2- 172.21.15.174:0/1376320604 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fce30157290 0x7fce3015a6e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:34.233 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.222+0000 7fce3523d700 1 -- 172.21.15.174:0/1376320604 >> 172.21.15.174:0/1376320604 conn(0x7fce300cd1a0 msgr2=0x7fce300cd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:06:34.233 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.222+0000 7fce3523d700 1 -- 172.21.15.174:0/1376320604 shutdown_connections 2023-10-06T05:06:34.234 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.222+0000 7fce3523d700 1 -- 172.21.15.174:0/1376320604 wait complete. 2023-10-06T05:06:34.234 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.223+0000 7fce3523d700 1 Processor -- start 2023-10-06T05:06:34.234 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.223+0000 7fce3523d700 1 -- start start 2023-10-06T05:06:34.234 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.224+0000 7fce3523d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce300675b0 0x7fce30151ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:34.234 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.224+0000 7fce3523d700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fce30156ec0 0x7fce300b0910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:34.234 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.224+0000 7fce3523d700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fce3016a4b0 0x7fce300b2300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:34.234 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.224+0000 7fce3523d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce300c8aa0 con 0x7fce300675b0 2023-10-06T05:06:34.234 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.224+0000 7fce3523d700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce300b0e50 con 0x7fce30156ec0 2023-10-06T05:06:34.234 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.224+0000 7fce3523d700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce300c8ec0 con 0x7fce3016a4b0 2023-10-06T05:06:34.234 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.224+0000 7fce2e59c700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fce30156ec0 0x7fce300b0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:34.235 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.224+0000 7fce2f59e700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fce3016a4b0 0x7fce300b2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:34.235 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.224+0000 7fce2e59c700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fce30156ec0 0x7fce300b0910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.174:36228/0 (socket says 172.21.15.174:36228) 2023-10-06T05:06:34.235 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.224+0000 7fce2e59c700 1 -- 172.21.15.174:0/3593765389 learned_addr learned my addr 172.21.15.174:0/3593765389 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-10-06T05:06:34.235 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.224+0000 7fce2ed9d700 1 --2- 172.21.15.174:0/3593765389 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce300675b0 0x7fce30151ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:34.235 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.224+0000 7fce2e59c700 1 -- 172.21.15.174:0/3593765389 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fce3016a4b0 msgr2=0x7fce300b2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:34.235 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.224+0000 7fce2e59c700 1 --2- 172.21.15.174:0/3593765389 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fce3016a4b0 0x7fce300b2300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:34.235 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.224+0000 7fce2e59c700 1 -- 172.21.15.174:0/3593765389 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce300675b0 msgr2=0x7fce30151ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:34.236 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.224+0000 7fce2e59c700 1 --2- 172.21.15.174:0/3593765389 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce300675b0 0x7fce30151ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:34.236 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.224+0000 7fce2e59c700 1 -- 172.21.15.174:0/3593765389 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce24009420 con 0x7fce30156ec0 2023-10-06T05:06:34.236 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.224+0000 7fce2f59e700 1 --2- 172.21.15.174:0/3593765389 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fce3016a4b0 0x7fce300b2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T05:06:34.236 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.224+0000 7fce2e59c700 1 --2- 172.21.15.174:0/3593765389 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fce30156ec0 0x7fce300b0910 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fce24009fb0 tx=0x7fce24017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:06:34.236 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.224+0000 7fce16ffd700 1 -- 172.21.15.174:0/3593765389 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce2400bb60 con 0x7fce30156ec0 2023-10-06T05:06:34.236 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.224+0000 7fce16ffd700 1 -- 172.21.15.174:0/3593765389 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce24000c90 con 0x7fce30156ec0 2023-10-06T05:06:34.236 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.224+0000 7fce16ffd700 1 -- 172.21.15.174:0/3593765389 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce24027680 con 0x7fce30156ec0 2023-10-06T05:06:34.237 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.224+0000 7fce3523d700 1 -- 172.21.15.174:0/3593765389 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce300c84e0 con 0x7fce30156ec0 2023-10-06T05:06:34.237 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.224+0000 7fce3523d700 1 -- 172.21.15.174:0/3593765389 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce300c91a0 con 0x7fce30156ec0 2023-10-06T05:06:34.237 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.225+0000 7fce16ffd700 1 -- 172.21.15.174:0/3593765389 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fce240279f0 con 0x7fce30156ec0 2023-10-06T05:06:34.237 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.225+0000 7fce16ffd700 1 -- 172.21.15.174:0/3593765389 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 951+0+0 (secure 0 0 0) 0x7fce24011940 con 0x7fce30156ec0 2023-10-06T05:06:34.237 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.225+0000 7fce3523d700 1 -- 172.21.15.174:0/3593765389 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce300c8c10 con 0x7fce30156ec0 2023-10-06T05:06:34.237 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.229+0000 7fce16ffd700 1 -- 172.21.15.174:0/3593765389 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fce2402c020 con 0x7fce30156ec0 2023-10-06T05:06:34.309 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-10-06T05:06:34.307+0000 7f983cd62180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-06T05:06:34.313 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.312+0000 7fce3523d700 1 -- 172.21.15.174:0/3593765389 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd new", "uuid": "5ab9e028-945c-43e2-8c95-3b0b4cf38e39", "id": 2} v 0) v1 -- 0x7fce30064a90 con 0x7fce30156ec0 2023-10-06T05:06:34.321 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.320+0000 7fce16ffd700 1 -- 172.21.15.174:0/3593765389 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5ab9e028-945c-43e2-8c95-3b0b4cf38e39", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7fce24015020 con 0x7fce30156ec0 2023-10-06T05:06:34.322 INFO:teuthology.orchestra.run.smithi174.stdout:2 2023-10-06T05:06:34.323 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.321+0000 7fce3523d700 1 -- 172.21.15.174:0/3593765389 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fce30156ec0 msgr2=0x7fce300b0910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:34.323 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.321+0000 7fce3523d700 1 --2- 172.21.15.174:0/3593765389 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fce30156ec0 0x7fce300b0910 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fce24009fb0 tx=0x7fce24017000 comp rx=0 tx=0).stop 2023-10-06T05:06:34.323 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.321+0000 7fce3523d700 1 -- 172.21.15.174:0/3593765389 shutdown_connections 2023-10-06T05:06:34.323 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.321+0000 7fce3523d700 1 --2- 172.21.15.174:0/3593765389 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fce3016a4b0 0x7fce300b2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:34.324 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.321+0000 7fce3523d700 1 --2- 172.21.15.174:0/3593765389 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce300675b0 0x7fce30151ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:34.324 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.321+0000 7fce3523d700 1 --2- 172.21.15.174:0/3593765389 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fce30156ec0 0x7fce300b0910 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:34.324 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.321+0000 7fce3523d700 1 -- 172.21.15.174:0/3593765389 >> 172.21.15.174:0/3593765389 conn(0x7fce300cd1a0 msgr2=0x7fce30154d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:06:34.324 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.321+0000 7fce3523d700 1 -- 172.21.15.174:0/3593765389 shutdown_connections 2023-10-06T05:06:34.324 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.321+0000 7fce3523d700 1 -- 172.21.15.174:0/3593765389 wait complete. 2023-10-06T05:06:34.332 INFO:tasks.ceph.mgr.x.smithi174.stderr:2023-10-06T05:06:34.328+0000 7fcbfec4d180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-06T05:06:34.335 DEBUG:teuthology.orchestra.run.smithi174:> sudo ceph --cluster ceph osd new 3bb803ff-1738-46bf-be62-2267bf378822 3 2023-10-06T05:06:34.388 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-10-06T05:06:34.383+0000 7f983cd62180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-06T05:06:34.422 INFO:tasks.ceph.mgr.x.smithi174.stderr:2023-10-06T05:06:34.408+0000 7fcbfec4d180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-06T05:06:34.469 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.466+0000 7f9a4ae98700 1 Processor -- start 2023-10-06T05:06:34.470 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.466+0000 7f9a4ae98700 1 -- start start 2023-10-06T05:06:34.470 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.467+0000 7f9a4ae98700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9a4406da40 0x7f9a440b1080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:34.470 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.467+0000 7f9a4ae98700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a4415ab50 con 0x7f9a4406d670 2023-10-06T05:06:34.470 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.467+0000 7f9a4ae98700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a4415d650 con 0x7f9a4406ce10 2023-10-06T05:06:34.470 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.467+0000 7f9a4ae98700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a4415a590 con 0x7f9a4406da40 2023-10-06T05:06:34.470 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.467+0000 7f9a49435700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9a4406da40 0x7f9a440b1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:34.470 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.467+0000 7f9a43fff700 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f9a4406ce10 0x7f9a44159af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.174:52734/0 (socket says 172.21.15.174:52734) 2023-10-06T05:06:34.470 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.467+0000 7f9a43fff700 1 -- 172.21.15.174:0/1288614652 learned_addr learned my addr 172.21.15.174:0/1288614652 (peer_addr_for_me v1:172.21.15.174:0/0) 2023-10-06T05:06:34.470 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.467+0000 7f9a437fe700 1 -- 172.21.15.174:0/1288614652 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2763480095 0 0) 0x7f9a4415d650 con 0x7f9a4406ce10 2023-10-06T05:06:34.471 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.467+0000 7f9a49435700 1 -- 172.21.15.174:0/1288614652 >> v1:172.21.15.174:6789/0 conn(0x7f9a4406ce10 legacy=0x7f9a44159af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:34.471 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.467+0000 7f9a49435700 1 -- 172.21.15.174:0/1288614652 >> v1:172.21.15.98:6789/0 conn(0x7f9a4406d670 legacy=0x7f9a4415dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:34.471 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.467+0000 7f9a49435700 1 -- 172.21.15.174:0/1288614652 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a4415a870 con 0x7f9a4406da40 2023-10-06T05:06:34.471 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.467+0000 7f9a49435700 1 --2- 172.21.15.174:0/1288614652 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9a4406da40 0x7f9a440b1080 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c009ec0 tx=0x7f9a3c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67959e108123d4c4 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:06:34.471 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.467+0000 7f9a437fe700 1 -- 172.21.15.174:0/1288614652 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a3c00e070 con 0x7f9a4406da40 2023-10-06T05:06:34.471 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.468+0000 7f9a437fe700 1 -- 172.21.15.174:0/1288614652 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a3c00bbf0 con 0x7f9a4406da40 2023-10-06T05:06:34.471 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.468+0000 7f9a437fe700 1 -- 172.21.15.174:0/1288614652 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a3c012430 con 0x7f9a4406da40 2023-10-06T05:06:34.471 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.468+0000 7f9a4ae98700 1 -- 172.21.15.174:0/1288614652 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9a4406da40 msgr2=0x7f9a440b1080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:34.471 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.468+0000 7f9a4ae98700 1 --2- 172.21.15.174:0/1288614652 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9a4406da40 0x7f9a440b1080 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c009ec0 tx=0x7f9a3c009780 comp rx=0 tx=0).stop 2023-10-06T05:06:34.471 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.468+0000 7f9a4ae98700 1 -- 172.21.15.174:0/1288614652 shutdown_connections 2023-10-06T05:06:34.471 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.468+0000 7f9a4ae98700 1 --2- 172.21.15.174:0/1288614652 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9a4406da40 0x7f9a440b1080 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:34.472 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.468+0000 7f9a4ae98700 1 -- 172.21.15.174:0/1288614652 >> 172.21.15.174:0/1288614652 conn(0x7f9a4406b600 msgr2=0x7f9a4406ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:06:34.472 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.468+0000 7f9a4ae98700 1 -- 172.21.15.174:0/1288614652 shutdown_connections 2023-10-06T05:06:34.472 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.468+0000 7f9a4ae98700 1 -- 172.21.15.174:0/1288614652 wait complete. 2023-10-06T05:06:34.474 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.469+0000 7f9a4ae98700 1 Processor -- start 2023-10-06T05:06:34.474 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.470+0000 7f9a4ae98700 1 -- start start 2023-10-06T05:06:34.474 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.470+0000 7f9a4ae98700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9a4406d670 0x7f9a440b06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:34.474 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.470+0000 7f9a4ae98700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a4406da40 0x7f9a440adb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:34.475 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.470+0000 7f9a4ae98700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9a44147c60 0x7f9a44155aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:34.475 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.470+0000 7f9a4ae98700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a440ae390 con 0x7f9a4406da40 2023-10-06T05:06:34.475 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.470+0000 7f9a4ae98700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a440ae670 con 0x7f9a4406d670 2023-10-06T05:06:34.475 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.470+0000 7f9a4ae98700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a440ae950 con 0x7f9a44147c60 2023-10-06T05:06:34.475 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.470+0000 7f9a49435700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9a44147c60 0x7f9a44155aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:34.475 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.470+0000 7f9a48c34700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9a4406d670 0x7f9a440b06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:34.475 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.470+0000 7f9a49435700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9a44147c60 0x7f9a44155aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.174:38370/0 (socket says 172.21.15.174:38370) 2023-10-06T05:06:34.475 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.470+0000 7f9a49435700 1 -- 172.21.15.174:0/1618445465 learned_addr learned my addr 172.21.15.174:0/1618445465 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-10-06T05:06:34.475 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.470+0000 7f9a43fff700 1 --2- 172.21.15.174:0/1618445465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a4406da40 0x7f9a440adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:34.475 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.471+0000 7f9a48c34700 1 -- 172.21.15.174:0/1618445465 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9a44147c60 msgr2=0x7f9a44155aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:34.475 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.471+0000 7f9a48c34700 1 --2- 172.21.15.174:0/1618445465 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9a44147c60 0x7f9a44155aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:34.475 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.471+0000 7f9a48c34700 1 -- 172.21.15.174:0/1618445465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a4406da40 msgr2=0x7f9a440adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:34.476 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.471+0000 7f9a48c34700 1 --2- 172.21.15.174:0/1618445465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a4406da40 0x7f9a440adb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:34.476 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.471+0000 7f9a48c34700 1 -- 172.21.15.174:0/1618445465 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a3c009a20 con 0x7f9a4406d670 2023-10-06T05:06:34.476 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.471+0000 7f9a49435700 1 --2- 172.21.15.174:0/1618445465 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9a44147c60 0x7f9a44155aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T05:06:34.476 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.471+0000 7f9a48c34700 1 --2- 172.21.15.174:0/1618445465 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9a4406d670 0x7f9a440b06c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9a3800c560 tx=0x7f9a38007eb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:06:34.476 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.471+0000 7f9a40ff9700 1 -- 172.21.15.174:0/1618445465 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a38011590 con 0x7f9a4406d670 2023-10-06T05:06:34.476 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.471+0000 7f9a40ff9700 1 -- 172.21.15.174:0/1618445465 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a38011710 con 0x7f9a4406d670 2023-10-06T05:06:34.476 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.471+0000 7f9a40ff9700 1 -- 172.21.15.174:0/1618445465 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a38011ad0 con 0x7f9a4406d670 2023-10-06T05:06:34.476 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.471+0000 7f9a4ae98700 1 -- 172.21.15.174:0/1618445465 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a440aec30 con 0x7f9a4406d670 2023-10-06T05:06:34.476 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.471+0000 7f9a4ae98700 1 -- 172.21.15.174:0/1618445465 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a440ae0a0 con 0x7f9a4406d670 2023-10-06T05:06:34.476 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.471+0000 7f9a40ff9700 1 -- 172.21.15.174:0/1618445465 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f9a3801c6c0 con 0x7f9a4406d670 2023-10-06T05:06:34.476 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.471+0000 7f9a40ff9700 1 -- 172.21.15.174:0/1618445465 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1066+0+0 (secure 0 0 0) 0x7f9a380128c0 con 0x7f9a4406d670 2023-10-06T05:06:34.476 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.472+0000 7f9a4ae98700 1 -- 172.21.15.174:0/1618445465 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a44064a90 con 0x7f9a4406d670 2023-10-06T05:06:34.477 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.474+0000 7f9a40ff9700 1 -- 172.21.15.174:0/1618445465 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f9a38017070 con 0x7f9a4406d670 2023-10-06T05:06:34.559 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.557+0000 7f9a4ae98700 1 -- 172.21.15.174:0/1618445465 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd new", "uuid": "3bb803ff-1738-46bf-be62-2267bf378822", "id": 3} v 0) v1 -- 0x7f9a4415d2e0 con 0x7f9a4406d670 2023-10-06T05:06:34.562 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.561+0000 7f9a40ff9700 1 -- 172.21.15.174:0/1618445465 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3bb803ff-1738-46bf-be62-2267bf378822", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f9a4415ab50 con 0x7f9a4406d670 2023-10-06T05:06:34.562 INFO:teuthology.orchestra.run.smithi174.stdout:3 2023-10-06T05:06:34.563 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.562+0000 7f9a4ae98700 1 -- 172.21.15.174:0/1618445465 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9a4406d670 msgr2=0x7f9a440b06c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:34.563 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.562+0000 7f9a4ae98700 1 --2- 172.21.15.174:0/1618445465 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9a4406d670 0x7f9a440b06c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9a3800c560 tx=0x7f9a38007eb0 comp rx=0 tx=0).stop 2023-10-06T05:06:34.563 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.562+0000 7f9a4ae98700 1 -- 172.21.15.174:0/1618445465 shutdown_connections 2023-10-06T05:06:34.563 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.562+0000 7f9a4ae98700 1 --2- 172.21.15.174:0/1618445465 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9a44147c60 0x7f9a44155aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:34.564 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.562+0000 7f9a4ae98700 1 --2- 172.21.15.174:0/1618445465 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9a4406d670 0x7f9a440b06c0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:34.564 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.562+0000 7f9a4ae98700 1 --2- 172.21.15.174:0/1618445465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a4406da40 0x7f9a440adb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:34.564 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.562+0000 7f9a4ae98700 1 -- 172.21.15.174:0/1618445465 >> 172.21.15.174:0/1618445465 conn(0x7f9a4406b600 msgr2=0x7f9a4415acf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:06:34.564 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.562+0000 7f9a4ae98700 1 -- 172.21.15.174:0/1618445465 shutdown_connections 2023-10-06T05:06:34.564 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.562+0000 7f9a4ae98700 1 -- 172.21.15.174:0/1618445465 wait complete. 2023-10-06T05:06:34.569 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-10-06T05:06:34.567+0000 7f983cd62180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-06T05:06:34.574 DEBUG:teuthology.orchestra.run.smithi174:> sudo ceph --cluster ceph osd new 7d57a4c9-6462-43f6-b64a-7f9e9b4353d9 4 2023-10-06T05:06:34.605 INFO:tasks.ceph.mgr.x.smithi174.stderr:2023-10-06T05:06:34.603+0000 7fcbfec4d180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-06T05:06:34.692 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.691+0000 7f29ed53f700 1 Processor -- start 2023-10-06T05:06:34.692 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.691+0000 7f29ed53f700 1 -- start start 2023-10-06T05:06:34.694 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.692+0000 7f29ed53f700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f29e8150940 0x7f29e8159c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:34.694 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.692+0000 7f29ed53f700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29e815ac30 con 0x7f29e815a160 2023-10-06T05:06:34.695 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.692+0000 7f29ed53f700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29e80cb6a0 con 0x7f29e80cd4d0 2023-10-06T05:06:34.695 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.692+0000 7f29ed53f700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29e80cb090 con 0x7f29e8150940 2023-10-06T05:06:34.695 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.692+0000 7f29e77fe700 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f29e80cd4d0 0x7f29e80b2050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.174:52742/0 (socket says 172.21.15.174:52742) 2023-10-06T05:06:34.695 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.692+0000 7f29e77fe700 1 -- 172.21.15.174:0/1946702563 learned_addr learned my addr 172.21.15.174:0/1946702563 (peer_addr_for_me v1:172.21.15.174:0/0) 2023-10-06T05:06:34.695 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.692+0000 7f29e6ffd700 1 --2- 172.21.15.174:0/1946702563 >> v2:172.21.15.98:3300/0 conn(0x7f29e8150940 0x7f29e8159c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:34.695 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.692+0000 7f29e5ffb700 1 -- 172.21.15.174:0/1946702563 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1433164827 0 0) 0x7f29e80cb6a0 con 0x7f29e80cd4d0 2023-10-06T05:06:34.695 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.692+0000 7f29e5ffb700 1 -- 172.21.15.174:0/1946702563 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29d0003570 con 0x7f29e80cd4d0 2023-10-06T05:06:34.695 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.692+0000 7f29e5ffb700 1 -- 172.21.15.174:0/1946702563 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3506535148 0 0) 0x7f29dc001110 con 0x7f29e815a160 2023-10-06T05:06:34.695 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.692+0000 7f29e5ffb700 1 -- 172.21.15.174:0/1946702563 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29e80cb6a0 con 0x7f29e815a160 2023-10-06T05:06:34.695 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.692+0000 7f29e6ffd700 1 -- 172.21.15.174:0/1946702563 >> v1:172.21.15.174:6789/0 conn(0x7f29e80cd4d0 legacy=0x7f29e80b2050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:34.696 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.692+0000 7f29e6ffd700 1 -- 172.21.15.174:0/1946702563 >> v1:172.21.15.98:6789/0 conn(0x7f29e815a160 legacy=0x7f29e80ccdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:34.696 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.692+0000 7f29e6ffd700 1 -- 172.21.15.174:0/1946702563 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29e815a670 con 0x7f29e8150940 2023-10-06T05:06:34.696 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.692+0000 7f29e6ffd700 1 --2- 172.21.15.174:0/1946702563 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29e8150940 0x7f29e8159c20 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f29d80098d0 tx=0x7f29d8009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=853c97eb8543c0c9 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:06:34.696 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.693+0000 7f29e5ffb700 1 -- 172.21.15.174:0/1946702563 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29d8005c10 con 0x7f29e8150940 2023-10-06T05:06:34.696 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.693+0000 7f29e5ffb700 1 -- 172.21.15.174:0/1946702563 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29d8005d70 con 0x7f29e8150940 2023-10-06T05:06:34.696 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.693+0000 7f29e5ffb700 1 -- 172.21.15.174:0/1946702563 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29d8006260 con 0x7f29e8150940 2023-10-06T05:06:34.696 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.693+0000 7f29ed53f700 1 -- 172.21.15.174:0/1946702563 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29e8150940 msgr2=0x7f29e8159c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:34.696 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.693+0000 7f29ed53f700 1 --2- 172.21.15.174:0/1946702563 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29e8150940 0x7f29e8159c20 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f29d80098d0 tx=0x7f29d8009120 comp rx=0 tx=0).stop 2023-10-06T05:06:34.696 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.693+0000 7f29ed53f700 1 -- 172.21.15.174:0/1946702563 shutdown_connections 2023-10-06T05:06:34.696 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.693+0000 7f29ed53f700 1 --2- 172.21.15.174:0/1946702563 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29e8150940 0x7f29e8159c20 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:34.697 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.693+0000 7f29ed53f700 1 -- 172.21.15.174:0/1946702563 >> 172.21.15.174:0/1946702563 conn(0x7f29e8099700 msgr2=0x7f29e806f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:06:34.697 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.693+0000 7f29ed53f700 1 -- 172.21.15.174:0/1946702563 shutdown_connections 2023-10-06T05:06:34.697 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.693+0000 7f29ed53f700 1 -- 172.21.15.174:0/1946702563 wait complete. 2023-10-06T05:06:34.698 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.694+0000 7f29ed53f700 1 Processor -- start 2023-10-06T05:06:34.698 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.694+0000 7f29ed53f700 1 -- start start 2023-10-06T05:06:34.698 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.695+0000 7f29ed53f700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f29e80cd4d0 0x7f29e814f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:34.699 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.695+0000 7f29ed53f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29e815a160 0x7f29e8151fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:34.699 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.695+0000 7f29ed53f700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f29e8169ac0 0x7f29e8169f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:34.699 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.695+0000 7f29ed53f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e816ab30 con 0x7f29e815a160 2023-10-06T05:06:34.699 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.695+0000 7f29ed53f700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e8151b40 con 0x7f29e8169ac0 2023-10-06T05:06:34.699 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.695+0000 7f29ed53f700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29e816a570 con 0x7f29e80cd4d0 2023-10-06T05:06:34.699 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.695+0000 7f29e77fe700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f29e8169ac0 0x7f29e8169f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:34.699 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.695+0000 7f29e6ffd700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f29e80cd4d0 0x7f29e814f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:34.699 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.695+0000 7f29e77fe700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f29e8169ac0 0x7f29e8169f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.174:36248/0 (socket says 172.21.15.174:36248) 2023-10-06T05:06:34.699 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.695+0000 7f29e6ffd700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f29e80cd4d0 0x7f29e814f1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.174:38374/0 (socket says 172.21.15.174:38374) 2023-10-06T05:06:34.699 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.695+0000 7f29e77fe700 1 -- 172.21.15.174:0/2349256895 learned_addr learned my addr 172.21.15.174:0/2349256895 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-10-06T05:06:34.699 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.696+0000 7f29e77fe700 1 -- 172.21.15.174:0/2349256895 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f29e80cd4d0 msgr2=0x7f29e814f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:34.699 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.696+0000 7f29e77fe700 1 --2- 172.21.15.174:0/2349256895 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f29e80cd4d0 0x7f29e814f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:34.700 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.696+0000 7f29e77fe700 1 -- 172.21.15.174:0/2349256895 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29e815a160 msgr2=0x7f29e8151fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:34.700 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.696+0000 7f29e67fc700 1 --2- 172.21.15.174:0/2349256895 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29e815a160 0x7f29e8151fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:34.700 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.696+0000 7f29e77fe700 1 --2- 172.21.15.174:0/2349256895 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29e815a160 0x7f29e8151fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:34.700 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.696+0000 7f29e77fe700 1 -- 172.21.15.174:0/2349256895 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29d80093c0 con 0x7f29e8169ac0 2023-10-06T05:06:34.701 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.696+0000 7f29e6ffd700 1 --2- 172.21.15.174:0/2349256895 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f29e80cd4d0 0x7f29e814f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T05:06:34.702 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.696+0000 7f29e67fc700 1 --2- 172.21.15.174:0/2349256895 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29e815a160 0x7f29e8151fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:06:34.702 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.696+0000 7f29e77fe700 1 --2- 172.21.15.174:0/2349256895 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f29e8169ac0 0x7f29e8169f60 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f29e000cc80 tx=0x7f29e0001ae0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:06:34.702 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.696+0000 7f29ceffd700 1 -- 172.21.15.174:0/2349256895 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29e000a960 con 0x7f29e8169ac0 2023-10-06T05:06:34.702 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.696+0000 7f29ceffd700 1 -- 172.21.15.174:0/2349256895 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29e000aac0 con 0x7f29e8169ac0 2023-10-06T05:06:34.702 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.696+0000 7f29ed53f700 1 -- 172.21.15.174:0/2349256895 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29e816a850 con 0x7f29e8169ac0 2023-10-06T05:06:34.702 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.696+0000 7f29ceffd700 1 -- 172.21.15.174:0/2349256895 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29e000ae60 con 0x7f29e8169ac0 2023-10-06T05:06:34.702 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.696+0000 7f29ed53f700 1 -- 172.21.15.174:0/2349256895 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29e814f720 con 0x7f29e8169ac0 2023-10-06T05:06:34.702 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.696+0000 7f29ceffd700 1 -- 172.21.15.174:0/2349256895 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f29e0007300 con 0x7f29e8169ac0 2023-10-06T05:06:34.703 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.696+0000 7f29ceffd700 1 -- 172.21.15.174:0/2349256895 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1181+0+0 (secure 0 0 0) 0x7f29e0012400 con 0x7f29e8169ac0 2023-10-06T05:06:34.703 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.697+0000 7f29ed53f700 1 -- 172.21.15.174:0/2349256895 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29e8151630 con 0x7f29e8169ac0 2023-10-06T05:06:34.703 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.699+0000 7f29ceffd700 1 -- 172.21.15.174:0/2349256895 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f29e8151630 con 0x7f29e8169ac0 2023-10-06T05:06:34.742 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-10-06T05:06:34.740+0000 7f983cd62180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-06T05:06:34.783 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.781+0000 7f29ed53f700 1 -- 172.21.15.174:0/2349256895 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd new", "uuid": "7d57a4c9-6462-43f6-b64a-7f9e9b4353d9", "id": 4} v 0) v1 -- 0x7f29e808a9d0 con 0x7f29e8169ac0 2023-10-06T05:06:34.784 INFO:tasks.ceph.mgr.x.smithi174.stderr:2023-10-06T05:06:34.783+0000 7fcbfec4d180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-06T05:06:34.786 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.785+0000 7f29ceffd700 1 -- 172.21.15.174:0/2349256895 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7d57a4c9-6462-43f6-b64a-7f9e9b4353d9", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f29e00075b0 con 0x7f29e8169ac0 2023-10-06T05:06:34.786 INFO:teuthology.orchestra.run.smithi174.stdout:4 2023-10-06T05:06:34.788 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.786+0000 7f29ed53f700 1 -- 172.21.15.174:0/2349256895 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f29e8169ac0 msgr2=0x7f29e8169f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:34.788 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.786+0000 7f29ed53f700 1 --2- 172.21.15.174:0/2349256895 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f29e8169ac0 0x7f29e8169f60 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f29e000cc80 tx=0x7f29e0001ae0 comp rx=0 tx=0).stop 2023-10-06T05:06:34.788 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.786+0000 7f29ed53f700 1 -- 172.21.15.174:0/2349256895 shutdown_connections 2023-10-06T05:06:34.788 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.786+0000 7f29ed53f700 1 --2- 172.21.15.174:0/2349256895 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f29e8169ac0 0x7f29e8169f60 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:34.788 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.786+0000 7f29ed53f700 1 --2- 172.21.15.174:0/2349256895 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f29e80cd4d0 0x7f29e814f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:34.788 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.786+0000 7f29ed53f700 1 --2- 172.21.15.174:0/2349256895 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29e815a160 0x7f29e8151fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:34.789 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.786+0000 7f29ed53f700 1 -- 172.21.15.174:0/2349256895 >> 172.21.15.174:0/2349256895 conn(0x7f29e8099700 msgr2=0x7f29e80d0b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:06:34.789 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.786+0000 7f29ed53f700 1 -- 172.21.15.174:0/2349256895 shutdown_connections 2023-10-06T05:06:34.789 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.786+0000 7f29ed53f700 1 -- 172.21.15.174:0/2349256895 wait complete. 2023-10-06T05:06:34.799 DEBUG:teuthology.orchestra.run.smithi174:> sudo ceph --cluster ceph osd new 83287ea4-e44d-4af9-bc77-36d47c60d4a2 5 2023-10-06T05:06:34.820 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-10-06T05:06:34.817+0000 7f983cd62180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-06T05:06:34.866 INFO:tasks.ceph.mgr.x.smithi174.stderr:2023-10-06T05:06:34.865+0000 7fcbfec4d180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-06T05:06:34.954 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-10-06T05:06:34.951+0000 7f983cd62180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-06T05:06:34.963 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.961+0000 7f0b8b6aa700 1 Processor -- start 2023-10-06T05:06:34.963 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.962+0000 7f0b8b6aa700 1 -- start start 2023-10-06T05:06:34.965 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.962+0000 7f0b8b6aa700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0b8406bfb0 0x7f0b8415b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:34.965 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.962+0000 7f0b8b6aa700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b840b0500 con 0x7f0b8415b730 2023-10-06T05:06:34.965 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.962+0000 7f0b8b6aa700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b840afbd0 con 0x7f0b84151980 2023-10-06T05:06:34.965 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.962+0000 7f0b8b6aa700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b840afeb0 con 0x7f0b8406bfb0 2023-10-06T05:06:34.965 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.962+0000 7f0b89446700 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f0b84151980 0x7f0b8406b8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.174:52746/0 (socket says 172.21.15.174:52746) 2023-10-06T05:06:34.966 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.962+0000 7f0b89446700 1 -- 172.21.15.174:0/2903357643 learned_addr learned my addr 172.21.15.174:0/2903357643 (peer_addr_for_me v1:172.21.15.174:0/0) 2023-10-06T05:06:34.966 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.963+0000 7f0b88c45700 1 --2- 172.21.15.174:0/2903357643 >> v2:172.21.15.98:3300/0 conn(0x7f0b8406bfb0 0x7f0b8415b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:34.966 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.963+0000 7f0b7bfff700 1 -- 172.21.15.174:0/2903357643 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 321686043 0 0) 0x7f0b840afbd0 con 0x7f0b84151980 2023-10-06T05:06:34.966 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.963+0000 7f0b7bfff700 1 -- 172.21.15.174:0/2903357643 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b70003570 con 0x7f0b84151980 2023-10-06T05:06:34.966 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.963+0000 7f0b7bfff700 1 -- 172.21.15.174:0/2903357643 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2773383682 0 0) 0x7f0b840b0500 con 0x7f0b8415b730 2023-10-06T05:06:34.966 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.963+0000 7f0b7bfff700 1 -- 172.21.15.174:0/2903357643 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b840afbd0 con 0x7f0b8415b730 2023-10-06T05:06:34.966 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.963+0000 7f0b88c45700 1 -- 172.21.15.174:0/2903357643 >> v1:172.21.15.174:6789/0 conn(0x7f0b84151980 legacy=0x7f0b8406b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:34.966 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.963+0000 7f0b88c45700 1 -- 172.21.15.174:0/2903357643 >> v1:172.21.15.98:6789/0 conn(0x7f0b8415b730 legacy=0x7f0b84158b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:34.967 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.963+0000 7f0b88c45700 1 -- 172.21.15.174:0/2903357643 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b840b0190 con 0x7f0b8406bfb0 2023-10-06T05:06:34.967 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.963+0000 7f0b88c45700 1 --2- 172.21.15.174:0/2903357643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b8406bfb0 0x7f0b8415b1f0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f0b800098e0 tx=0x7f0b800091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cc3742b14f298b4 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:06:34.967 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.963+0000 7f0b7bfff700 1 -- 172.21.15.174:0/2903357643 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b80011630 con 0x7f0b8406bfb0 2023-10-06T05:06:34.967 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.963+0000 7f0b7bfff700 1 -- 172.21.15.174:0/2903357643 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b80011790 con 0x7f0b8406bfb0 2023-10-06T05:06:34.967 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.963+0000 7f0b7bfff700 1 -- 172.21.15.174:0/2903357643 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b80011b30 con 0x7f0b8406bfb0 2023-10-06T05:06:34.967 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.964+0000 7f0b8b6aa700 1 -- 172.21.15.174:0/2903357643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b8406bfb0 msgr2=0x7f0b8415b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:34.967 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.964+0000 7f0b8b6aa700 1 --2- 172.21.15.174:0/2903357643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b8406bfb0 0x7f0b8415b1f0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f0b800098e0 tx=0x7f0b800091a0 comp rx=0 tx=0).stop 2023-10-06T05:06:34.967 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.964+0000 7f0b8b6aa700 1 -- 172.21.15.174:0/2903357643 shutdown_connections 2023-10-06T05:06:34.967 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.964+0000 7f0b8b6aa700 1 --2- 172.21.15.174:0/2903357643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b8406bfb0 0x7f0b8415b1f0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:34.968 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.964+0000 7f0b8b6aa700 1 -- 172.21.15.174:0/2903357643 >> 172.21.15.174:0/2903357643 conn(0x7f0b840ce2f0 msgr2=0x7f0b840ce6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:06:34.968 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.964+0000 7f0b8b6aa700 1 -- 172.21.15.174:0/2903357643 shutdown_connections 2023-10-06T05:06:34.968 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.964+0000 7f0b8b6aa700 1 -- 172.21.15.174:0/2903357643 wait complete. 2023-10-06T05:06:34.970 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.965+0000 7f0b8b6aa700 1 Processor -- start 2023-10-06T05:06:34.971 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.965+0000 7f0b8b6aa700 1 -- start start 2023-10-06T05:06:34.971 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.966+0000 7f0b8b6aa700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0b8406bfb0 0x7f0b8415a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:34.971 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.966+0000 7f0b8b6aa700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b84151980 0x7f0b84156630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:34.971 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.966+0000 7f0b8b6aa700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0b8415b730 0x7f0b840c8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:34.971 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.966+0000 7f0b8b6aa700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b840b25e0 con 0x7f0b84151980 2023-10-06T05:06:34.971 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.966+0000 7f0b8b6aa700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b840b28c0 con 0x7f0b8415b730 2023-10-06T05:06:34.971 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.966+0000 7f0b8b6aa700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b840b2ba0 con 0x7f0b8406bfb0 2023-10-06T05:06:34.971 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.966+0000 7f0b89446700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0b8406bfb0 0x7f0b8415a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:34.971 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.966+0000 7f0b89c47700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0b8415b730 0x7f0b840c8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:34.972 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.966+0000 7f0b89446700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0b8406bfb0 0x7f0b8415a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.174:38384/0 (socket says 172.21.15.174:38384) 2023-10-06T05:06:34.972 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.966+0000 7f0b89446700 1 -- 172.21.15.174:0/2141449227 learned_addr learned my addr 172.21.15.174:0/2141449227 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-10-06T05:06:34.972 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.966+0000 7f0b88c45700 1 --2- 172.21.15.174:0/2141449227 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b84151980 0x7f0b84156630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:34.972 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.966+0000 7f0b89c47700 1 -- 172.21.15.174:0/2141449227 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0b8406bfb0 msgr2=0x7f0b8415a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:34.972 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.966+0000 7f0b89c47700 1 --2- 172.21.15.174:0/2141449227 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0b8406bfb0 0x7f0b8415a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:34.972 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.966+0000 7f0b89c47700 1 -- 172.21.15.174:0/2141449227 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b84151980 msgr2=0x7f0b84156630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:34.972 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.966+0000 7f0b89c47700 1 --2- 172.21.15.174:0/2141449227 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b84151980 0x7f0b84156630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:34.972 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.966+0000 7f0b89c47700 1 -- 172.21.15.174:0/2141449227 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b80009420 con 0x7f0b8415b730 2023-10-06T05:06:34.972 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.966+0000 7f0b89446700 1 --2- 172.21.15.174:0/2141449227 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0b8406bfb0 0x7f0b8415a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T05:06:34.972 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.966+0000 7f0b88c45700 1 --2- 172.21.15.174:0/2141449227 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b84151980 0x7f0b84156630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T05:06:34.972 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.967+0000 7f0b89c47700 1 --2- 172.21.15.174:0/2141449227 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0b8415b730 0x7f0b840c8ee0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0b7c00c810 tx=0x7f0b7c002650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:06:34.972 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.967+0000 7f0b797fa700 1 -- 172.21.15.174:0/2141449227 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b7c012630 con 0x7f0b8415b730 2023-10-06T05:06:34.973 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.967+0000 7f0b797fa700 1 -- 172.21.15.174:0/2141449227 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b7c0127b0 con 0x7f0b8415b730 2023-10-06T05:06:34.973 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.967+0000 7f0b797fa700 1 -- 172.21.15.174:0/2141449227 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b7c012b70 con 0x7f0b8415b730 2023-10-06T05:06:34.973 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.967+0000 7f0b8b6aa700 1 -- 172.21.15.174:0/2141449227 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b840b2e80 con 0x7f0b8415b730 2023-10-06T05:06:34.973 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.967+0000 7f0b8b6aa700 1 -- 172.21.15.174:0/2141449227 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b8415a750 con 0x7f0b8415b730 2023-10-06T05:06:34.973 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.967+0000 7f0b797fa700 1 -- 172.21.15.174:0/2141449227 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f0b7c01b680 con 0x7f0b8415b730 2023-10-06T05:06:34.973 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.967+0000 7f0b797fa700 1 -- 172.21.15.174:0/2141449227 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1296+0+0 (secure 0 0 0) 0x7f0b7c01a3b0 con 0x7f0b8415b730 2023-10-06T05:06:34.973 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.968+0000 7f0b8b6aa700 1 -- 172.21.15.174:0/2141449227 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b840b33a0 con 0x7f0b8415b730 2023-10-06T05:06:34.973 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:34.970+0000 7f0b797fa700 1 -- 172.21.15.174:0/2141449227 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f0b7c03a6d0 con 0x7f0b8415b730 2023-10-06T05:06:35.009 INFO:tasks.ceph.mgr.x.smithi174.stderr:2023-10-06T05:06:35.008+0000 7fcbfec4d180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-06T05:06:35.050 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-10-06T05:06:35.048+0000 7f983cd62180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-06T05:06:35.054 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.053+0000 7f0b8b6aa700 1 -- 172.21.15.174:0/2141449227 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd new", "uuid": "83287ea4-e44d-4af9-bc77-36d47c60d4a2", "id": 5} v 0) v1 -- 0x7f0b84064a90 con 0x7f0b8415b730 2023-10-06T05:06:35.058 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.057+0000 7f0b797fa700 1 -- 172.21.15.174:0/2141449227 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "83287ea4-e44d-4af9-bc77-36d47c60d4a2", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f0b7c029070 con 0x7f0b8415b730 2023-10-06T05:06:35.058 INFO:teuthology.orchestra.run.smithi174.stdout:5 2023-10-06T05:06:35.060 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.058+0000 7f0b8b6aa700 1 -- 172.21.15.174:0/2141449227 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0b8415b730 msgr2=0x7f0b840c8ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:35.060 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.058+0000 7f0b8b6aa700 1 --2- 172.21.15.174:0/2141449227 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0b8415b730 0x7f0b840c8ee0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0b7c00c810 tx=0x7f0b7c002650 comp rx=0 tx=0).stop 2023-10-06T05:06:35.060 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.058+0000 7f0b8b6aa700 1 -- 172.21.15.174:0/2141449227 shutdown_connections 2023-10-06T05:06:35.060 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.058+0000 7f0b8b6aa700 1 --2- 172.21.15.174:0/2141449227 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0b8415b730 0x7f0b840c8ee0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:35.060 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.058+0000 7f0b8b6aa700 1 --2- 172.21.15.174:0/2141449227 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0b8406bfb0 0x7f0b8415a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:35.061 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.058+0000 7f0b8b6aa700 1 --2- 172.21.15.174:0/2141449227 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b84151980 0x7f0b84156630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:35.061 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.058+0000 7f0b8b6aa700 1 -- 172.21.15.174:0/2141449227 >> 172.21.15.174:0/2141449227 conn(0x7f0b840ce2f0 msgr2=0x7f0b840c3b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:06:35.061 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.058+0000 7f0b8b6aa700 1 -- 172.21.15.174:0/2141449227 shutdown_connections 2023-10-06T05:06:35.061 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.058+0000 7f0b8b6aa700 1 -- 172.21.15.174:0/2141449227 wait complete. 2023-10-06T05:06:35.071 DEBUG:teuthology.orchestra.run.smithi174:> sudo ceph --cluster ceph osd new 496a6272-892e-43c4-a145-206623c22f99 6 2023-10-06T05:06:35.108 INFO:tasks.ceph.mgr.x.smithi174.stderr:2023-10-06T05:06:35.107+0000 7fcbfec4d180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-06T05:06:35.224 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.222+0000 7fc577e9b700 1 Processor -- start 2023-10-06T05:06:35.224 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.223+0000 7fc577e9b700 1 -- start start 2023-10-06T05:06:35.225 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.224+0000 7fc577e9b700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fc57006da40 0x7fc5700b1080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:35.226 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.224+0000 7fc577e9b700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc57015ab50 con 0x7fc57006d670 2023-10-06T05:06:35.226 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.224+0000 7fc577e9b700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc57015d650 con 0x7fc57006ce10 2023-10-06T05:06:35.226 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.224+0000 7fc577e9b700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc57015a590 con 0x7fc57006da40 2023-10-06T05:06:35.226 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.224+0000 7fc576438700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fc57006da40 0x7fc5700b1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:35.226 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.224+0000 7fc576438700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fc57006da40 0x7fc5700b1080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.174:36264/0 (socket says 172.21.15.174:36264) 2023-10-06T05:06:35.226 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.224+0000 7fc575436700 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fc57006ce10 0x7fc570159af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.174:52758/0 (socket says 172.21.15.174:52758) 2023-10-06T05:06:35.226 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.224+0000 7fc576438700 1 -- 172.21.15.174:0/3534719126 learned_addr learned my addr 172.21.15.174:0/3534719126 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-10-06T05:06:35.226 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.224+0000 7fc576438700 1 -- 172.21.15.174:0/3534719126 >> v1:172.21.15.174:6789/0 conn(0x7fc57006ce10 legacy=0x7fc570159af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:35.226 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.224+0000 7fc574c35700 1 -- 172.21.15.174:0/3534719126 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3695932654 0 0) 0x7fc560001160 con 0x7fc57006d670 2023-10-06T05:06:35.226 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.224+0000 7fc576438700 1 -- 172.21.15.174:0/3534719126 >> v1:172.21.15.98:6789/0 conn(0x7fc57006d670 legacy=0x7fc57015dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:35.226 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.224+0000 7fc576438700 1 -- 172.21.15.174:0/3534719126 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc57015a870 con 0x7fc57006da40 2023-10-06T05:06:35.227 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.224+0000 7fc576438700 1 --2- 172.21.15.174:0/3534719126 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc57006da40 0x7fc5700b1080 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fc568009f50 tx=0x7fc568009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b0c124a16d95fdae server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:06:35.227 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.224+0000 7fc574c35700 1 -- 172.21.15.174:0/3534719126 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc56800e070 con 0x7fc57006da40 2023-10-06T05:06:35.227 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.224+0000 7fc574c35700 1 -- 172.21.15.174:0/3534719126 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc56800ba70 con 0x7fc57006da40 2023-10-06T05:06:35.227 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.224+0000 7fc574c35700 1 -- 172.21.15.174:0/3534719126 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5680092e0 con 0x7fc57006da40 2023-10-06T05:06:35.230 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.225+0000 7fc577e9b700 1 -- 172.21.15.174:0/3534719126 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc57006da40 msgr2=0x7fc5700b1080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:35.230 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.225+0000 7fc577e9b700 1 --2- 172.21.15.174:0/3534719126 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc57006da40 0x7fc5700b1080 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fc568009f50 tx=0x7fc568009840 comp rx=0 tx=0).stop 2023-10-06T05:06:35.230 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.225+0000 7fc577e9b700 1 -- 172.21.15.174:0/3534719126 shutdown_connections 2023-10-06T05:06:35.230 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.225+0000 7fc577e9b700 1 --2- 172.21.15.174:0/3534719126 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc57006da40 0x7fc5700b1080 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:35.230 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.225+0000 7fc577e9b700 1 -- 172.21.15.174:0/3534719126 >> 172.21.15.174:0/3534719126 conn(0x7fc57006b600 msgr2=0x7fc57006ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:06:35.230 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.225+0000 7fc577e9b700 1 -- 172.21.15.174:0/3534719126 shutdown_connections 2023-10-06T05:06:35.231 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.225+0000 7fc577e9b700 1 -- 172.21.15.174:0/3534719126 wait complete. 2023-10-06T05:06:35.231 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.226+0000 7fc577e9b700 1 Processor -- start 2023-10-06T05:06:35.231 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.226+0000 7fc577e9b700 1 -- start start 2023-10-06T05:06:35.231 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.227+0000 7fc577e9b700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc57006d670 0x7fc570147ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:35.231 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.227+0000 7fc577e9b700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc57006da40 0x7fc5700aeb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:35.231 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.227+0000 7fc577e9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5700af070 0x7fc570150b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:35.231 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.227+0000 7fc577e9b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc570152750 con 0x7fc5700af070 2023-10-06T05:06:35.231 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.227+0000 7fc577e9b700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5701483e0 con 0x7fc57006da40 2023-10-06T05:06:35.231 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.227+0000 7fc577e9b700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5700af6a0 con 0x7fc57006d670 2023-10-06T05:06:35.231 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.227+0000 7fc575436700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc57006da40 0x7fc5700aeb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:35.231 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.227+0000 7fc575c37700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc57006d670 0x7fc570147ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:35.232 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.227+0000 7fc575436700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc57006da40 0x7fc5700aeb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.174:36268/0 (socket says 172.21.15.174:36268) 2023-10-06T05:06:35.232 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.227+0000 7fc575c37700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc57006d670 0x7fc570147ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.174:38398/0 (socket says 172.21.15.174:38398) 2023-10-06T05:06:35.232 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.227+0000 7fc575436700 1 -- 172.21.15.174:0/6254310 learned_addr learned my addr 172.21.15.174:0/6254310 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-10-06T05:06:35.232 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.227+0000 7fc576438700 1 --2- 172.21.15.174:0/6254310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5700af070 0x7fc570150b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:35.232 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.227+0000 7fc575436700 1 -- 172.21.15.174:0/6254310 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc57006d670 msgr2=0x7fc570147ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:35.232 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.227+0000 7fc575436700 1 --2- 172.21.15.174:0/6254310 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc57006d670 0x7fc570147ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:35.232 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.227+0000 7fc575436700 1 -- 172.21.15.174:0/6254310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5700af070 msgr2=0x7fc570150b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:35.232 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.227+0000 7fc575436700 1 --2- 172.21.15.174:0/6254310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5700af070 0x7fc570150b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:35.232 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.227+0000 7fc575436700 1 -- 172.21.15.174:0/6254310 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc568009ae0 con 0x7fc57006da40 2023-10-06T05:06:35.232 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.227+0000 7fc575c37700 1 --2- 172.21.15.174:0/6254310 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc57006d670 0x7fc570147ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T05:06:35.232 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.227+0000 7fc575436700 1 --2- 172.21.15.174:0/6254310 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc57006da40 0x7fc5700aeb30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc56c00c560 tx=0x7fc56c007f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:06:35.232 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.228+0000 7fc565ffb700 1 -- 172.21.15.174:0/6254310 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc56c005a70 con 0x7fc57006da40 2023-10-06T05:06:35.233 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.228+0000 7fc565ffb700 1 -- 172.21.15.174:0/6254310 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc56c005bd0 con 0x7fc57006da40 2023-10-06T05:06:35.233 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.228+0000 7fc577e9b700 1 -- 172.21.15.174:0/6254310 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5700af980 con 0x7fc57006da40 2023-10-06T05:06:35.233 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.228+0000 7fc565ffb700 1 -- 172.21.15.174:0/6254310 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc56c0070d0 con 0x7fc57006da40 2023-10-06T05:06:35.233 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.228+0000 7fc565ffb700 1 -- 172.21.15.174:0/6254310 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fc56c012400 con 0x7fc57006da40 2023-10-06T05:06:35.233 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.228+0000 7fc577e9b700 1 -- 172.21.15.174:0/6254310 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5700aff40 con 0x7fc57006da40 2023-10-06T05:06:35.233 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.228+0000 7fc565ffb700 1 -- 172.21.15.174:0/6254310 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1411+0+0 (secure 0 0 0) 0x7fc56c0076c0 con 0x7fc57006da40 2023-10-06T05:06:35.233 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.229+0000 7fc577e9b700 1 -- 172.21.15.174:0/6254310 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5700af470 con 0x7fc57006da40 2023-10-06T05:06:35.233 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.231+0000 7fc565ffb700 1 -- 172.21.15.174:0/6254310 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fc56c017070 con 0x7fc57006da40 2023-10-06T05:06:35.313 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.312+0000 7fc577e9b700 1 -- 172.21.15.174:0/6254310 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd new", "uuid": "496a6272-892e-43c4-a145-206623c22f99", "id": 6} v 0) v1 -- 0x7fc570064a90 con 0x7fc57006da40 2023-10-06T05:06:35.319 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.318+0000 7fc565ffb700 1 -- 172.21.15.174:0/6254310 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "496a6272-892e-43c4-a145-206623c22f99", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7fc56c0128e0 con 0x7fc57006da40 2023-10-06T05:06:35.319 INFO:teuthology.orchestra.run.smithi174.stdout:6 2023-10-06T05:06:35.321 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.319+0000 7fc577e9b700 1 -- 172.21.15.174:0/6254310 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc57006da40 msgr2=0x7fc5700aeb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:35.321 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.319+0000 7fc577e9b700 1 --2- 172.21.15.174:0/6254310 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc57006da40 0x7fc5700aeb30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc56c00c560 tx=0x7fc56c007f60 comp rx=0 tx=0).stop 2023-10-06T05:06:35.321 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.319+0000 7fc577e9b700 1 -- 172.21.15.174:0/6254310 shutdown_connections 2023-10-06T05:06:35.321 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.319+0000 7fc577e9b700 1 --2- 172.21.15.174:0/6254310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5700af070 0x7fc570150b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:35.321 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.319+0000 7fc577e9b700 1 --2- 172.21.15.174:0/6254310 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc57006d670 0x7fc570147ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:35.321 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.319+0000 7fc577e9b700 1 --2- 172.21.15.174:0/6254310 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc57006da40 0x7fc5700aeb30 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:35.321 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.319+0000 7fc577e9b700 1 -- 172.21.15.174:0/6254310 >> 172.21.15.174:0/6254310 conn(0x7fc57006b600 msgr2=0x7fc570158460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:06:35.321 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.319+0000 7fc577e9b700 1 -- 172.21.15.174:0/6254310 shutdown_connections 2023-10-06T05:06:35.321 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.319+0000 7fc577e9b700 1 -- 172.21.15.174:0/6254310 wait complete. 2023-10-06T05:06:35.332 DEBUG:teuthology.orchestra.run.smithi174:> sudo ceph --cluster ceph osd new 49967e34-8b07-4bab-a13e-0c16d6a0f3a5 7 2023-10-06T05:06:35.451 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.449+0000 7f35c4197700 1 Processor -- start 2023-10-06T05:06:35.451 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.450+0000 7f35c4197700 1 -- start start 2023-10-06T05:06:35.453 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.450+0000 7f35c4197700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f35bc156ec0 0x7f35bc15a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:35.453 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.450+0000 7f35c4197700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35bc15da20 con 0x7f35bc0675b0 2023-10-06T05:06:35.454 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.450+0000 7f35c4197700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35bc0cb650 con 0x7f35bc157290 2023-10-06T05:06:35.454 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.450+0000 7f35c4197700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35bc15d0f0 con 0x7f35bc156ec0 2023-10-06T05:06:35.454 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.451+0000 7f35c1f33700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f35bc156ec0 0x7f35bc15a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:35.454 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.451+0000 7f35c1f33700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f35bc156ec0 0x7f35bc15a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.174:36280/0 (socket says 172.21.15.174:36280) 2023-10-06T05:06:35.454 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.451+0000 7f35c1f33700 1 -- 172.21.15.174:0/3161513953 learned_addr learned my addr 172.21.15.174:0/3161513953 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-10-06T05:06:35.454 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.451+0000 7f35c1f33700 1 -- 172.21.15.174:0/3161513953 >> v1:172.21.15.174:6789/0 conn(0x7f35bc157290 legacy=0x7f35bc0cad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:35.454 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.451+0000 7f35c1f33700 1 -- 172.21.15.174:0/3161513953 >> v1:172.21.15.98:6789/0 conn(0x7f35bc0675b0 legacy=0x7f35bc06b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:35.454 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.451+0000 7f35c1f33700 1 -- 172.21.15.174:0/3161513953 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35bc15d3d0 con 0x7f35bc156ec0 2023-10-06T05:06:35.454 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.451+0000 7f35c1f33700 1 --2- 172.21.15.174:0/3161513953 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f35bc156ec0 0x7f35bc15a6e0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f35ac0099b0 tx=0x7f35ac009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b6be5b683be1e37 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:06:35.454 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.451+0000 7f35c0f31700 1 -- 172.21.15.174:0/3161513953 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35ac00b860 con 0x7f35bc156ec0 2023-10-06T05:06:35.454 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.451+0000 7f35c0f31700 1 -- 172.21.15.174:0/3161513953 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35ac00b9c0 con 0x7f35bc156ec0 2023-10-06T05:06:35.455 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.451+0000 7f35c0f31700 1 -- 172.21.15.174:0/3161513953 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35ac00bd60 con 0x7f35bc156ec0 2023-10-06T05:06:35.455 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.452+0000 7f35c4197700 1 -- 172.21.15.174:0/3161513953 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f35bc156ec0 msgr2=0x7f35bc15a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:35.455 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.452+0000 7f35c4197700 1 --2- 172.21.15.174:0/3161513953 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f35bc156ec0 0x7f35bc15a6e0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f35ac0099b0 tx=0x7f35ac009250 comp rx=0 tx=0).stop 2023-10-06T05:06:35.455 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.452+0000 7f35c4197700 1 -- 172.21.15.174:0/3161513953 shutdown_connections 2023-10-06T05:06:35.455 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.452+0000 7f35c4197700 1 --2- 172.21.15.174:0/3161513953 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f35bc156ec0 0x7f35bc15a6e0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:35.455 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.452+0000 7f35c4197700 1 -- 172.21.15.174:0/3161513953 >> 172.21.15.174:0/3161513953 conn(0x7f35bc0cd1a0 msgr2=0x7f35bc0cd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:06:35.455 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.452+0000 7f35c4197700 1 -- 172.21.15.174:0/3161513953 shutdown_connections 2023-10-06T05:06:35.455 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.452+0000 7f35c4197700 1 -- 172.21.15.174:0/3161513953 wait complete. 2023-10-06T05:06:35.457 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.453+0000 7f35c4197700 1 Processor -- start 2023-10-06T05:06:35.458 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.453+0000 7f35c4197700 1 -- start start 2023-10-06T05:06:35.458 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.454+0000 7f35c4197700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f35bc0675b0 0x7f35bc0c8bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:35.458 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.454+0000 7f35c4197700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f35bc156ec0 0x7f35bc16d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:35.458 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.454+0000 7f35c4197700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35bc0c4330 0x7f35bc154750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:35.458 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.454+0000 7f35c4197700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35bc16e090 con 0x7f35bc0c4330 2023-10-06T05:06:35.458 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.454+0000 7f35c4197700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35bc16b160 con 0x7f35bc0675b0 2023-10-06T05:06:35.458 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.454+0000 7f35c4197700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35bc16dad0 con 0x7f35bc156ec0 2023-10-06T05:06:35.458 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.454+0000 7f35c1f33700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f35bc0675b0 0x7f35bc0c8bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:35.458 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.454+0000 7f35c1f33700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f35bc0675b0 0x7f35bc0c8bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.174:36292/0 (socket says 172.21.15.174:36292) 2023-10-06T05:06:35.458 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.454+0000 7f35c1732700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f35bc156ec0 0x7f35bc16d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:35.459 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.454+0000 7f35c1f33700 1 -- 172.21.15.174:0/2224396986 learned_addr learned my addr 172.21.15.174:0/2224396986 (peer_addr_for_me v2:172.21.15.174:0/0) 2023-10-06T05:06:35.459 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.454+0000 7f35c2734700 1 --2- 172.21.15.174:0/2224396986 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35bc0c4330 0x7f35bc154750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:35.459 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.454+0000 7f35c1f33700 1 -- 172.21.15.174:0/2224396986 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f35bc156ec0 msgr2=0x7f35bc16d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:35.459 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.454+0000 7f35c1f33700 1 --2- 172.21.15.174:0/2224396986 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f35bc156ec0 0x7f35bc16d180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:35.459 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.454+0000 7f35c1f33700 1 -- 172.21.15.174:0/2224396986 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35bc0c4330 msgr2=0x7f35bc154750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:35.459 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.454+0000 7f35c1f33700 1 --2- 172.21.15.174:0/2224396986 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35bc0c4330 0x7f35bc154750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:35.459 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.454+0000 7f35c1f33700 1 -- 172.21.15.174:0/2224396986 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35ac0094f0 con 0x7f35bc0675b0 2023-10-06T05:06:35.459 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.454+0000 7f35c1732700 1 --2- 172.21.15.174:0/2224396986 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f35bc156ec0 0x7f35bc16d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T05:06:35.459 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.454+0000 7f35c1f33700 1 --2- 172.21.15.174:0/2224396986 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f35bc0675b0 0x7f35bc0c8bf0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f35bc06d340 tx=0x7f35ac017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:06:35.459 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.455+0000 7f35b1ffb700 1 -- 172.21.15.174:0/2224396986 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35ac024440 con 0x7f35bc0675b0 2023-10-06T05:06:35.459 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.455+0000 7f35b1ffb700 1 -- 172.21.15.174:0/2224396986 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35ac0245a0 con 0x7f35bc0675b0 2023-10-06T05:06:35.459 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.455+0000 7f35c4197700 1 -- 172.21.15.174:0/2224396986 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35bc16ddb0 con 0x7f35bc0675b0 2023-10-06T05:06:35.460 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.455+0000 7f35b1ffb700 1 -- 172.21.15.174:0/2224396986 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35ac024920 con 0x7f35bc0675b0 2023-10-06T05:06:35.460 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.455+0000 7f35c4197700 1 -- 172.21.15.174:0/2224396986 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35bc0c9130 con 0x7f35bc0675b0 2023-10-06T05:06:35.460 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.455+0000 7f35b1ffb700 1 -- 172.21.15.174:0/2224396986 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f35ac024d50 con 0x7f35bc0675b0 2023-10-06T05:06:35.460 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.455+0000 7f35b1ffb700 1 -- 172.21.15.174:0/2224396986 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1526+0+0 (secure 0 0 0) 0x7f35ac023a50 con 0x7f35bc0675b0 2023-10-06T05:06:35.460 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.456+0000 7f35c4197700 1 -- 172.21.15.174:0/2224396986 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35bc064a90 con 0x7f35bc0675b0 2023-10-06T05:06:35.461 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.460+0000 7f35b1ffb700 1 -- 172.21.15.174:0/2224396986 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f35ac015020 con 0x7f35bc0675b0 2023-10-06T05:06:35.544 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.543+0000 7f35c4197700 1 -- 172.21.15.174:0/2224396986 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd new", "uuid": "49967e34-8b07-4bab-a13e-0c16d6a0f3a5", "id": 7} v 0) v1 -- 0x7f35bc04bb40 con 0x7f35bc0675b0 2023-10-06T05:06:35.555 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.553+0000 7f35b1ffb700 1 -- 172.21.15.174:0/2224396986 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "49967e34-8b07-4bab-a13e-0c16d6a0f3a5", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f35ac023d50 con 0x7f35bc0675b0 2023-10-06T05:06:35.555 INFO:teuthology.orchestra.run.smithi174.stdout:7 2023-10-06T05:06:35.556 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.555+0000 7f35c4197700 1 -- 172.21.15.174:0/2224396986 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f35bc0675b0 msgr2=0x7f35bc0c8bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:35.556 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.555+0000 7f35c4197700 1 --2- 172.21.15.174:0/2224396986 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f35bc0675b0 0x7f35bc0c8bf0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f35bc06d340 tx=0x7f35ac017000 comp rx=0 tx=0).stop 2023-10-06T05:06:35.557 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.555+0000 7f35c4197700 1 -- 172.21.15.174:0/2224396986 shutdown_connections 2023-10-06T05:06:35.557 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.555+0000 7f35c4197700 1 --2- 172.21.15.174:0/2224396986 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35bc0c4330 0x7f35bc154750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:35.557 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.555+0000 7f35c4197700 1 --2- 172.21.15.174:0/2224396986 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f35bc0675b0 0x7f35bc0c8bf0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:35.557 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.555+0000 7f35c4197700 1 --2- 172.21.15.174:0/2224396986 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f35bc156ec0 0x7f35bc16d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:35.557 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.555+0000 7f35c4197700 1 -- 172.21.15.174:0/2224396986 >> 172.21.15.174:0/2224396986 conn(0x7f35bc0cd1a0 msgr2=0x7f35bc0c1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:06:35.557 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.555+0000 7f35c4197700 1 -- 172.21.15.174:0/2224396986 shutdown_connections 2023-10-06T05:06:35.557 INFO:teuthology.orchestra.run.smithi174.stderr:2023-10-06T05:06:35.555+0000 7f35c4197700 1 -- 172.21.15.174:0/2224396986 wait complete. 2023-10-06T05:06:35.568 INFO:tasks.ceph.osd.0:Restarting daemon 2023-10-06T05:06:35.568 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-10-06T05:06:35.572 INFO:tasks.ceph.osd.0:Started 2023-10-06T05:06:35.572 INFO:tasks.ceph.osd.1:Restarting daemon 2023-10-06T05:06:35.573 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-10-06T05:06:35.577 INFO:tasks.ceph.osd.1:Started 2023-10-06T05:06:35.577 INFO:tasks.ceph.osd.2:Restarting daemon 2023-10-06T05:06:35.577 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-10-06T05:06:35.577 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-10-06T05:06:35.575+0000 7f983cd62180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-06T05:06:35.582 INFO:tasks.ceph.osd.2:Started 2023-10-06T05:06:35.582 INFO:tasks.ceph.osd.3:Restarting daemon 2023-10-06T05:06:35.582 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-10-06T05:06:35.587 INFO:tasks.ceph.osd.3:Started 2023-10-06T05:06:35.587 INFO:tasks.ceph.osd.4:Restarting daemon 2023-10-06T05:06:35.587 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-10-06T05:06:35.593 INFO:tasks.ceph.osd.4:Started 2023-10-06T05:06:35.594 INFO:tasks.ceph.osd.5:Restarting daemon 2023-10-06T05:06:35.594 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-10-06T05:06:35.599 INFO:tasks.ceph.osd.5:Started 2023-10-06T05:06:35.599 INFO:tasks.ceph.osd.6:Restarting daemon 2023-10-06T05:06:35.599 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2023-10-06T05:06:35.602 INFO:tasks.ceph.osd.6:Started 2023-10-06T05:06:35.603 INFO:tasks.ceph.osd.7:Restarting daemon 2023-10-06T05:06:35.603 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2023-10-06T05:06:35.608 INFO:tasks.ceph.osd.7:Started 2023-10-06T05:06:35.609 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T05:06:35.662 INFO:tasks.ceph.mgr.x.smithi174.stderr:2023-10-06T05:06:35.661+0000 7fcbfec4d180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-06T05:06:35.675 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-10-06T05:06:35.673+0000 7f983cd62180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-06T05:06:35.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:35.744+0000 7f83e243c700 1 Processor -- start 2023-10-06T05:06:35.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:35.744+0000 7f83e243c700 1 -- start start 2023-10-06T05:06:35.748 INFO:tasks.ceph.mgr.x.smithi174.stderr:2023-10-06T05:06:35.745+0000 7fcbfec4d180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-06T05:06:35.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:35.745+0000 7f83e243c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f83dc159560 0x7f83dc0aa530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:35.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:35.745+0000 7f83e243c700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83dc0b0c60 con 0x7f83dc0b0760 2023-10-06T05:06:35.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:35.745+0000 7f83e243c700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83dc0b0f40 con 0x7f83dc0b49a0 2023-10-06T05:06:35.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:35.745+0000 7f83e243c700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83dc0b1220 con 0x7f83dc159560 2023-10-06T05:06:35.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:35.745+0000 7f83dbfff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f83dc159560 0x7f83dc0aa530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:35.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:35.745+0000 7f83e09d9700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f83dc0b0760 0x7f83dc0b1bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:41850/0 (socket says 172.21.15.98:41850) 2023-10-06T05:06:35.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:35.745+0000 7f83e09d9700 1 -- 172.21.15.98:0/3683851326 learned_addr learned my addr 172.21.15.98:0/3683851326 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:06:35.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:35.746+0000 7f83dbfff700 1 -- 172.21.15.98:0/3683851326 >> v1:172.21.15.174:6789/0 conn(0x7f83dc0b49a0 legacy=0x7f83dc0aaa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:35.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:35.746+0000 7f83dbfff700 1 -- 172.21.15.98:0/3683851326 >> v1:172.21.15.98:6789/0 conn(0x7f83dc0b0760 legacy=0x7f83dc0b1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:35.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:35.746+0000 7f83daffd700 1 -- 172.21.15.98:0/3683851326 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 39725027 0 0) 0x7f83dc0b0c60 con 0x7f83dc0b0760 2023-10-06T05:06:35.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:35.746+0000 7f83dbfff700 1 -- 172.21.15.98:0/3683851326 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83dc0b1500 con 0x7f83dc159560 2023-10-06T05:06:35.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:35.746+0000 7f83dbfff700 1 --2- 172.21.15.98:0/3683851326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83dc159560 0x7f83dc0aa530 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f83d0009900 tx=0x7f83d0009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11ab8265df4e3743 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:06:35.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:35.746+0000 7f83daffd700 1 -- 172.21.15.98:0/3683851326 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d0009ea0 con 0x7f83dc159560 2023-10-06T05:06:35.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:35.746+0000 7f83daffd700 1 -- 172.21.15.98:0/3683851326 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83d0005c10 con 0x7f83dc159560 2023-10-06T05:06:35.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:35.746+0000 7f83daffd700 1 -- 172.21.15.98:0/3683851326 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d000b610 con 0x7f83dc159560 2023-10-06T05:06:35.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:35.746+0000 7f83e243c700 1 -- 172.21.15.98:0/3683851326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83dc159560 msgr2=0x7f83dc0aa530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:35.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:35.746+0000 7f83e243c700 1 --2- 172.21.15.98:0/3683851326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83dc159560 0x7f83dc0aa530 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f83d0009900 tx=0x7f83d0009120 comp rx=0 tx=0).stop 2023-10-06T05:06:35.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:35.746+0000 7f83e243c700 1 -- 172.21.15.98:0/3683851326 shutdown_connections 2023-10-06T05:06:35.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:35.746+0000 7f83e243c700 1 --2- 172.21.15.98:0/3683851326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83dc159560 0x7f83dc0aa530 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:35.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:35.746+0000 7f83e243c700 1 -- 172.21.15.98:0/3683851326 >> 172.21.15.98:0/3683851326 conn(0x7f83dc0997f0 msgr2=0x7f83dc06f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:06:35.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:35.747+0000 7f83e243c700 1 -- 172.21.15.98:0/3683851326 shutdown_connections 2023-10-06T05:06:35.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:35.747+0000 7f83e243c700 1 -- 172.21.15.98:0/3683851326 wait complete. 2023-10-06T05:06:35.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:35.748+0000 7f83e243c700 1 Processor -- start 2023-10-06T05:06:35.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:35.748+0000 7f83e243c700 1 -- start start 2023-10-06T05:06:35.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:35.749+0000 7f83e243c700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f83dc0b0760 0x7f83dc0acfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:35.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:35.749+0000 7f83e243c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83dc0b49a0 0x7f83dc157aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:35.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:35.749+0000 7f83e243c700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f83dc159560 0x7f83dc09a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:35.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:35.749+0000 7f83e243c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83dc0ab290 con 0x7f83dc0b49a0 2023-10-06T05:06:35.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:35.749+0000 7f83e243c700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83dc0abb30 con 0x7f83dc0b0760 2023-10-06T05:06:35.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:35.749+0000 7f83e243c700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83dc09dd80 con 0x7f83dc159560 2023-10-06T05:06:35.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:35.749+0000 7f83db7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83dc0b49a0 0x7f83dc157aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:35.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:35.749+0000 7f83db7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83dc0b49a0 0x7f83dc157aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39182/0 (socket says 172.21.15.98:39182) 2023-10-06T05:06:35.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:35.749+0000 7f83dbfff700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f83dc0b0760 0x7f83dc0acfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:35.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:35.749+0000 7f83db7fe700 1 -- 172.21.15.98:0/1791032330 learned_addr learned my addr 172.21.15.98:0/1791032330 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:06:35.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:35.749+0000 7f83e09d9700 1 --2- 172.21.15.98:0/1791032330 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f83dc159560 0x7f83dc09a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:35.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:35.749+0000 7f83db7fe700 1 -- 172.21.15.98:0/1791032330 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f83dc159560 msgr2=0x7f83dc09a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:35.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:35.749+0000 7f83db7fe700 1 --2- 172.21.15.98:0/1791032330 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f83dc159560 0x7f83dc09a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:35.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:35.749+0000 7f83db7fe700 1 -- 172.21.15.98:0/1791032330 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f83dc0b0760 msgr2=0x7f83dc0acfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:35.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:35.749+0000 7f83db7fe700 1 --2- 172.21.15.98:0/1791032330 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f83dc0b0760 0x7f83dc0acfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:35.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:35.749+0000 7f83db7fe700 1 -- 172.21.15.98:0/1791032330 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83d00093c0 con 0x7f83dc0b49a0 2023-10-06T05:06:35.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:35.749+0000 7f83db7fe700 1 --2- 172.21.15.98:0/1791032330 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83dc0b49a0 0x7f83dc157aa0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f83cc00c790 tx=0x7f83cc007f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:06:35.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:35.749+0000 7f83c3fff700 1 -- 172.21.15.98:0/1791032330 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83cc011590 con 0x7f83dc0b49a0 2023-10-06T05:06:35.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:35.749+0000 7f83e243c700 1 -- 172.21.15.98:0/1791032330 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83dc0ab4e0 con 0x7f83dc0b49a0 2023-10-06T05:06:35.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:35.749+0000 7f83c3fff700 1 -- 172.21.15.98:0/1791032330 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83cc011710 con 0x7f83dc0b49a0 2023-10-06T05:06:35.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:35.749+0000 7f83c3fff700 1 -- 172.21.15.98:0/1791032330 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83cc011ad0 con 0x7f83dc0b49a0 2023-10-06T05:06:35.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:35.749+0000 7f83e243c700 1 -- 172.21.15.98:0/1791032330 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83dc157fe0 con 0x7f83dc0b49a0 2023-10-06T05:06:35.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:35.750+0000 7f83c3fff700 1 -- 172.21.15.98:0/1791032330 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f83cc01c6c0 con 0x7f83dc0b49a0 2023-10-06T05:06:35.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:35.750+0000 7f83c3fff700 1 -- 172.21.15.98:0/1791032330 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1641+0+0 (secure 0 0 0) 0x7f83cc011e40 con 0x7f83dc0b49a0 2023-10-06T05:06:35.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:35.750+0000 7f83e243c700 1 -- 172.21.15.98:0/1791032330 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83dc0abca0 con 0x7f83dc0b49a0 2023-10-06T05:06:35.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:35.752+0000 7f83c3fff700 1 -- 172.21.15.98:0/1791032330 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f83cc017070 con 0x7f83dc0b49a0 2023-10-06T05:06:35.759 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-10-06T05:06:35.757+0000 7f983cd62180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-06T05:06:35.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:35.821+0000 7f83e243c700 1 -- 172.21.15.98:0/1791032330 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f83dc064a90 con 0x7f83dc0b49a0 2023-10-06T05:06:35.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:35.822+0000 7f83c3fff700 1 -- 172.21.15.98:0/1791032330 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7013 (secure 0 0 0) 0x7f83cc012c90 con 0x7f83dc0b49a0 2023-10-06T05:06:35.824 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:06:35.825 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":10,"fsid":"f50c5df5-cad5-47a4-9bb1-8c791d4311ec","created":"2023-10-06T05:06:32.971175+0000","modified":"2023-10-06T05:06:35.544511+0000","last_up_change":"0.000000","last_in_change":"2023-10-06T05:06:35.544511+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"d9495d95-da75-4c3b-8927-e9bae6a8b093","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f25ba5be-d65e-4707-a134-154092deb754","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ab9e028-945c-43e2-8c95-3b0b4cf38e39","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3bb803ff-1738-46bf-be62-2267bf378822","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d57a4c9-6462-43f6-b64a-7f9e9b4353d9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"83287ea4-e44d-4af9-bc77-36d47c60d4a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"496a6272-892e-43c4-a145-206623c22f99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"49967e34-8b07-4bab-a13e-0c16d6a0f3a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T05:06:35.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:35.823+0000 7f83e243c700 1 -- 172.21.15.98:0/1791032330 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83dc0b49a0 msgr2=0x7f83dc157aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:35.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:35.823+0000 7f83e243c700 1 --2- 172.21.15.98:0/1791032330 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83dc0b49a0 0x7f83dc157aa0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f83cc00c790 tx=0x7f83cc007f20 comp rx=0 tx=0).stop 2023-10-06T05:06:35.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:35.823+0000 7f83e243c700 1 -- 172.21.15.98:0/1791032330 shutdown_connections 2023-10-06T05:06:35.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:35.823+0000 7f83e243c700 1 --2- 172.21.15.98:0/1791032330 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f83dc159560 0x7f83dc09a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:35.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:35.823+0000 7f83e243c700 1 --2- 172.21.15.98:0/1791032330 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f83dc0b0760 0x7f83dc0acfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:35.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:35.823+0000 7f83e243c700 1 --2- 172.21.15.98:0/1791032330 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83dc0b49a0 0x7f83dc157aa0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:35.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:35.823+0000 7f83e243c700 1 -- 172.21.15.98:0/1791032330 >> 172.21.15.98:0/1791032330 conn(0x7f83dc0997f0 msgr2=0x7f83dc158960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:06:35.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:35.823+0000 7f83e243c700 1 -- 172.21.15.98:0/1791032330 shutdown_connections 2023-10-06T05:06:35.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:35.823+0000 7f83e243c700 1 -- 172.21.15.98:0/1791032330 wait complete. 2023-10-06T05:06:35.828 INFO:tasks.ceph.mgr.x.smithi174.stderr:2023-10-06T05:06:35.824+0000 7fcbfec4d180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-06T05:06:35.838 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-10-06T05:06:35.838 INFO:tasks.ceph:Waiting for OSDs to come up 2023-10-06T05:06:36.174 INFO:tasks.ceph.osd.0.smithi098.stderr:2023-10-06T05:06:36.171+0000 7f2f6854d540 -1 Falling back to public interface 2023-10-06T05:06:36.186 INFO:tasks.ceph.osd.1.smithi098.stderr:2023-10-06T05:06:36.184+0000 7f418ebbe540 -1 Falling back to public interface 2023-10-06T05:06:36.202 INFO:tasks.ceph.osd.4.smithi174.stderr:2023-10-06T05:06:36.201+0000 7fc0ab2b8540 -1 Falling back to public interface 2023-10-06T05:06:36.203 INFO:tasks.ceph.osd.5.smithi174.stderr:2023-10-06T05:06:36.202+0000 7fc956ddc540 -1 Falling back to public interface 2023-10-06T05:06:36.205 INFO:tasks.ceph.osd.3.smithi098.stderr:2023-10-06T05:06:36.203+0000 7f7d60511540 -1 Falling back to public interface 2023-10-06T05:06:36.206 INFO:tasks.ceph.osd.2.smithi098.stderr:2023-10-06T05:06:36.203+0000 7fc32aa08540 -1 Falling back to public interface 2023-10-06T05:06:36.227 INFO:tasks.ceph.osd.6.smithi174.stderr:2023-10-06T05:06:36.225+0000 7faa4bbca540 -1 Falling back to public interface 2023-10-06T05:06:36.237 INFO:tasks.ceph.osd.7.smithi174.stderr:2023-10-06T05:06:36.236+0000 7f2ce9d0d540 -1 Falling back to public interface 2023-10-06T05:06:36.355 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-10-06T05:06:36.353+0000 7f983cd62180 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-06T05:06:36.406 INFO:tasks.ceph.mgr.x.smithi174.stderr:2023-10-06T05:06:36.404+0000 7fcbfec4d180 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-06T05:06:36.493 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-10-06T05:06:36.491+0000 7f983cd62180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-06T05:06:36.545 INFO:tasks.ceph.mgr.x.smithi174.stderr:2023-10-06T05:06:36.543+0000 7fcbfec4d180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-06T05:06:36.585 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-10-06T05:06:36.583+0000 7f983cd62180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-06T05:06:36.636 INFO:tasks.ceph.mgr.x.smithi174.stderr:2023-10-06T05:06:36.635+0000 7fcbfec4d180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-06T05:06:36.643 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-06T05:06:36.755 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:36.753+0000 7fe4ed961700 1 Processor -- start 2023-10-06T05:06:36.755 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:36.753+0000 7fe4ed961700 1 -- start start 2023-10-06T05:06:36.757 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:36.753+0000 7fe4ed961700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fe4e80cc450 0x7fe4e806aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:36.757 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:36.753+0000 7fe4ed961700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe4e806b030 0x7fe4e8158500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:36.757 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:36.753+0000 7fe4ed961700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4e806b820 con 0x7fe4e8156310 2023-10-06T05:06:36.757 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:36.753+0000 7fe4ed961700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4e81576f0 con 0x7fe4e806b030 2023-10-06T05:06:36.757 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:36.753+0000 7fe4ed961700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4e806bb00 con 0x7fe4e80cc450 2023-10-06T05:06:36.758 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:36.754+0000 7fe4e6ffd700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fe4e8156310 0x7fe4e80cbd40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:41862/0 (socket says 172.21.15.98:41862) 2023-10-06T05:06:36.758 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:36.754+0000 7fe4e6ffd700 1 -- 172.21.15.98:0/2191952183 learned_addr learned my addr 172.21.15.98:0/2191952183 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:06:36.758 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:36.754+0000 7fe4e77fe700 1 --2- 172.21.15.98:0/2191952183 >> v2:172.21.15.98:3300/0 conn(0x7fe4e806b030 0x7fe4e8158500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:36.758 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:36.754+0000 7fe4e67fc700 1 --2- 172.21.15.98:0/2191952183 >> v2:172.21.15.174:3300/0 conn(0x7fe4e80cc450 0x7fe4e806aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:36.758 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:36.754+0000 7fe4e5ffb700 1 -- 172.21.15.98:0/2191952183 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 536494761 0 0) 0x7fe4e806b820 con 0x7fe4e8156310 2023-10-06T05:06:36.758 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:36.754+0000 7fe4e5ffb700 1 -- 172.21.15.98:0/2191952183 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4d0003570 con 0x7fe4e8156310 2023-10-06T05:06:36.758 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:36.754+0000 7fe4e77fe700 1 -- 172.21.15.98:0/2191952183 >> v2:172.21.15.174:3300/0 conn(0x7fe4e80cc450 msgr2=0x7fe4e806aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:06:36.759 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:36.754+0000 7fe4e77fe700 1 --2- 172.21.15.98:0/2191952183 >> v2:172.21.15.174:3300/0 conn(0x7fe4e80cc450 0x7fe4e806aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:36.759 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:36.754+0000 7fe4e77fe700 1 -- 172.21.15.98:0/2191952183 >> v1:172.21.15.98:6789/0 conn(0x7fe4e8156310 legacy=0x7fe4e80cbd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:36.759 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:36.754+0000 7fe4e77fe700 1 -- 172.21.15.98:0/2191952183 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4e8157a20 con 0x7fe4e806b030 2023-10-06T05:06:36.759 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:36.754+0000 7fe4e77fe700 1 --2- 172.21.15.98:0/2191952183 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4e806b030 0x7fe4e8158500 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe4e80cac20 tx=0x7fe4e0009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c51b4895700ad663 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:06:36.759 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:36.754+0000 7fe4e5ffb700 1 -- 172.21.15.98:0/2191952183 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4e000e070 con 0x7fe4e806b030 2023-10-06T05:06:36.759 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:36.754+0000 7fe4e5ffb700 1 -- 172.21.15.98:0/2191952183 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4e000bab0 con 0x7fe4e806b030 2023-10-06T05:06:36.759 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:36.754+0000 7fe4e5ffb700 1 -- 172.21.15.98:0/2191952183 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4e000be30 con 0x7fe4e806b030 2023-10-06T05:06:36.759 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:36.755+0000 7fe4ed961700 1 -- 172.21.15.98:0/2191952183 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4e806b030 msgr2=0x7fe4e8158500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:36.760 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:36.755+0000 7fe4ed961700 1 --2- 172.21.15.98:0/2191952183 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4e806b030 0x7fe4e8158500 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe4e80cac20 tx=0x7fe4e0009780 comp rx=0 tx=0).stop 2023-10-06T05:06:36.760 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:36.755+0000 7fe4ed961700 1 -- 172.21.15.98:0/2191952183 shutdown_connections 2023-10-06T05:06:36.760 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:36.755+0000 7fe4ed961700 1 --2- 172.21.15.98:0/2191952183 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4e806b030 0x7fe4e8158500 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:36.760 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:36.755+0000 7fe4ed961700 1 --2- 172.21.15.98:0/2191952183 >> v2:172.21.15.174:3300/0 conn(0x7fe4e80cc450 0x7fe4e806aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:36.760 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:36.755+0000 7fe4ed961700 1 -- 172.21.15.98:0/2191952183 >> 172.21.15.98:0/2191952183 conn(0x7fe4e80675b0 msgr2=0x7fe4e80679b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:06:36.760 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:36.755+0000 7fe4ed961700 1 -- 172.21.15.98:0/2191952183 shutdown_connections 2023-10-06T05:06:36.760 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:36.755+0000 7fe4ed961700 1 -- 172.21.15.98:0/2191952183 wait complete. 2023-10-06T05:06:36.763 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:36.756+0000 7fe4ed961700 1 Processor -- start 2023-10-06T05:06:36.763 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:36.756+0000 7fe4ed961700 1 -- start start 2023-10-06T05:06:36.764 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:36.757+0000 7fe4ed961700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4e806b030 0x7fe4e8152df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:36.764 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:36.757+0000 7fe4ed961700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe4e80cc450 0x7fe4e806c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:36.764 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:36.757+0000 7fe4ed961700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe4e8156310 0x7fe4e806c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:36.764 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:36.757+0000 7fe4ed961700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e815aac0 con 0x7fe4e806b030 2023-10-06T05:06:36.764 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:36.757+0000 7fe4ed961700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e8159bd0 con 0x7fe4e80cc450 2023-10-06T05:06:36.764 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:36.757+0000 7fe4ed961700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e8159eb0 con 0x7fe4e8156310 2023-10-06T05:06:36.764 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:36.757+0000 7fe4e6ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4e806b030 0x7fe4e8152df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:36.765 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:36.757+0000 7fe4e6ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4e806b030 0x7fe4e8152df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39194/0 (socket says 172.21.15.98:39194) 2023-10-06T05:06:36.765 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:36.757+0000 7fe4e6ffd700 1 -- 172.21.15.98:0/1966597273 learned_addr learned my addr 172.21.15.98:0/1966597273 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:06:36.765 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:36.757+0000 7fe4e67fc700 1 --2- 172.21.15.98:0/1966597273 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe4e80cc450 0x7fe4e806c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:36.765 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:36.757+0000 7fe4e77fe700 1 --2- 172.21.15.98:0/1966597273 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe4e8156310 0x7fe4e806c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:36.765 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:36.757+0000 7fe4e6ffd700 1 -- 172.21.15.98:0/1966597273 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe4e8156310 msgr2=0x7fe4e806c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:36.765 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:36.757+0000 7fe4e6ffd700 1 --2- 172.21.15.98:0/1966597273 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe4e8156310 0x7fe4e806c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:36.765 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:36.757+0000 7fe4e6ffd700 1 -- 172.21.15.98:0/1966597273 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe4e80cc450 msgr2=0x7fe4e806c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:36.765 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:36.757+0000 7fe4e6ffd700 1 --2- 172.21.15.98:0/1966597273 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe4e80cc450 0x7fe4e806c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:36.766 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:36.757+0000 7fe4e6ffd700 1 -- 172.21.15.98:0/1966597273 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4e0009a20 con 0x7fe4e806b030 2023-10-06T05:06:36.766 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:36.757+0000 7fe4e6ffd700 1 --2- 172.21.15.98:0/1966597273 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4e806b030 0x7fe4e8152df0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe4d80011e0 tx=0x7fe4d8007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:06:36.766 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:36.757+0000 7fe4ceffd700 1 -- 172.21.15.98:0/1966597273 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4d8005b30 con 0x7fe4e806b030 2023-10-06T05:06:36.766 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:36.757+0000 7fe4ceffd700 1 -- 172.21.15.98:0/1966597273 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4d8005c90 con 0x7fe4e806b030 2023-10-06T05:06:36.766 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:36.757+0000 7fe4ed961700 1 -- 172.21.15.98:0/1966597273 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4e815a190 con 0x7fe4e806b030 2023-10-06T05:06:36.766 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:36.757+0000 7fe4ceffd700 1 -- 172.21.15.98:0/1966597273 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4d8007320 con 0x7fe4e806b030 2023-10-06T05:06:36.766 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:36.757+0000 7fe4ed961700 1 -- 172.21.15.98:0/1966597273 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4e815a750 con 0x7fe4e806b030 2023-10-06T05:06:36.766 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:36.758+0000 7fe4ceffd700 1 -- 172.21.15.98:0/1966597273 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fe4d801c440 con 0x7fe4e806b030 2023-10-06T05:06:36.767 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:36.758+0000 7fe4ceffd700 1 -- 172.21.15.98:0/1966597273 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1641+0+0 (secure 0 0 0) 0x7fe4d8005e00 con 0x7fe4e806b030 2023-10-06T05:06:36.767 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:36.758+0000 7fe4ed961700 1 -- 172.21.15.98:0/1966597273 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4e8064a90 con 0x7fe4e806b030 2023-10-06T05:06:36.767 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:36.761+0000 7fe4ceffd700 1 -- 172.21.15.98:0/1966597273 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fe4d8017070 con 0x7fe4e806b030 2023-10-06T05:06:36.833 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:36.831+0000 7fe4ed961700 1 -- 172.21.15.98:0/1966597273 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe4e8157d00 con 0x7fe4e806b030 2023-10-06T05:06:36.833 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:36.831+0000 7fe4ceffd700 1 -- 172.21.15.98:0/1966597273 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7013 (secure 0 0 0) 0x7fe4d801c920 con 0x7fe4e806b030 2023-10-06T05:06:36.834 INFO:teuthology.misc.health.smithi098.stdout: 2023-10-06T05:06:36.834 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":10,"fsid":"f50c5df5-cad5-47a4-9bb1-8c791d4311ec","created":"2023-10-06T05:06:32.971175+0000","modified":"2023-10-06T05:06:35.544511+0000","last_up_change":"0.000000","last_in_change":"2023-10-06T05:06:35.544511+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"d9495d95-da75-4c3b-8927-e9bae6a8b093","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f25ba5be-d65e-4707-a134-154092deb754","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5ab9e028-945c-43e2-8c95-3b0b4cf38e39","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3bb803ff-1738-46bf-be62-2267bf378822","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d57a4c9-6462-43f6-b64a-7f9e9b4353d9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"83287ea4-e44d-4af9-bc77-36d47c60d4a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"496a6272-892e-43c4-a145-206623c22f99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"49967e34-8b07-4bab-a13e-0c16d6a0f3a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T05:06:36.835 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:36.833+0000 7fe4ed961700 1 -- 172.21.15.98:0/1966597273 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4e806b030 msgr2=0x7fe4e8152df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:36.835 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:36.833+0000 7fe4ed961700 1 --2- 172.21.15.98:0/1966597273 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4e806b030 0x7fe4e8152df0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe4d80011e0 tx=0x7fe4d8007df0 comp rx=0 tx=0).stop 2023-10-06T05:06:36.835 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:36.833+0000 7fe4ed961700 1 -- 172.21.15.98:0/1966597273 shutdown_connections 2023-10-06T05:06:36.836 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:36.833+0000 7fe4ed961700 1 --2- 172.21.15.98:0/1966597273 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe4e8156310 0x7fe4e806c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:36.836 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:36.833+0000 7fe4ed961700 1 --2- 172.21.15.98:0/1966597273 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4e806b030 0x7fe4e8152df0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:36.836 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:36.833+0000 7fe4ed961700 1 --2- 172.21.15.98:0/1966597273 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe4e80cc450 0x7fe4e806c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:36.836 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:36.833+0000 7fe4ed961700 1 -- 172.21.15.98:0/1966597273 >> 172.21.15.98:0/1966597273 conn(0x7fe4e80675b0 msgr2=0x7fe4e81552d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:06:36.836 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:36.833+0000 7fe4ed961700 1 -- 172.21.15.98:0/1966597273 shutdown_connections 2023-10-06T05:06:36.836 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:36.833+0000 7fe4ed961700 1 -- 172.21.15.98:0/1966597273 wait complete. 2023-10-06T05:06:36.840 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-10-06T05:06:36.838+0000 7f983cd62180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-06T05:06:36.846 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-10-06T05:06:36.887 INFO:tasks.ceph.mgr.x.smithi174.stderr:2023-10-06T05:06:36.886+0000 7fcbfec4d180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-06T05:06:36.919 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-10-06T05:06:36.917+0000 7f983cd62180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-06T05:06:36.965 INFO:tasks.ceph.mgr.x.smithi174.stderr:2023-10-06T05:06:36.964+0000 7fcbfec4d180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-06T05:06:37.750 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-10-06T05:06:37.748+0000 7f983cd62180 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-06T05:06:37.799 INFO:tasks.ceph.mgr.x.smithi174.stderr:2023-10-06T05:06:37.798+0000 7fcbfec4d180 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-06T05:06:37.824 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-10-06T05:06:37.822+0000 7f983cd62180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-06T05:06:37.874 INFO:tasks.ceph.mgr.x.smithi174.stderr:2023-10-06T05:06:37.873+0000 7fcbfec4d180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-06T05:06:37.899 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-10-06T05:06:37.897+0000 7f983cd62180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-06T05:06:37.949 INFO:tasks.ceph.mgr.x.smithi174.stderr:2023-10-06T05:06:37.948+0000 7fcbfec4d180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-06T05:06:38.033 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-10-06T05:06:38.031+0000 7f983cd62180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-06T05:06:38.081 INFO:tasks.ceph.mgr.x.smithi174.stderr:2023-10-06T05:06:38.080+0000 7fcbfec4d180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-06T05:06:38.117 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-10-06T05:06:38.115+0000 7f983cd62180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-06T05:06:38.167 INFO:tasks.ceph.mgr.x.smithi174.stderr:2023-10-06T05:06:38.166+0000 7fcbfec4d180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-06T05:06:38.270 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-10-06T05:06:38.268+0000 7f983cd62180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-06T05:06:38.320 INFO:tasks.ceph.mgr.x.smithi174.stderr:2023-10-06T05:06:38.318+0000 7fcbfec4d180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-06T05:06:38.347 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-10-06T05:06:38.344+0000 7f983cd62180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-06T05:06:38.396 INFO:tasks.ceph.mgr.x.smithi174.stderr:2023-10-06T05:06:38.395+0000 7fcbfec4d180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-06T05:06:38.423 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-10-06T05:06:38.421+0000 7f983cd62180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-06T05:06:38.474 INFO:tasks.ceph.mgr.x.smithi174.stderr:2023-10-06T05:06:38.472+0000 7fcbfec4d180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-06T05:06:38.499 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-10-06T05:06:38.497+0000 7f983cd62180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-06T05:06:38.550 INFO:tasks.ceph.mgr.x.smithi174.stderr:2023-10-06T05:06:38.549+0000 7fcbfec4d180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-06T05:06:39.415 INFO:tasks.ceph.osd.0.smithi098.stderr:2023-10-06T05:06:39.413+0000 7f2f6854d540 -1 osd.0 0 log_to_monitors true 2023-10-06T05:06:39.481 INFO:tasks.ceph.osd.1.smithi098.stderr:2023-10-06T05:06:39.479+0000 7f418ebbe540 -1 osd.1 0 log_to_monitors true 2023-10-06T05:06:39.501 INFO:tasks.ceph.osd.7.smithi174.stderr:2023-10-06T05:06:39.500+0000 7f2ce9d0d540 -1 osd.7 0 log_to_monitors true 2023-10-06T05:06:39.522 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-10-06T05:06:39.520+0000 7f97f6165700 -1 mgr.server handle_report got status from non-daemon mon.c 2023-10-06T05:06:39.522 INFO:tasks.ceph.mgr.y.smithi098.stderr:2023-10-06T05:06:39.520+0000 7f97f6165700 -1 mgr.server handle_report got status from non-daemon mon.b 2023-10-06T05:06:39.724 INFO:tasks.ceph.osd.5.smithi174.stderr:2023-10-06T05:06:39.723+0000 7fc956ddc540 -1 osd.5 0 log_to_monitors true 2023-10-06T05:06:39.763 INFO:tasks.ceph.osd.2.smithi098.stderr:2023-10-06T05:06:39.761+0000 7fc32aa08540 -1 osd.2 0 log_to_monitors true 2023-10-06T05:06:39.767 INFO:tasks.ceph.osd.4.smithi174.stderr:2023-10-06T05:06:39.766+0000 7fc0ab2b8540 -1 osd.4 0 log_to_monitors true 2023-10-06T05:06:39.769 INFO:tasks.ceph.osd.6.smithi174.stderr:2023-10-06T05:06:39.768+0000 7faa4bbca540 -1 osd.6 0 log_to_monitors true 2023-10-06T05:06:39.800 INFO:tasks.ceph.osd.3.smithi098.stderr:2023-10-06T05:06:39.798+0000 7f7d60511540 -1 osd.3 0 log_to_monitors true 2023-10-06T05:06:41.502 INFO:tasks.ceph.osd.7.smithi174.stderr:2023-10-06T05:06:41.500+0000 7f2ce092f700 -1 osd.7 0 waiting for initial osdmap 2023-10-06T05:06:41.506 INFO:tasks.ceph.osd.7.smithi174.stderr:2023-10-06T05:06:41.504+0000 7f2cd9726700 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-06T05:06:41.875 INFO:tasks.ceph.osd.1.smithi098.stderr:2023-10-06T05:06:41.874+0000 7f41857e0700 -1 osd.1 0 waiting for initial osdmap 2023-10-06T05:06:41.885 INFO:tasks.ceph.osd.1.smithi098.stderr:2023-10-06T05:06:41.883+0000 7f417e5d7700 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-06T05:06:41.952 INFO:tasks.ceph.osd.0.smithi098.stderr:2023-10-06T05:06:41.950+0000 7f2f5f16f700 -1 osd.0 0 waiting for initial osdmap 2023-10-06T05:06:41.955 INFO:tasks.ceph.osd.0.smithi098.stderr:2023-10-06T05:06:41.954+0000 7f2f57f66700 -1 osd.0 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-06T05:06:43.016 INFO:tasks.ceph.osd.2.smithi098.stderr:2023-10-06T05:06:43.014+0000 7fc32162a700 -1 osd.2 0 waiting for initial osdmap 2023-10-06T05:06:43.020 INFO:tasks.ceph.osd.2.smithi098.stderr:2023-10-06T05:06:43.018+0000 7fc31a421700 -1 osd.2 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-06T05:06:43.047 INFO:tasks.ceph.osd.3.smithi098.stderr:2023-10-06T05:06:43.045+0000 7f7d57133700 -1 osd.3 0 waiting for initial osdmap 2023-10-06T05:06:43.051 INFO:tasks.ceph.osd.3.smithi098.stderr:2023-10-06T05:06:43.049+0000 7f7d4ff2a700 -1 osd.3 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-06T05:06:43.324 INFO:tasks.ceph.osd.4.smithi174.stderr:2023-10-06T05:06:43.322+0000 7fc0a36dd700 -1 osd.4 0 waiting for initial osdmap 2023-10-06T05:06:43.329 INFO:tasks.ceph.osd.4.smithi174.stderr:2023-10-06T05:06:43.326+0000 7fc09acd1700 -1 osd.4 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-06T05:06:43.361 INFO:tasks.ceph.osd.6.smithi174.stderr:2023-10-06T05:06:43.358+0000 7faa427ec700 -1 osd.6 0 waiting for initial osdmap 2023-10-06T05:06:43.367 INFO:tasks.ceph.osd.6.smithi174.stderr:2023-10-06T05:06:43.363+0000 7faa3b5e3700 -1 osd.6 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-06T05:06:43.652 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-06T05:06:43.662 INFO:tasks.ceph.osd.5.smithi174.stderr:2023-10-06T05:06:43.660+0000 7fc94d9fe700 -1 osd.5 0 waiting for initial osdmap 2023-10-06T05:06:43.667 INFO:tasks.ceph.osd.5.smithi174.stderr:2023-10-06T05:06:43.664+0000 7fc9467f5700 -1 osd.5 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-06T05:06:43.764 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:43.762+0000 7fdbac2ae700 1 Processor -- start 2023-10-06T05:06:43.764 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:43.762+0000 7fdbac2ae700 1 -- start start 2023-10-06T05:06:43.767 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:43.763+0000 7fdbac2ae700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fdba40b20c0 0x7fdba40b5300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:43.767 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:43.763+0000 7fdbac2ae700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdba4157480 0x7fdba40b3b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:43.767 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:43.763+0000 7fdbac2ae700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdba40b5c70 con 0x7fdba415aa00 2023-10-06T05:06:43.767 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:43.763+0000 7fdbac2ae700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdba415d480 con 0x7fdba40b20c0 2023-10-06T05:06:43.767 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:43.763+0000 7fdbac2ae700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdba415d760 con 0x7fdba4157480 2023-10-06T05:06:43.768 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:43.763+0000 7fdbaa04a700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fdba40b20c0 0x7fdba40b5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:43.768 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:43.763+0000 7fdbaa04a700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fdba40b20c0 0x7fdba40b5300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54622/0 (socket says 172.21.15.98:54622) 2023-10-06T05:06:43.768 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:43.763+0000 7fdbaa04a700 1 -- 172.21.15.98:0/2501637129 learned_addr learned my addr 172.21.15.98:0/2501637129 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:06:43.768 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:43.763+0000 7fdbaa84b700 1 --2- 172.21.15.98:0/2501637129 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdba4157480 0x7fdba40b3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:43.768 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:43.764+0000 7fdbaa04a700 1 -- 172.21.15.98:0/2501637129 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdba4157480 msgr2=0x7fdba40b3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:06:43.768 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:43.764+0000 7fdbaa04a700 1 --2- 172.21.15.98:0/2501637129 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdba4157480 0x7fdba40b3b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:43.768 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:43.764+0000 7fdbaa04a700 1 -- 172.21.15.98:0/2501637129 >> v1:172.21.15.174:6789/0 conn(0x7fdba415aa00 legacy=0x7fdba4156d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:43.768 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:43.764+0000 7fdbaa04a700 1 -- 172.21.15.98:0/2501637129 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdba415da40 con 0x7fdba40b20c0 2023-10-06T05:06:43.768 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:43.764+0000 7fdbaa84b700 1 --2- 172.21.15.98:0/2501637129 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdba4157480 0x7fdba40b3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:06:43.768 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:43.764+0000 7fdbaa04a700 1 --2- 172.21.15.98:0/2501637129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdba40b20c0 0x7fdba40b5300 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fdba0009f20 tx=0x7fdba0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=172ac961977ad8 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:06:43.768 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:43.764+0000 7fdba9048700 1 -- 172.21.15.98:0/2501637129 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdba000e070 con 0x7fdba40b20c0 2023-10-06T05:06:43.769 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:43.764+0000 7fdba9048700 1 -- 172.21.15.98:0/2501637129 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdba000ba80 con 0x7fdba40b20c0 2023-10-06T05:06:43.769 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:43.764+0000 7fdba9048700 1 -- 172.21.15.98:0/2501637129 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdba00092e0 con 0x7fdba40b20c0 2023-10-06T05:06:43.769 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:43.764+0000 7fdbac2ae700 1 -- 172.21.15.98:0/2501637129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdba40b20c0 msgr2=0x7fdba40b5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:43.769 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:43.764+0000 7fdbac2ae700 1 --2- 172.21.15.98:0/2501637129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdba40b20c0 0x7fdba40b5300 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fdba0009f20 tx=0x7fdba0009840 comp rx=0 tx=0).stop 2023-10-06T05:06:43.769 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:43.764+0000 7fdbac2ae700 1 -- 172.21.15.98:0/2501637129 shutdown_connections 2023-10-06T05:06:43.769 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:43.764+0000 7fdbac2ae700 1 --2- 172.21.15.98:0/2501637129 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdba4157480 0x7fdba40b3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:43.769 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:43.764+0000 7fdbac2ae700 1 --2- 172.21.15.98:0/2501637129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdba40b20c0 0x7fdba40b5300 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:43.769 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:43.764+0000 7fdbac2ae700 1 -- 172.21.15.98:0/2501637129 >> 172.21.15.98:0/2501637129 conn(0x7fdba406c2b0 msgr2=0x7fdba406c6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:06:43.769 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:43.764+0000 7fdbac2ae700 1 -- 172.21.15.98:0/2501637129 shutdown_connections 2023-10-06T05:06:43.770 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:43.764+0000 7fdbac2ae700 1 -- 172.21.15.98:0/2501637129 wait complete. 2023-10-06T05:06:43.773 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:43.765+0000 7fdbac2ae700 1 Processor -- start 2023-10-06T05:06:43.773 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:43.766+0000 7fdbac2ae700 1 -- start start 2023-10-06T05:06:43.773 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:43.766+0000 7fdbac2ae700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdba40b20c0 0x7fdba4154000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:43.773 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:43.766+0000 7fdbac2ae700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdba4157480 0x7fdba40c8e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:43.774 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:43.766+0000 7fdbac2ae700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdba415aa00 0x7fdba416b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:43.774 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:43.766+0000 7fdbac2ae700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdba40c6220 con 0x7fdba415aa00 2023-10-06T05:06:43.774 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:43.766+0000 7fdbac2ae700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdba40c6500 con 0x7fdba4157480 2023-10-06T05:06:43.774 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:43.766+0000 7fdbac2ae700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdba40c67e0 con 0x7fdba40b20c0 2023-10-06T05:06:43.774 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:43.766+0000 7fdbaa84b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdba415aa00 0x7fdba416b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:43.774 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:43.766+0000 7fdbaa84b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdba415aa00 0x7fdba416b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54628/0 (socket says 172.21.15.98:54628) 2023-10-06T05:06:43.774 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:43.766+0000 7fdbaa84b700 1 -- 172.21.15.98:0/2717333456 learned_addr learned my addr 172.21.15.98:0/2717333456 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:06:43.774 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:43.766+0000 7fdba9849700 1 --2- 172.21.15.98:0/2717333456 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdba4157480 0x7fdba40c8e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:43.774 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:43.766+0000 7fdbaa04a700 1 --2- 172.21.15.98:0/2717333456 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdba40b20c0 0x7fdba4154000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:43.774 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:43.767+0000 7fdbaa84b700 1 -- 172.21.15.98:0/2717333456 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdba40b20c0 msgr2=0x7fdba4154000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:43.775 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:43.767+0000 7fdbaa84b700 1 --2- 172.21.15.98:0/2717333456 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdba40b20c0 0x7fdba4154000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:43.775 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:43.767+0000 7fdbaa84b700 1 -- 172.21.15.98:0/2717333456 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdba4157480 msgr2=0x7fdba40c8e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:43.775 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:43.767+0000 7fdbaa84b700 1 --2- 172.21.15.98:0/2717333456 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdba4157480 0x7fdba40c8e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:43.775 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:43.767+0000 7fdbaa84b700 1 -- 172.21.15.98:0/2717333456 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdba0009ae0 con 0x7fdba415aa00 2023-10-06T05:06:43.775 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:43.767+0000 7fdbaa84b700 1 --2- 172.21.15.98:0/2717333456 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdba415aa00 0x7fdba416b3f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c00e870 tx=0x7fdb9c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:06:43.775 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:43.767+0000 7fdb9a7fc700 1 -- 172.21.15.98:0/2717333456 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb9c013410 con 0x7fdba415aa00 2023-10-06T05:06:43.775 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:43.767+0000 7fdb9a7fc700 1 -- 172.21.15.98:0/2717333456 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb9c013570 con 0x7fdba415aa00 2023-10-06T05:06:43.775 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:43.767+0000 7fdb9a7fc700 1 -- 172.21.15.98:0/2717333456 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb9c0138f0 con 0x7fdba415aa00 2023-10-06T05:06:43.775 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:43.767+0000 7fdbac2ae700 1 -- 172.21.15.98:0/2717333456 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdba40c6ac0 con 0x7fdba415aa00 2023-10-06T05:06:43.775 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:43.767+0000 7fdbac2ae700 1 -- 172.21.15.98:0/2717333456 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdba4154540 con 0x7fdba415aa00 2023-10-06T05:06:43.776 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:43.768+0000 7fdbac2ae700 1 -- 172.21.15.98:0/2717333456 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdba40c6fe0 con 0x7fdba415aa00 2023-10-06T05:06:43.776 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:43.769+0000 7fdb9a7fc700 1 -- 172.21.15.98:0/2717333456 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 4) v1 ==== 84684+0+0 (secure 0 0 0) 0x7fdb9c013ca0 con 0x7fdba415aa00 2023-10-06T05:06:43.776 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:43.769+0000 7fdb9a7fc700 1 --2- 172.21.15.98:0/2717333456 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fdb88066ea0 0x7fdb88069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:43.776 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:43.769+0000 7fdb9a7fc700 1 -- 172.21.15.98:0/2717333456 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 3067+0+0 (secure 0 0 0) 0x7fdb9c083620 con 0x7fdba415aa00 2023-10-06T05:06:43.776 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:43.769+0000 7fdbaa04a700 1 --2- 172.21.15.98:0/2717333456 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fdb88066ea0 0x7fdb88069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:43.776 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:43.769+0000 7fdbaa04a700 1 --2- 172.21.15.98:0/2717333456 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fdb88066ea0 0x7fdb88069350 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fdba4155930 tx=0x7fdba0018000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:06:43.777 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:43.773+0000 7fdb9a7fc700 1 -- 172.21.15.98:0/2717333456 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fdb9c0513c0 con 0x7fdba415aa00 2023-10-06T05:06:43.964 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:43.962+0000 7fdbac2ae700 1 -- 172.21.15.98:0/2717333456 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdba4154760 con 0x7fdba415aa00 2023-10-06T05:06:43.964 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:43.962+0000 7fdb9a7fc700 1 -- 172.21.15.98:0/2717333456 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) v1 ==== 74+0+8313 (secure 0 0 0) 0x7fdb9c0549e0 con 0x7fdba415aa00 2023-10-06T05:06:43.965 INFO:teuthology.misc.health.smithi098.stdout: 2023-10-06T05:06:43.965 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":14,"fsid":"f50c5df5-cad5-47a4-9bb1-8c791d4311ec","created":"2023-10-06T05:06:32.971175+0000","modified":"2023-10-06T05:06:42.856799+0000","last_up_change":"2023-10-06T05:06:42.856799+0000","last_in_change":"2023-10-06T05:06:35.544511+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"d9495d95-da75-4c3b-8927-e9bae6a8b093","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":90627},{"type":"v1","addr":"172.21.15.98:6801","nonce":90627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":90627},{"type":"v1","addr":"172.21.15.98:6803","nonce":90627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":90627},{"type":"v1","addr":"172.21.15.98:6807","nonce":90627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":90627},{"type":"v1","addr":"172.21.15.98:6805","nonce":90627}]},"public_addr":"172.21.15.98:6801/90627","cluster_addr":"172.21.15.98:6803/90627","heartbeat_back_addr":"172.21.15.98:6807/90627","heartbeat_front_addr":"172.21.15.98:6805/90627","state":["exists","up"]},{"osd":1,"uuid":"f25ba5be-d65e-4707-a134-154092deb754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":90628},{"type":"v1","addr":"172.21.15.98:6809","nonce":90628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":90628},{"type":"v1","addr":"172.21.15.98:6811","nonce":90628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":90628},{"type":"v1","addr":"172.21.15.98:6815","nonce":90628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":90628},{"type":"v1","addr":"172.21.15.98:6813","nonce":90628}]},"public_addr":"172.21.15.98:6809/90628","cluster_addr":"172.21.15.98:6811/90628","heartbeat_back_addr":"172.21.15.98:6815/90628","heartbeat_front_addr":"172.21.15.98:6813/90628","state":["exists","up"]},{"osd":2,"uuid":"5ab9e028-945c-43e2-8c95-3b0b4cf38e39","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3bb803ff-1738-46bf-be62-2267bf378822","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7d57a4c9-6462-43f6-b64a-7f9e9b4353d9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"83287ea4-e44d-4af9-bc77-36d47c60d4a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"496a6272-892e-43c4-a145-206623c22f99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"49967e34-8b07-4bab-a13e-0c16d6a0f3a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":90878},{"type":"v1","addr":"172.21.15.174:6825","nonce":90878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":90878},{"type":"v1","addr":"172.21.15.174:6827","nonce":90878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":90878},{"type":"v1","addr":"172.21.15.174:6831","nonce":90878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":90878},{"type":"v1","addr":"172.21.15.174:6829","nonce":90878}]},"public_addr":"172.21.15.174:6825/90878","cluster_addr":"172.21.15.174:6827/90878","heartbeat_back_addr":"172.21.15.174:6831/90878","heartbeat_front_addr":"172.21.15.174:6829/90878","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.508682+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T05:06:43.967 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:43.964+0000 7fdbac2ae700 1 -- 172.21.15.98:0/2717333456 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fdb88066ea0 msgr2=0x7fdb88069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:43.967 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:43.964+0000 7fdbac2ae700 1 --2- 172.21.15.98:0/2717333456 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fdb88066ea0 0x7fdb88069350 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fdba4155930 tx=0x7fdba0018000 comp rx=0 tx=0).stop 2023-10-06T05:06:43.967 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:43.964+0000 7fdbac2ae700 1 -- 172.21.15.98:0/2717333456 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdba415aa00 msgr2=0x7fdba416b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:43.967 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:43.964+0000 7fdbac2ae700 1 --2- 172.21.15.98:0/2717333456 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdba415aa00 0x7fdba416b3f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c00e870 tx=0x7fdb9c008040 comp rx=0 tx=0).stop 2023-10-06T05:06:43.967 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:43.964+0000 7fdbac2ae700 1 -- 172.21.15.98:0/2717333456 shutdown_connections 2023-10-06T05:06:43.967 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:43.964+0000 7fdbac2ae700 1 --2- 172.21.15.98:0/2717333456 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fdb88066ea0 0x7fdb88069350 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:43.967 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:43.964+0000 7fdbac2ae700 1 --2- 172.21.15.98:0/2717333456 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdba415aa00 0x7fdba416b3f0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:43.967 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:43.964+0000 7fdbac2ae700 1 --2- 172.21.15.98:0/2717333456 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdba40b20c0 0x7fdba4154000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:43.967 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:43.964+0000 7fdbac2ae700 1 --2- 172.21.15.98:0/2717333456 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdba4157480 0x7fdba40c8e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:43.968 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:43.964+0000 7fdbac2ae700 1 -- 172.21.15.98:0/2717333456 >> 172.21.15.98:0/2717333456 conn(0x7fdba406c2b0 msgr2=0x7fdba40afd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:06:43.968 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:43.964+0000 7fdbac2ae700 1 -- 172.21.15.98:0/2717333456 shutdown_connections 2023-10-06T05:06:43.968 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:43.964+0000 7fdbac2ae700 1 -- 172.21.15.98:0/2717333456 wait complete. 2023-10-06T05:06:43.977 DEBUG:teuthology.misc:3 of 8 OSDs are up 2023-10-06T05:06:50.784 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-06T05:06:50.900 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:50.898+0000 7fb177f7a700 1 Processor -- start 2023-10-06T05:06:50.900 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:50.898+0000 7fb177f7a700 1 -- start start 2023-10-06T05:06:50.902 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:50.899+0000 7fb177f7a700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb170156160 0x7fb170159fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:50.902 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:50.899+0000 7fb177f7a700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb17006c2b0 0x7fb170156530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:50.902 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:50.899+0000 7fb177f7a700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fb1700c8060 0x7fb1700c84e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:50.902 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:50.899+0000 7fb177f7a700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1700c8a20 con 0x7fb17006c2b0 2023-10-06T05:06:50.903 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:50.899+0000 7fb177f7a700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1700b07b0 con 0x7fb1700c8060 2023-10-06T05:06:50.903 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:50.899+0000 7fb177f7a700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb17015a5a0 con 0x7fb170156160 2023-10-06T05:06:50.903 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:50.899+0000 7fb175515700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb17006c2b0 0x7fb170156530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:50.903 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:50.899+0000 7fb175515700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb17006c2b0 0x7fb170156530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53986/0 (socket says 172.21.15.98:53986) 2023-10-06T05:06:50.903 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:50.899+0000 7fb175515700 1 -- 172.21.15.98:0/2387368637 learned_addr learned my addr 172.21.15.98:0/2387368637 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:06:50.903 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:50.899+0000 7fb176517700 1 --2- 172.21.15.98:0/2387368637 >> v2:172.21.15.174:3300/0 conn(0x7fb1700c8060 0x7fb1700c84e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:50.903 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:50.899+0000 7fb175d16700 1 --2- 172.21.15.98:0/2387368637 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb170156160 0x7fb170159fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:50.903 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:50.899+0000 7fb175515700 1 -- 172.21.15.98:0/2387368637 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb170156160 msgr2=0x7fb170159fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:06:50.903 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:50.899+0000 7fb175515700 1 --2- 172.21.15.98:0/2387368637 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb170156160 0x7fb170159fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:50.903 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:50.899+0000 7fb175515700 1 -- 172.21.15.98:0/2387368637 >> v2:172.21.15.174:3300/0 conn(0x7fb1700c8060 msgr2=0x7fb1700c84e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:06:50.903 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:50.899+0000 7fb175515700 1 --2- 172.21.15.98:0/2387368637 >> v2:172.21.15.174:3300/0 conn(0x7fb1700c8060 0x7fb1700c84e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:50.904 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:50.899+0000 7fb175515700 1 -- 172.21.15.98:0/2387368637 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb17015a880 con 0x7fb17006c2b0 2023-10-06T05:06:50.904 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:50.899+0000 7fb175515700 1 --2- 172.21.15.98:0/2387368637 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb17006c2b0 0x7fb170156530 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb170070cc0 tx=0x7fb160009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe68a9fd1941b2e0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:06:50.904 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:50.899+0000 7fb174d14700 1 -- 172.21.15.98:0/2387368637 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb16000b860 con 0x7fb17006c2b0 2023-10-06T05:06:50.904 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:50.900+0000 7fb174d14700 1 -- 172.21.15.98:0/2387368637 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb16000b9c0 con 0x7fb17006c2b0 2023-10-06T05:06:50.904 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:50.900+0000 7fb174d14700 1 -- 172.21.15.98:0/2387368637 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb16000bd60 con 0x7fb17006c2b0 2023-10-06T05:06:50.904 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:50.900+0000 7fb177f7a700 1 -- 172.21.15.98:0/2387368637 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb17006c2b0 msgr2=0x7fb170156530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:50.904 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:50.900+0000 7fb177f7a700 1 --2- 172.21.15.98:0/2387368637 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb17006c2b0 0x7fb170156530 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb170070cc0 tx=0x7fb160009250 comp rx=0 tx=0).stop 2023-10-06T05:06:50.904 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:50.900+0000 7fb177f7a700 1 -- 172.21.15.98:0/2387368637 shutdown_connections 2023-10-06T05:06:50.904 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:50.900+0000 7fb177f7a700 1 --2- 172.21.15.98:0/2387368637 >> v2:172.21.15.174:3300/0 conn(0x7fb1700c8060 0x7fb1700c84e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:50.904 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:50.900+0000 7fb177f7a700 1 --2- 172.21.15.98:0/2387368637 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb170156160 0x7fb170159fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:50.904 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:50.900+0000 7fb177f7a700 1 --2- 172.21.15.98:0/2387368637 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb17006c2b0 0x7fb170156530 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:50.904 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:50.900+0000 7fb177f7a700 1 -- 172.21.15.98:0/2387368637 >> 172.21.15.98:0/2387368637 conn(0x7fb17006e680 msgr2=0x7fb17006ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:06:50.905 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:50.900+0000 7fb177f7a700 1 -- 172.21.15.98:0/2387368637 shutdown_connections 2023-10-06T05:06:50.905 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:50.900+0000 7fb177f7a700 1 -- 172.21.15.98:0/2387368637 wait complete. 2023-10-06T05:06:50.907 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:50.901+0000 7fb177f7a700 1 Processor -- start 2023-10-06T05:06:50.908 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:50.901+0000 7fb177f7a700 1 -- start start 2023-10-06T05:06:50.908 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:50.902+0000 7fb177f7a700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb17006c2b0 0x7fb17014faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:50.908 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:50.902+0000 7fb177f7a700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb1700c8060 0x7fb1701526c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:50.908 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:50.902+0000 7fb177f7a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb170156160 0x7fb1700c5bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:50.908 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:50.902+0000 7fb177f7a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1701542d0 con 0x7fb170156160 2023-10-06T05:06:50.908 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:50.902+0000 7fb177f7a700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1701546f0 con 0x7fb1700c8060 2023-10-06T05:06:50.908 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:50.902+0000 7fb177f7a700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb170153650 con 0x7fb17006c2b0 2023-10-06T05:06:50.908 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:50.902+0000 7fb176517700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb170156160 0x7fb1700c5bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:50.908 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:50.902+0000 7fb176517700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb170156160 0x7fb1700c5bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53998/0 (socket says 172.21.15.98:53998) 2023-10-06T05:06:50.908 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:50.902+0000 7fb176517700 1 -- 172.21.15.98:0/1390477794 learned_addr learned my addr 172.21.15.98:0/1390477794 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:06:50.908 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:50.902+0000 7fb175d16700 1 --2- 172.21.15.98:0/1390477794 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb17006c2b0 0x7fb17014faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:50.909 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:50.902+0000 7fb175515700 1 --2- 172.21.15.98:0/1390477794 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb1700c8060 0x7fb1701526c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:50.909 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:50.902+0000 7fb176517700 1 -- 172.21.15.98:0/1390477794 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb17006c2b0 msgr2=0x7fb17014faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:50.909 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:50.902+0000 7fb176517700 1 --2- 172.21.15.98:0/1390477794 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb17006c2b0 0x7fb17014faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:50.909 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:50.902+0000 7fb176517700 1 -- 172.21.15.98:0/1390477794 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb1700c8060 msgr2=0x7fb1701526c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:50.909 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:50.902+0000 7fb176517700 1 --2- 172.21.15.98:0/1390477794 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb1700c8060 0x7fb1701526c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:50.909 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:50.902+0000 7fb176517700 1 -- 172.21.15.98:0/1390477794 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1600094f0 con 0x7fb170156160 2023-10-06T05:06:50.909 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:50.902+0000 7fb175515700 1 --2- 172.21.15.98:0/1390477794 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb1700c8060 0x7fb1701526c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:06:50.909 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:50.903+0000 7fb176517700 1 --2- 172.21.15.98:0/1390477794 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb170156160 0x7fb1700c5bc0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb168009fa0 tx=0x7fb168009900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:06:50.909 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:50.903+0000 7fb165ffb700 1 -- 172.21.15.98:0/1390477794 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb16800ed20 con 0x7fb170156160 2023-10-06T05:06:50.909 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:50.903+0000 7fb165ffb700 1 -- 172.21.15.98:0/1390477794 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb16800ee80 con 0x7fb170156160 2023-10-06T05:06:50.909 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:50.903+0000 7fb177f7a700 1 -- 172.21.15.98:0/1390477794 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb170153930 con 0x7fb170156160 2023-10-06T05:06:50.910 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:50.903+0000 7fb165ffb700 1 -- 172.21.15.98:0/1390477794 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb168014670 con 0x7fb170156160 2023-10-06T05:06:50.910 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:50.903+0000 7fb177f7a700 1 -- 172.21.15.98:0/1390477794 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb170153ef0 con 0x7fb170156160 2023-10-06T05:06:50.910 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:50.904+0000 7fb177f7a700 1 -- 172.21.15.98:0/1390477794 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb170154410 con 0x7fb170156160 2023-10-06T05:06:50.910 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:50.904+0000 7fb165ffb700 1 -- 172.21.15.98:0/1390477794 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fb1680149e0 con 0x7fb170156160 2023-10-06T05:06:50.910 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:50.904+0000 7fb165ffb700 1 --2- 172.21.15.98:0/1390477794 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fb154066f30 0x7fb1540693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:50.910 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:50.905+0000 7fb165ffb700 1 -- 172.21.15.98:0/1390477794 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4853+0+0 (secure 0 0 0) 0x7fb168084a20 con 0x7fb170156160 2023-10-06T05:06:50.910 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:50.905+0000 7fb175d16700 1 --2- 172.21.15.98:0/1390477794 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fb154066f30 0x7fb1540693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:50.910 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:50.905+0000 7fb175d16700 1 --2- 172.21.15.98:0/1390477794 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fb154066f30 0x7fb1540693e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb16c00ab30 tx=0x7fb16c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:06:50.911 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:50.909+0000 7fb165ffb700 1 -- 172.21.15.98:0/1390477794 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb170154410 con 0x7fb170156160 2023-10-06T05:06:51.095 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:51.093+0000 7fb177f7a700 1 -- 172.21.15.98:0/1390477794 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb170154410 con 0x7fb170156160 2023-10-06T05:06:51.095 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:51.094+0000 7fb165ffb700 1 -- 172.21.15.98:0/1390477794 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) v1 ==== 74+0+12127 (secure 0 0 0) 0x7fb170154410 con 0x7fb170156160 2023-10-06T05:06:51.096 INFO:teuthology.misc.health.smithi098.stdout: 2023-10-06T05:06:51.096 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":18,"fsid":"f50c5df5-cad5-47a4-9bb1-8c791d4311ec","created":"2023-10-06T05:06:32.971175+0000","modified":"2023-10-06T05:06:47.025708+0000","last_up_change":"2023-10-06T05:06:44.016489+0000","last_in_change":"2023-10-06T05:06:35.544511+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-06T05:06:44.559671+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"d9495d95-da75-4c3b-8927-e9bae6a8b093","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":90627},{"type":"v1","addr":"172.21.15.98:6801","nonce":90627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":90627},{"type":"v1","addr":"172.21.15.98:6803","nonce":90627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":90627},{"type":"v1","addr":"172.21.15.98:6807","nonce":90627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":90627},{"type":"v1","addr":"172.21.15.98:6805","nonce":90627}]},"public_addr":"172.21.15.98:6801/90627","cluster_addr":"172.21.15.98:6803/90627","heartbeat_back_addr":"172.21.15.98:6807/90627","heartbeat_front_addr":"172.21.15.98:6805/90627","state":["exists","up"]},{"osd":1,"uuid":"f25ba5be-d65e-4707-a134-154092deb754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":90628},{"type":"v1","addr":"172.21.15.98:6809","nonce":90628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":90628},{"type":"v1","addr":"172.21.15.98:6811","nonce":90628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":90628},{"type":"v1","addr":"172.21.15.98:6815","nonce":90628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":90628},{"type":"v1","addr":"172.21.15.98:6813","nonce":90628}]},"public_addr":"172.21.15.98:6809/90628","cluster_addr":"172.21.15.98:6811/90628","heartbeat_back_addr":"172.21.15.98:6815/90628","heartbeat_front_addr":"172.21.15.98:6813/90628","state":["exists","up"]},{"osd":2,"uuid":"5ab9e028-945c-43e2-8c95-3b0b4cf38e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":90662},{"type":"v1","addr":"172.21.15.98:6818","nonce":90662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":90662},{"type":"v1","addr":"172.21.15.98:6822","nonce":90662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":90662},{"type":"v1","addr":"172.21.15.98:6830","nonce":90662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":90662},{"type":"v1","addr":"172.21.15.98:6826","nonce":90662}]},"public_addr":"172.21.15.98:6818/90662","cluster_addr":"172.21.15.98:6822/90662","heartbeat_back_addr":"172.21.15.98:6830/90662","heartbeat_front_addr":"172.21.15.98:6826/90662","state":["exists","up"]},{"osd":3,"uuid":"3bb803ff-1738-46bf-be62-2267bf378822","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6817","nonce":90663},{"type":"v1","addr":"172.21.15.98:6819","nonce":90663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6821","nonce":90663},{"type":"v1","addr":"172.21.15.98:6823","nonce":90663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6829","nonce":90663},{"type":"v1","addr":"172.21.15.98:6831","nonce":90663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6825","nonce":90663},{"type":"v1","addr":"172.21.15.98:6827","nonce":90663}]},"public_addr":"172.21.15.98:6819/90663","cluster_addr":"172.21.15.98:6823/90663","heartbeat_back_addr":"172.21.15.98:6831/90663","heartbeat_front_addr":"172.21.15.98:6827/90663","state":["exists","up"]},{"osd":4,"uuid":"7d57a4c9-6462-43f6-b64a-7f9e9b4353d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":90847},{"type":"v1","addr":"172.21.15.174:6801","nonce":90847}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":90847},{"type":"v1","addr":"172.21.15.174:6803","nonce":90847}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":90847},{"type":"v1","addr":"172.21.15.174:6807","nonce":90847}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":90847},{"type":"v1","addr":"172.21.15.174:6805","nonce":90847}]},"public_addr":"172.21.15.174:6801/90847","cluster_addr":"172.21.15.174:6803/90847","heartbeat_back_addr":"172.21.15.174:6807/90847","heartbeat_front_addr":"172.21.15.174:6805/90847","state":["exists","up"]},{"osd":5,"uuid":"83287ea4-e44d-4af9-bc77-36d47c60d4a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":90848},{"type":"v1","addr":"172.21.15.174:6809","nonce":90848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":90848},{"type":"v1","addr":"172.21.15.174:6811","nonce":90848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":90848},{"type":"v1","addr":"172.21.15.174:6815","nonce":90848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":90848},{"type":"v1","addr":"172.21.15.174:6813","nonce":90848}]},"public_addr":"172.21.15.174:6809/90848","cluster_addr":"172.21.15.174:6811/90848","heartbeat_back_addr":"172.21.15.174:6815/90848","heartbeat_front_addr":"172.21.15.174:6813/90848","state":["exists","up"]},{"osd":6,"uuid":"496a6272-892e-43c4-a145-206623c22f99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":90877},{"type":"v1","addr":"172.21.15.174:6817","nonce":90877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":90877},{"type":"v1","addr":"172.21.15.174:6819","nonce":90877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":90877},{"type":"v1","addr":"172.21.15.174:6823","nonce":90877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":90877},{"type":"v1","addr":"172.21.15.174:6821","nonce":90877}]},"public_addr":"172.21.15.174:6817/90877","cluster_addr":"172.21.15.174:6819/90877","heartbeat_back_addr":"172.21.15.174:6823/90877","heartbeat_front_addr":"172.21.15.174:6821/90877","state":["exists","up"]},{"osd":7,"uuid":"49967e34-8b07-4bab-a13e-0c16d6a0f3a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":90878},{"type":"v1","addr":"172.21.15.174:6825","nonce":90878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":90878},{"type":"v1","addr":"172.21.15.174:6827","nonce":90878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":90878},{"type":"v1","addr":"172.21.15.174:6831","nonce":90878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":90878},{"type":"v1","addr":"172.21.15.174:6829","nonce":90878}]},"public_addr":"172.21.15.174:6825/90878","cluster_addr":"172.21.15.174:6827/90878","heartbeat_back_addr":"172.21.15.174:6831/90878","heartbeat_front_addr":"172.21.15.174:6829/90878","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.446479+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.514266+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.794108+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.810894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.731919+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.717756+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.737435+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.508682+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T05:06:51.098 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:51.096+0000 7fb177f7a700 1 -- 172.21.15.98:0/1390477794 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fb154066f30 msgr2=0x7fb1540693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:51.098 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:51.096+0000 7fb177f7a700 1 --2- 172.21.15.98:0/1390477794 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fb154066f30 0x7fb1540693e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb16c00ab30 tx=0x7fb16c008040 comp rx=0 tx=0).stop 2023-10-06T05:06:51.098 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:51.096+0000 7fb177f7a700 1 -- 172.21.15.98:0/1390477794 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb170156160 msgr2=0x7fb1700c5bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:51.098 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:51.096+0000 7fb177f7a700 1 --2- 172.21.15.98:0/1390477794 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb170156160 0x7fb1700c5bc0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb168009fa0 tx=0x7fb168009900 comp rx=0 tx=0).stop 2023-10-06T05:06:51.098 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:51.096+0000 7fb177f7a700 1 -- 172.21.15.98:0/1390477794 shutdown_connections 2023-10-06T05:06:51.098 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:51.096+0000 7fb177f7a700 1 --2- 172.21.15.98:0/1390477794 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fb154066f30 0x7fb1540693e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:51.099 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:51.096+0000 7fb177f7a700 1 --2- 172.21.15.98:0/1390477794 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb170156160 0x7fb1700c5bc0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:51.099 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:51.096+0000 7fb177f7a700 1 --2- 172.21.15.98:0/1390477794 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb17006c2b0 0x7fb17014faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:51.099 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:51.096+0000 7fb177f7a700 1 --2- 172.21.15.98:0/1390477794 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb1700c8060 0x7fb1701526c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:51.099 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:51.096+0000 7fb177f7a700 1 -- 172.21.15.98:0/1390477794 >> 172.21.15.98:0/1390477794 conn(0x7fb17006e680 msgr2=0x7fb17006ac80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:06:51.099 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:51.096+0000 7fb177f7a700 1 -- 172.21.15.98:0/1390477794 shutdown_connections 2023-10-06T05:06:51.099 INFO:teuthology.misc.health.smithi098.stderr:2023-10-06T05:06:51.096+0000 7fb177f7a700 1 -- 172.21.15.98:0/1390477794 wait complete. 2023-10-06T05:06:51.108 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-10-06T05:06:51.109 INFO:tasks.ceph:Creating RBD pool 2023-10-06T05:06:51.109 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --cluster ceph osd pool create rbd 8 2023-10-06T05:06:51.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.220+0000 7f263b59e700 1 Processor -- start 2023-10-06T05:06:51.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.220+0000 7f263b59e700 1 -- start start 2023-10-06T05:06:51.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.221+0000 7f263b59e700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f263c0c3f50 0x7f263c162240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:51.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.221+0000 7f263b59e700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f263c0cadd0 con 0x7f263c0b05d0 2023-10-06T05:06:51.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.221+0000 7f263b59e700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f263c1577d0 con 0x7f263c0cce20 2023-10-06T05:06:51.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.221+0000 7f263b59e700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f263c0c9ae0 con 0x7f263c0c3f50 2023-10-06T05:06:51.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.221+0000 7f263a59c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f263c0b05d0 0x7f263c0cc710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:49302/0 (socket says 172.21.15.98:49302) 2023-10-06T05:06:51.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.221+0000 7f263a59c700 1 -- 172.21.15.98:0/4143721397 learned_addr learned my addr 172.21.15.98:0/4143721397 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:06:51.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.221+0000 7f263ad9d700 1 --2- 172.21.15.98:0/4143721397 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f263c0c3f50 0x7f263c162240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:51.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.221+0000 7f263959a700 1 -- 172.21.15.98:0/4143721397 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1520657936 0 0) 0x7f263c0cadd0 con 0x7f263c0b05d0 2023-10-06T05:06:51.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.221+0000 7f263959a700 1 -- 172.21.15.98:0/4143721397 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2624003570 con 0x7f263c0b05d0 2023-10-06T05:06:51.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.221+0000 7f263959a700 1 -- 172.21.15.98:0/4143721397 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2725603259 0 0) 0x7f263c1577d0 con 0x7f263c0cce20 2023-10-06T05:06:51.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.221+0000 7f263959a700 1 -- 172.21.15.98:0/4143721397 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f263c0cadd0 con 0x7f263c0cce20 2023-10-06T05:06:51.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.221+0000 7f263959a700 1 -- 172.21.15.98:0/4143721397 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2996362359 0 0) 0x7f2624003570 con 0x7f263c0b05d0 2023-10-06T05:06:51.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.221+0000 7f263959a700 1 -- 172.21.15.98:0/4143721397 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f263c0c3f50 msgr2=0x7f263c162240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:06:51.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.221+0000 7f263959a700 1 --2- 172.21.15.98:0/4143721397 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f263c0c3f50 0x7f263c162240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:51.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.221+0000 7f263959a700 1 -- 172.21.15.98:0/4143721397 >> v1:172.21.15.174:6789/0 conn(0x7f263c0cce20 legacy=0x7f263c0c8e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:51.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.221+0000 7f263959a700 1 -- 172.21.15.98:0/4143721397 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f263c0c9dc0 con 0x7f263c0b05d0 2023-10-06T05:06:51.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.221+0000 7f263959a700 1 -- 172.21.15.98:0/4143721397 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1543991876 0 0) 0x7f262c001470 con 0x7f263c0b05d0 2023-10-06T05:06:51.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.221+0000 7f263959a700 1 -- 172.21.15.98:0/4143721397 >> v1:172.21.15.98:6789/0 conn(0x7f263c0b05d0 legacy=0x7f263c0cc710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:51.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.221+0000 7f263959a700 1 --2- 172.21.15.98:0/4143721397 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2624003c20 0x7f2624004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:51.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.221+0000 7f263959a700 1 --2- 172.21.15.98:0/4143721397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2624004810 0x7f2624004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:51.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.222+0000 7f263959a700 1 --2- 172.21.15.98:0/4143721397 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f263c0c3f50 0x7f26240051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:51.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.222+0000 7f263959a700 1 -- 172.21.15.98:0/4143721397 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f263c0c9ae0 con 0x7f2624004810 2023-10-06T05:06:51.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.222+0000 7f263959a700 1 -- 172.21.15.98:0/4143721397 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f262400bb70 con 0x7f2624003c20 2023-10-06T05:06:51.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.222+0000 7f263959a700 1 -- 172.21.15.98:0/4143721397 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f262400bd30 con 0x7f263c0c3f50 2023-10-06T05:06:51.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.222+0000 7f2639d9b700 1 --2- 172.21.15.98:0/4143721397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2624004810 0x7f2624004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:51.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.222+0000 7f263a59c700 1 --2- 172.21.15.98:0/4143721397 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2624003c20 0x7f2624004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:51.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.222+0000 7f263ad9d700 1 --2- 172.21.15.98:0/4143721397 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f263c0c3f50 0x7f26240051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:51.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.222+0000 7f2639d9b700 1 -- 172.21.15.98:0/4143721397 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f263c0c3f50 msgr2=0x7f26240051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:06:51.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.222+0000 7f2639d9b700 1 --2- 172.21.15.98:0/4143721397 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f263c0c3f50 0x7f26240051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:51.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.222+0000 7f2639d9b700 1 -- 172.21.15.98:0/4143721397 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2624003c20 msgr2=0x7f2624004010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:06:51.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.222+0000 7f2639d9b700 1 --2- 172.21.15.98:0/4143721397 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2624003c20 0x7f2624004010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:51.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.222+0000 7f2639d9b700 1 -- 172.21.15.98:0/4143721397 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2624003570 con 0x7f2624004810 2023-10-06T05:06:51.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.222+0000 7f2639d9b700 1 --2- 172.21.15.98:0/4143721397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2624004810 0x7f2624004c80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f263c0cd220 tx=0x7f2630007d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16784ac48d088300 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:06:51.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.222+0000 7f263959a700 1 -- 172.21.15.98:0/4143721397 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26300013f0 con 0x7f2624004810 2023-10-06T05:06:51.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.222+0000 7f263959a700 1 -- 172.21.15.98:0/4143721397 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f263000c5d0 con 0x7f2624004810 2023-10-06T05:06:51.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.222+0000 7f263959a700 1 -- 172.21.15.98:0/4143721397 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f263000c990 con 0x7f2624004810 2023-10-06T05:06:51.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.223+0000 7f263b59e700 1 -- 172.21.15.98:0/4143721397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2624004810 msgr2=0x7f2624004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:51.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.223+0000 7f263b59e700 1 --2- 172.21.15.98:0/4143721397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2624004810 0x7f2624004c80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f263c0cd220 tx=0x7f2630007d80 comp rx=0 tx=0).stop 2023-10-06T05:06:51.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.223+0000 7f263ad9d700 1 -- 172.21.15.98:0/4143721397 reap_dead start 2023-10-06T05:06:51.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.223+0000 7f263b59e700 1 -- 172.21.15.98:0/4143721397 shutdown_connections 2023-10-06T05:06:51.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.223+0000 7f263b59e700 1 -- 172.21.15.98:0/4143721397 >> 172.21.15.98:0/4143721397 conn(0x7f263c096a30 msgr2=0x7f263c096e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:06:51.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.223+0000 7f263b59e700 1 -- 172.21.15.98:0/4143721397 shutdown_connections 2023-10-06T05:06:51.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.223+0000 7f263b59e700 1 -- 172.21.15.98:0/4143721397 wait complete. 2023-10-06T05:06:51.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.224+0000 7f263b59e700 1 Processor -- start 2023-10-06T05:06:51.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.224+0000 7f263b59e700 1 -- start start 2023-10-06T05:06:51.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.225+0000 7f263b59e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2624004810 0x7f263c142a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:51.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.225+0000 7f263b59e700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f263c06a0b0 0x7f263c0b2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:51.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.225+0000 7f263b59e700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f263c0b30c0 0x7f263c0af050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:51.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.225+0000 7f263b59e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f263c0b3590 con 0x7f2624004810 2023-10-06T05:06:51.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.225+0000 7f263b59e700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f263c154f10 con 0x7f263c0b30c0 2023-10-06T05:06:51.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.225+0000 7f263b59e700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f263c1551f0 con 0x7f263c06a0b0 2023-10-06T05:06:51.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.225+0000 7f263a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2624004810 0x7f263c142a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:51.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.225+0000 7f263a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2624004810 0x7f263c142a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54016/0 (socket says 172.21.15.98:54016) 2023-10-06T05:06:51.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.225+0000 7f263a59c700 1 -- 172.21.15.98:0/2147101890 learned_addr learned my addr 172.21.15.98:0/2147101890 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:06:51.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.225+0000 7f263ad9d700 1 --2- 172.21.15.98:0/2147101890 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f263c0b30c0 0x7f263c0af050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:51.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.225+0000 7f2639d9b700 1 --2- 172.21.15.98:0/2147101890 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f263c06a0b0 0x7f263c0b2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:51.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.225+0000 7f263a59c700 1 -- 172.21.15.98:0/2147101890 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f263c06a0b0 msgr2=0x7f263c0b2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:51.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.225+0000 7f263a59c700 1 --2- 172.21.15.98:0/2147101890 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f263c06a0b0 0x7f263c0b2b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:51.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.225+0000 7f263a59c700 1 -- 172.21.15.98:0/2147101890 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f263c0b30c0 msgr2=0x7f263c0af050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:51.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.225+0000 7f263a59c700 1 --2- 172.21.15.98:0/2147101890 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f263c0b30c0 0x7f263c0af050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:51.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.225+0000 7f263a59c700 1 -- 172.21.15.98:0/2147101890 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f263c1554d0 con 0x7f2624004810 2023-10-06T05:06:51.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.225+0000 7f263a59c700 1 --2- 172.21.15.98:0/2147101890 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2624004810 0x7f263c142a60 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f262c007390 tx=0x7f262c001df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:06:51.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.225+0000 7f262a7fc700 1 -- 172.21.15.98:0/2147101890 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f262c018030 con 0x7f2624004810 2023-10-06T05:06:51.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.225+0000 7f262a7fc700 1 -- 172.21.15.98:0/2147101890 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f262c00f640 con 0x7f2624004810 2023-10-06T05:06:51.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.225+0000 7f263b59e700 1 -- 172.21.15.98:0/2147101890 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f263c1557b0 con 0x7f2624004810 2023-10-06T05:06:51.236 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.225+0000 7f262a7fc700 1 -- 172.21.15.98:0/2147101890 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f262c00f9e0 con 0x7f2624004810 2023-10-06T05:06:51.236 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.225+0000 7f263b59e700 1 -- 172.21.15.98:0/2147101890 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f263c0af590 con 0x7f2624004810 2023-10-06T05:06:51.236 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.226+0000 7f263b59e700 1 -- 172.21.15.98:0/2147101890 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f263c0ceb20 con 0x7f2624004810 2023-10-06T05:06:51.236 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.227+0000 7f262a7fc700 1 -- 172.21.15.98:0/2147101890 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f262c01b070 con 0x7f2624004810 2023-10-06T05:06:51.236 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.227+0000 7f262a7fc700 1 --2- 172.21.15.98:0/2147101890 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f261c066f30 0x7f261c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:51.236 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.227+0000 7f2639d9b700 1 --2- 172.21.15.98:0/2147101890 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f261c066f30 0x7f261c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:51.236 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.227+0000 7f262a7fc700 1 -- 172.21.15.98:0/2147101890 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4853+0+0 (secure 0 0 0) 0x7f262c019070 con 0x7f2624004810 2023-10-06T05:06:51.236 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.227+0000 7f2639d9b700 1 --2- 172.21.15.98:0/2147101890 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f261c066f30 0x7f261c0693e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f263c16d0f0 tx=0x7f2630015000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:06:51.236 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.231+0000 7f262a7fc700 1 -- 172.21.15.98:0/2147101890 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f262c0514b0 con 0x7f2624004810 2023-10-06T05:06:51.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.414+0000 7f263b59e700 1 -- 172.21.15.98:0/2147101890 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f263c06c7d0 con 0x7f2624004810 2023-10-06T05:06:51.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.576+0000 7f262a7fc700 1 -- 172.21.15.98:0/2147101890 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v19) v1 ==== 109+0+0 (secure 0 0 0) 0x7f262c054ad0 con 0x7f2624004810 2023-10-06T05:06:51.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.578+0000 7f263b59e700 1 -- 172.21.15.98:0/2147101890 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f261c066f30 msgr2=0x7f261c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:51.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.578+0000 7f263b59e700 1 --2- 172.21.15.98:0/2147101890 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f261c066f30 0x7f261c0693e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f263c16d0f0 tx=0x7f2630015000 comp rx=0 tx=0).stop 2023-10-06T05:06:51.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.578+0000 7f263b59e700 1 -- 172.21.15.98:0/2147101890 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2624004810 msgr2=0x7f263c142a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:51.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.578+0000 7f263b59e700 1 --2- 172.21.15.98:0/2147101890 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2624004810 0x7f263c142a60 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f262c007390 tx=0x7f262c001df0 comp rx=0 tx=0).stop 2023-10-06T05:06:51.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.579+0000 7f263b59e700 1 -- 172.21.15.98:0/2147101890 shutdown_connections 2023-10-06T05:06:51.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.579+0000 7f263b59e700 1 --2- 172.21.15.98:0/2147101890 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f263c0b30c0 0x7f263c0af050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:51.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.579+0000 7f263b59e700 1 --2- 172.21.15.98:0/2147101890 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f261c066f30 0x7f261c0693e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:51.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.579+0000 7f263b59e700 1 --2- 172.21.15.98:0/2147101890 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2624004810 0x7f263c142a60 secure :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f262c007390 tx=0x7f262c001df0 comp rx=0 tx=0).stop 2023-10-06T05:06:51.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.579+0000 7f263b59e700 1 --2- 172.21.15.98:0/2147101890 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f263c06a0b0 0x7f263c0b2b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:51.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.579+0000 7f263b59e700 1 -- 172.21.15.98:0/2147101890 >> 172.21.15.98:0/2147101890 conn(0x7f263c096a30 msgr2=0x7f263c06c0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:06:51.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.579+0000 7f263b59e700 1 -- 172.21.15.98:0/2147101890 shutdown_connections 2023-10-06T05:06:51.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.579+0000 7f263b59e700 1 -- 172.21.15.98:0/2147101890 wait complete. 2023-10-06T05:06:51.582 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'rbd' created 2023-10-06T05:06:51.593 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-10-06T05:06:51.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.706+0000 7fc9be632700 1 Processor -- start 2023-10-06T05:06:51.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.706+0000 7fc9be632700 1 -- start start 2023-10-06T05:06:51.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.707+0000 7fc9be632700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fc9b8157550 0x7fc9b80ca350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:51.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.707+0000 7fc9be632700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9b80c99a0 con 0x7fc9b80ca890 2023-10-06T05:06:51.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.707+0000 7fc9be632700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9b80c81b0 con 0x7fc9b8157180 2023-10-06T05:06:51.713 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.707+0000 7fc9be632700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9b80c8490 con 0x7fc9b8157550 2023-10-06T05:06:51.713 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.707+0000 7fc9bcbcf700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fc9b80ca890 0x7fc9b80675b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:49318/0 (socket says 172.21.15.98:49318) 2023-10-06T05:06:51.713 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.707+0000 7fc9bcbcf700 1 -- 172.21.15.98:0/3620097346 learned_addr learned my addr 172.21.15.98:0/3620097346 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:06:51.713 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.708+0000 7fc9b77fe700 1 --2- 172.21.15.98:0/3620097346 >> v2:172.21.15.174:3300/0 conn(0x7fc9b8157550 0x7fc9b80ca350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:51.713 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.708+0000 7fc9b6ffd700 1 -- 172.21.15.98:0/3620097346 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2590440600 0 0) 0x7fc9b80c99a0 con 0x7fc9b80ca890 2023-10-06T05:06:51.713 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.708+0000 7fc9b6ffd700 1 -- 172.21.15.98:0/3620097346 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9a0003570 con 0x7fc9b80ca890 2023-10-06T05:06:51.713 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.708+0000 7fc9b6ffd700 1 -- 172.21.15.98:0/3620097346 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1597277698 0 0) 0x7fc9a0003570 con 0x7fc9b80ca890 2023-10-06T05:06:51.713 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.708+0000 7fc9b6ffd700 1 -- 172.21.15.98:0/3620097346 >> v2:172.21.15.174:3300/0 conn(0x7fc9b8157550 msgr2=0x7fc9b80ca350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:06:51.713 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.708+0000 7fc9b6ffd700 1 --2- 172.21.15.98:0/3620097346 >> v2:172.21.15.174:3300/0 conn(0x7fc9b8157550 0x7fc9b80ca350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:51.713 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.708+0000 7fc9b6ffd700 1 -- 172.21.15.98:0/3620097346 >> v1:172.21.15.174:6789/0 conn(0x7fc9b8157180 legacy=0x7fc9b80c4210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:51.713 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.708+0000 7fc9b6ffd700 1 -- 172.21.15.98:0/3620097346 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9b80c4920 con 0x7fc9b80ca890 2023-10-06T05:06:51.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.708+0000 7fc9b6ffd700 1 -- 172.21.15.98:0/3620097346 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1543991876 0 0) 0x7fc9b00017f0 con 0x7fc9b80ca890 2023-10-06T05:06:51.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.708+0000 7fc9b6ffd700 1 -- 172.21.15.98:0/3620097346 >> v1:172.21.15.98:6789/0 conn(0x7fc9b80ca890 legacy=0x7fc9b80675b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:51.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.708+0000 7fc9b6ffd700 1 --2- 172.21.15.98:0/3620097346 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc9a0003a60 0x7fc9a0003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:51.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.708+0000 7fc9b6ffd700 1 --2- 172.21.15.98:0/3620097346 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9a0004810 0x7fc9a0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:51.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.708+0000 7fc9b6ffd700 1 --2- 172.21.15.98:0/3620097346 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc9a00051c0 0x7fc9a000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:51.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.708+0000 7fc9b6ffd700 1 -- 172.21.15.98:0/3620097346 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9b80c8490 con 0x7fc9a0004810 2023-10-06T05:06:51.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.708+0000 7fc9b6ffd700 1 -- 172.21.15.98:0/3620097346 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a000c0d0 con 0x7fc9a0003a60 2023-10-06T05:06:51.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.708+0000 7fc9b6ffd700 1 -- 172.21.15.98:0/3620097346 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a000c290 con 0x7fc9a00051c0 2023-10-06T05:06:51.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.708+0000 7fc9b77fe700 1 --2- 172.21.15.98:0/3620097346 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9a0004810 0x7fc9a0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:51.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.708+0000 7fc9b7fff700 1 --2- 172.21.15.98:0/3620097346 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc9a0003a60 0x7fc9a0003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:51.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.708+0000 7fc9bcbcf700 1 --2- 172.21.15.98:0/3620097346 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc9a00051c0 0x7fc9a000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:51.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.708+0000 7fc9b77fe700 1 -- 172.21.15.98:0/3620097346 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc9a00051c0 msgr2=0x7fc9a000ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:06:51.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.708+0000 7fc9b77fe700 1 --2- 172.21.15.98:0/3620097346 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc9a00051c0 0x7fc9a000ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:51.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.708+0000 7fc9b77fe700 1 -- 172.21.15.98:0/3620097346 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc9a0003a60 msgr2=0x7fc9a0003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:06:51.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.708+0000 7fc9b77fe700 1 --2- 172.21.15.98:0/3620097346 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc9a0003a60 0x7fc9a0003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:51.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.708+0000 7fc9b77fe700 1 -- 172.21.15.98:0/3620097346 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9a0003570 con 0x7fc9a0004810 2023-10-06T05:06:51.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.708+0000 7fc9bcbcf700 1 -- 172.21.15.98:0/3620097346 reap_dead start 2023-10-06T05:06:51.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.708+0000 7fc9b77fe700 1 --2- 172.21.15.98:0/3620097346 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9a0004810 0x7fc9a0004c80 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc9ac009c00 tx=0x7fc9ac0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82ce41685b1971c7 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:06:51.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.709+0000 7fc9b6ffd700 1 -- 172.21.15.98:0/3620097346 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9ac00e710 con 0x7fc9a0004810 2023-10-06T05:06:51.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.709+0000 7fc9b6ffd700 1 -- 172.21.15.98:0/3620097346 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9ac00e870 con 0x7fc9a0004810 2023-10-06T05:06:51.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.709+0000 7fc9b6ffd700 1 -- 172.21.15.98:0/3620097346 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9ac00ec30 con 0x7fc9a0004810 2023-10-06T05:06:51.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.709+0000 7fc9be632700 1 -- 172.21.15.98:0/3620097346 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9a0004810 msgr2=0x7fc9a0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:51.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.709+0000 7fc9be632700 1 --2- 172.21.15.98:0/3620097346 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9a0004810 0x7fc9a0004c80 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc9ac009c00 tx=0x7fc9ac0091a0 comp rx=0 tx=0).stop 2023-10-06T05:06:51.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.709+0000 7fc9be632700 1 -- 172.21.15.98:0/3620097346 shutdown_connections 2023-10-06T05:06:51.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.709+0000 7fc9be632700 1 --2- 172.21.15.98:0/3620097346 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9a0004810 0x7fc9a0004c80 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:51.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.709+0000 7fc9be632700 1 -- 172.21.15.98:0/3620097346 >> 172.21.15.98:0/3620097346 conn(0x7fc9b80cdbc0 msgr2=0x7fc9b80c9bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:06:51.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.709+0000 7fc9be632700 1 -- 172.21.15.98:0/3620097346 shutdown_connections 2023-10-06T05:06:51.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.709+0000 7fc9be632700 1 -- 172.21.15.98:0/3620097346 wait complete. 2023-10-06T05:06:51.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.710+0000 7fc9be632700 1 Processor -- start 2023-10-06T05:06:51.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.711+0000 7fc9be632700 1 -- start start 2023-10-06T05:06:51.719 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.711+0000 7fc9be632700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc9a0004810 0x7fc9b815a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:51.719 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.711+0000 7fc9be632700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9b80b0970 0x7fc9b8142be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:51.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.711+0000 7fc9be632700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc9b816b5f0 0x7fc9b80c6680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:51.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.711+0000 7fc9be632700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9b816b160 con 0x7fc9b80b0970 2023-10-06T05:06:51.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.711+0000 7fc9be632700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9b8143120 con 0x7fc9a0004810 2023-10-06T05:06:51.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.711+0000 7fc9be632700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9b80b0ec0 con 0x7fc9b816b5f0 2023-10-06T05:06:51.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.711+0000 7fc9b77fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9b80b0970 0x7fc9b8142be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:51.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.711+0000 7fc9b77fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9b80b0970 0x7fc9b8142be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54034/0 (socket says 172.21.15.98:54034) 2023-10-06T05:06:51.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.711+0000 7fc9b77fe700 1 -- 172.21.15.98:0/1995888064 learned_addr learned my addr 172.21.15.98:0/1995888064 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:06:51.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.711+0000 7fc9b7fff700 1 --2- 172.21.15.98:0/1995888064 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc9a0004810 0x7fc9b815a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:51.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.711+0000 7fc9bcbcf700 1 --2- 172.21.15.98:0/1995888064 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc9b816b5f0 0x7fc9b80c6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:51.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.712+0000 7fc9b77fe700 1 -- 172.21.15.98:0/1995888064 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc9b816b5f0 msgr2=0x7fc9b80c6680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:51.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.712+0000 7fc9b77fe700 1 --2- 172.21.15.98:0/1995888064 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc9b816b5f0 0x7fc9b80c6680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:51.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.712+0000 7fc9b77fe700 1 -- 172.21.15.98:0/1995888064 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc9a0004810 msgr2=0x7fc9b815a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:51.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.712+0000 7fc9b77fe700 1 --2- 172.21.15.98:0/1995888064 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc9a0004810 0x7fc9b815a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:51.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.712+0000 7fc9b77fe700 1 -- 172.21.15.98:0/1995888064 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9b80b0100 con 0x7fc9b80b0970 2023-10-06T05:06:51.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.712+0000 7fc9bcbcf700 1 --2- 172.21.15.98:0/1995888064 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc9b816b5f0 0x7fc9b80c6680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:06:51.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.712+0000 7fc9b7fff700 1 --2- 172.21.15.98:0/1995888064 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc9a0004810 0x7fc9b815a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:06:51.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.712+0000 7fc9b77fe700 1 --2- 172.21.15.98:0/1995888064 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9b80b0970 0x7fc9b8142be0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc9ac00e4b0 tx=0x7fc9ac01a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:06:51.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.712+0000 7fc99ffff700 1 -- 172.21.15.98:0/1995888064 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9ac022070 con 0x7fc9b80b0970 2023-10-06T05:06:51.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.712+0000 7fc99ffff700 1 -- 172.21.15.98:0/1995888064 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9ac013cd0 con 0x7fc9b80b0970 2023-10-06T05:06:51.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.712+0000 7fc9be632700 1 -- 172.21.15.98:0/1995888064 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9b80b03e0 con 0x7fc9b80b0970 2023-10-06T05:06:51.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.712+0000 7fc99ffff700 1 -- 172.21.15.98:0/1995888064 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9ac01c6a0 con 0x7fc9b80b0970 2023-10-06T05:06:51.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.712+0000 7fc9be632700 1 -- 172.21.15.98:0/1995888064 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9b8153540 con 0x7fc9b80b0970 2023-10-06T05:06:51.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.713+0000 7fc9be632700 1 -- 172.21.15.98:0/1995888064 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9b816b2d0 con 0x7fc9b80b0970 2023-10-06T05:06:51.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.713+0000 7fc99ffff700 1 -- 172.21.15.98:0/1995888064 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fc9ac01cad0 con 0x7fc9b80b0970 2023-10-06T05:06:51.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.714+0000 7fc99ffff700 1 --2- 172.21.15.98:0/1995888064 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fc9a4066f30 0x7fc9a40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:51.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.714+0000 7fc99ffff700 1 -- 172.21.15.98:0/1995888064 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5195+0+0 (secure 0 0 0) 0x7fc9ac085940 con 0x7fc9b80b0970 2023-10-06T05:06:51.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.714+0000 7fc9b7fff700 1 --2- 172.21.15.98:0/1995888064 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fc9a4066f30 0x7fc9a40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:51.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.714+0000 7fc9b7fff700 1 --2- 172.21.15.98:0/1995888064 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fc9a4066f30 0x7fc9a40693e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc9a8007bc0 tx=0x7fc9a8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:06:51.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.718+0000 7fc99ffff700 1 -- 172.21.15.98:0/1995888064 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc9ac052df0 con 0x7fc9b80b0970 2023-10-06T05:06:51.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:51.903+0000 7fc9be632700 1 -- 172.21.15.98:0/1995888064 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fc9b8064a90 con 0x7fc9b80b0970 2023-10-06T05:06:52.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.581+0000 7fc99ffff700 1 -- 172.21.15.98:0/1995888064 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v20) v1 ==== 173+0+0 (secure 0 0 0) 0x7fc9ac017070 con 0x7fc9b80b0970 2023-10-06T05:06:52.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.584+0000 7fc9be632700 1 -- 172.21.15.98:0/1995888064 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fc9a4066f30 msgr2=0x7fc9a40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:52.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.584+0000 7fc9be632700 1 --2- 172.21.15.98:0/1995888064 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fc9a4066f30 0x7fc9a40693e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc9a8007bc0 tx=0x7fc9a8009040 comp rx=0 tx=0).stop 2023-10-06T05:06:52.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.584+0000 7fc9be632700 1 -- 172.21.15.98:0/1995888064 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9b80b0970 msgr2=0x7fc9b8142be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:52.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.584+0000 7fc9be632700 1 --2- 172.21.15.98:0/1995888064 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9b80b0970 0x7fc9b8142be0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc9ac00e4b0 tx=0x7fc9ac01a000 comp rx=0 tx=0).stop 2023-10-06T05:06:52.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.584+0000 7fc9be632700 1 -- 172.21.15.98:0/1995888064 shutdown_connections 2023-10-06T05:06:52.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.584+0000 7fc9be632700 1 --2- 172.21.15.98:0/1995888064 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc9b816b5f0 0x7fc9b80c6680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:52.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.584+0000 7fc9be632700 1 --2- 172.21.15.98:0/1995888064 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc9a0004810 0x7fc9b815a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:52.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.584+0000 7fc9be632700 1 --2- 172.21.15.98:0/1995888064 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fc9a4066f30 0x7fc9a40693e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:52.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.584+0000 7fc9be632700 1 --2- 172.21.15.98:0/1995888064 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9b80b0970 0x7fc9b8142be0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:52.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.584+0000 7fc9be632700 1 -- 172.21.15.98:0/1995888064 >> 172.21.15.98:0/1995888064 conn(0x7fc9b80cdbc0 msgr2=0x7fc9b8156350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:06:52.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.584+0000 7fc9be632700 1 -- 172.21.15.98:0/1995888064 shutdown_connections 2023-10-06T05:06:52.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.584+0000 7fc9be632700 1 -- 172.21.15.98:0/1995888064 wait complete. 2023-10-06T05:06:52.588 INFO:teuthology.orchestra.run.smithi098.stderr:enabled application 'rbd' on pool 'rbd' 2023-10-06T05:06:52.600 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-10-06T05:06:52.601 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-06T05:06:52.601 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-06T05:06:52.603 INFO:tasks.ceph.mds.a:Started 2023-10-06T05:06:52.603 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-06T05:06:52.603 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-06T05:06:52.605 INFO:tasks.ceph.mds.c:Started 2023-10-06T05:06:52.605 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-06T05:06:52.605 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-06T05:06:52.608 INFO:tasks.ceph.mds.b:Started 2023-10-06T05:06:52.608 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-06T05:06:52.608 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-06T05:06:52.610 INFO:tasks.ceph.mds.d:Started 2023-10-06T05:06:52.610 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2023-10-06T05:06:52.610 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T05:06:52.681 INFO:tasks.ceph.mds.a.smithi098.stdout:starting mds.a at 2023-10-06T05:06:52.682 INFO:tasks.ceph.mds.c.smithi098.stdout:starting mds.c at 2023-10-06T05:06:52.691 INFO:tasks.ceph.mds.b.smithi174.stdout:starting mds.b at 2023-10-06T05:06:52.691 INFO:tasks.ceph.mds.d.smithi174.stdout:starting mds.d at 2023-10-06T05:06:52.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.733+0000 7fc3935f2700 1 Processor -- start 2023-10-06T05:06:52.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.733+0000 7fc3935f2700 1 -- start start 2023-10-06T05:06:52.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.734+0000 7fc3935f2700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc38c156350 0x7fc38c0cc1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:52.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.734+0000 7fc3935f2700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc38c0ccb80 con 0x7fc38c06c330 2023-10-06T05:06:52.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.734+0000 7fc3935f2700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc38c06c9a0 con 0x7fc38c0cc710 2023-10-06T05:06:52.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.734+0000 7fc3935f2700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc38c06c790 con 0x7fc38c156350 2023-10-06T05:06:52.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.734+0000 7fc391b8f700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fc38c06c330 0x7fc38c0c8880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:49340/0 (socket says 172.21.15.98:49340) 2023-10-06T05:06:52.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.734+0000 7fc391b8f700 1 -- 172.21.15.98:0/1576055679 learned_addr learned my addr 172.21.15.98:0/1576055679 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:06:52.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.735+0000 7fc383fff700 1 -- 172.21.15.98:0/1576055679 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3038244675 0 0) 0x7fc38c0ccb80 con 0x7fc38c06c330 2023-10-06T05:06:52.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.735+0000 7fc39138e700 1 --2- 172.21.15.98:0/1576055679 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc38c156350 0x7fc38c0cc1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:52.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.735+0000 7fc383fff700 1 -- 172.21.15.98:0/1576055679 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc378003570 con 0x7fc38c06c330 2023-10-06T05:06:52.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.735+0000 7fc383fff700 1 -- 172.21.15.98:0/1576055679 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4003052652 0 0) 0x7fc378003570 con 0x7fc38c06c330 2023-10-06T05:06:52.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.735+0000 7fc383fff700 1 -- 172.21.15.98:0/1576055679 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc38c156350 msgr2=0x7fc38c0cc1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:06:52.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.735+0000 7fc383fff700 1 --2- 172.21.15.98:0/1576055679 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc38c156350 0x7fc38c0cc1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:52.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.735+0000 7fc383fff700 1 -- 172.21.15.98:0/1576055679 >> v1:172.21.15.174:6789/0 conn(0x7fc38c0cc710 legacy=0x7fc38c06bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:52.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.735+0000 7fc383fff700 1 -- 172.21.15.98:0/1576055679 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc38c156b20 con 0x7fc38c06c330 2023-10-06T05:06:52.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.735+0000 7fc383fff700 1 -- 172.21.15.98:0/1576055679 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1543991876 0 0) 0x7fc3840017f0 con 0x7fc38c06c330 2023-10-06T05:06:52.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.735+0000 7fc383fff700 1 -- 172.21.15.98:0/1576055679 >> v1:172.21.15.98:6789/0 conn(0x7fc38c06c330 legacy=0x7fc38c0c8880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:52.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.735+0000 7fc383fff700 1 --2- 172.21.15.98:0/1576055679 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc378003a90 0x7fc378003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:52.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.735+0000 7fc383fff700 1 --2- 172.21.15.98:0/1576055679 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc38c156350 0x7fc378004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:52.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.735+0000 7fc383fff700 1 --2- 172.21.15.98:0/1576055679 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc378004d50 0x7fc3780051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:52.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.735+0000 7fc383fff700 1 -- 172.21.15.98:0/1576055679 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc38c06c790 con 0x7fc378004d50 2023-10-06T05:06:52.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.735+0000 7fc383fff700 1 -- 172.21.15.98:0/1576055679 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc37800bb70 con 0x7fc378003a90 2023-10-06T05:06:52.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.735+0000 7fc383fff700 1 -- 172.21.15.98:0/1576055679 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc37800bd30 con 0x7fc38c156350 2023-10-06T05:06:52.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.735+0000 7fc391b8f700 1 --2- 172.21.15.98:0/1576055679 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc378004d50 0x7fc3780051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:52.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.735+0000 7fc39138e700 1 --2- 172.21.15.98:0/1576055679 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc378003a90 0x7fc378003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:52.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.735+0000 7fc390b8d700 1 --2- 172.21.15.98:0/1576055679 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc38c156350 0x7fc378004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:52.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.735+0000 7fc391b8f700 1 -- 172.21.15.98:0/1576055679 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc38c156350 msgr2=0x7fc378004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:06:52.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.735+0000 7fc391b8f700 1 --2- 172.21.15.98:0/1576055679 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc38c156350 0x7fc378004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:52.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.735+0000 7fc391b8f700 1 -- 172.21.15.98:0/1576055679 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc378003a90 msgr2=0x7fc378003ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:06:52.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.735+0000 7fc391b8f700 1 --2- 172.21.15.98:0/1576055679 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc378003a90 0x7fc378003ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:52.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.735+0000 7fc391b8f700 1 -- 172.21.15.98:0/1576055679 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc378003570 con 0x7fc378004d50 2023-10-06T05:06:52.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.736+0000 7fc391b8f700 1 --2- 172.21.15.98:0/1576055679 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc378004d50 0x7fc3780051c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc38400a930 tx=0x7fc3840021d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4ce20121e013171 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:06:52.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.736+0000 7fc383fff700 1 -- 172.21.15.98:0/1576055679 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3840070d0 con 0x7fc378004d50 2023-10-06T05:06:52.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.736+0000 7fc383fff700 1 -- 172.21.15.98:0/1576055679 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc384007230 con 0x7fc378004d50 2023-10-06T05:06:52.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.736+0000 7fc383fff700 1 -- 172.21.15.98:0/1576055679 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3840075d0 con 0x7fc378004d50 2023-10-06T05:06:52.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.736+0000 7fc3935f2700 1 -- 172.21.15.98:0/1576055679 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc378004d50 msgr2=0x7fc3780051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:52.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.736+0000 7fc3935f2700 1 --2- 172.21.15.98:0/1576055679 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc378004d50 0x7fc3780051c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc38400a930 tx=0x7fc3840021d0 comp rx=0 tx=0).stop 2023-10-06T05:06:52.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.736+0000 7fc391b8f700 1 -- 172.21.15.98:0/1576055679 reap_dead start 2023-10-06T05:06:52.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.736+0000 7fc3935f2700 1 -- 172.21.15.98:0/1576055679 shutdown_connections 2023-10-06T05:06:52.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.736+0000 7fc3935f2700 1 -- 172.21.15.98:0/1576055679 >> 172.21.15.98:0/1576055679 conn(0x7fc38c0cdeb0 msgr2=0x7fc38c0c9e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:06:52.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.736+0000 7fc3935f2700 1 -- 172.21.15.98:0/1576055679 shutdown_connections 2023-10-06T05:06:52.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.736+0000 7fc3935f2700 1 -- 172.21.15.98:0/1576055679 wait complete. 2023-10-06T05:06:52.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.737+0000 7fc3935f2700 1 Processor -- start 2023-10-06T05:06:52.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.737+0000 7fc3935f2700 1 -- start start 2023-10-06T05:06:52.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.738+0000 7fc3935f2700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc378004d50 0x7fc38c151ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:52.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.738+0000 7fc3935f2700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc38c0c23a0 0x7fc38c0c2810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:52.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.738+0000 7fc3935f2700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc38c0a62c0 0x7fc38c0a6760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:52.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.738+0000 7fc3935f2700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc38c0c2d50 con 0x7fc38c0c23a0 2023-10-06T05:06:52.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.738+0000 7fc3935f2700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc38c16ac00 con 0x7fc38c0a62c0 2023-10-06T05:06:52.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.738+0000 7fc3935f2700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc38c169ff0 con 0x7fc378004d50 2023-10-06T05:06:52.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.738+0000 7fc390b8d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc38c0c23a0 0x7fc38c0c2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:52.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.738+0000 7fc390b8d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc38c0c23a0 0x7fc38c0c2810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54084/0 (socket says 172.21.15.98:54084) 2023-10-06T05:06:52.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.738+0000 7fc390b8d700 1 -- 172.21.15.98:0/173452643 learned_addr learned my addr 172.21.15.98:0/173452643 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:06:52.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.738+0000 7fc39138e700 1 --2- 172.21.15.98:0/173452643 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc378004d50 0x7fc38c151ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:52.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.739+0000 7fc391b8f700 1 --2- 172.21.15.98:0/173452643 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc38c0a62c0 0x7fc38c0a6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:52.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.739+0000 7fc390b8d700 1 -- 172.21.15.98:0/173452643 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc378004d50 msgr2=0x7fc38c151ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:52.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.739+0000 7fc390b8d700 1 --2- 172.21.15.98:0/173452643 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc378004d50 0x7fc38c151ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:52.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.739+0000 7fc390b8d700 1 -- 172.21.15.98:0/173452643 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc38c0a62c0 msgr2=0x7fc38c0a6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:52.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.739+0000 7fc390b8d700 1 --2- 172.21.15.98:0/173452643 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc38c0a62c0 0x7fc38c0a6760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:52.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.739+0000 7fc390b8d700 1 -- 172.21.15.98:0/173452643 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc38c16a2d0 con 0x7fc38c0c23a0 2023-10-06T05:06:52.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.739+0000 7fc390b8d700 1 --2- 172.21.15.98:0/173452643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc38c0c23a0 0x7fc38c0c2810 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc37c002920 tx=0x7fc37c007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:06:52.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.739+0000 7fc3817fa700 1 -- 172.21.15.98:0/173452643 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc37c00f670 con 0x7fc38c0c23a0 2023-10-06T05:06:52.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.739+0000 7fc3817fa700 1 -- 172.21.15.98:0/173452643 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc37c00f7d0 con 0x7fc38c0c23a0 2023-10-06T05:06:52.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.739+0000 7fc3817fa700 1 -- 172.21.15.98:0/173452643 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc37c00fb90 con 0x7fc38c0c23a0 2023-10-06T05:06:52.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.739+0000 7fc3935f2700 1 -- 172.21.15.98:0/173452643 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc38c16a5b0 con 0x7fc38c0c23a0 2023-10-06T05:06:52.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.739+0000 7fc3935f2700 1 -- 172.21.15.98:0/173452643 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc38c1520e0 con 0x7fc38c0c23a0 2023-10-06T05:06:52.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.740+0000 7fc3935f2700 1 -- 172.21.15.98:0/173452643 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc38c16aa50 con 0x7fc38c0c23a0 2023-10-06T05:06:52.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.740+0000 7fc3817fa700 1 -- 172.21.15.98:0/173452643 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fc37c01c400 con 0x7fc38c0c23a0 2023-10-06T05:06:52.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.741+0000 7fc3817fa700 1 --2- 172.21.15.98:0/173452643 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fc370066f30 0x7fc3700693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:52.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.741+0000 7fc39138e700 1 --2- 172.21.15.98:0/173452643 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fc370066f30 0x7fc3700693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:52.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.741+0000 7fc3817fa700 1 -- 172.21.15.98:0/173452643 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5206+0+0 (secure 0 0 0) 0x7fc37c084770 con 0x7fc38c0c23a0 2023-10-06T05:06:52.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.741+0000 7fc39138e700 1 --2- 172.21.15.98:0/173452643 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fc370066f30 0x7fc3700693e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc388006f00 tx=0x7fc3880091a0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:06:52.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.745+0000 7fc3817fa700 1 -- 172.21.15.98:0/173452643 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc37c051c80 con 0x7fc38c0c23a0 2023-10-06T05:06:52.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.932+0000 7fc3935f2700 1 -- 172.21.15.98:0/173452643 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc38c16ad70 con 0x7fc38c0c23a0 2023-10-06T05:06:52.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.933+0000 7fc3817fa700 1 -- 172.21.15.98:0/173452643 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+13660 (secure 0 0 0) 0x7fc37c0552a0 con 0x7fc38c0c23a0 2023-10-06T05:06:52.937 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:06:52.937 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":20,"fsid":"f50c5df5-cad5-47a4-9bb1-8c791d4311ec","created":"2023-10-06T05:06:32.971175+0000","modified":"2023-10-06T05:06:52.576879+0000","last_up_change":"2023-10-06T05:06:44.016489+0000","last_in_change":"2023-10-06T05:06:35.544511+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-06T05:06:44.559671+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-06T05:06:51.432161+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d9495d95-da75-4c3b-8927-e9bae6a8b093","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":90627},{"type":"v1","addr":"172.21.15.98:6801","nonce":90627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":90627},{"type":"v1","addr":"172.21.15.98:6803","nonce":90627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":90627},{"type":"v1","addr":"172.21.15.98:6807","nonce":90627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":90627},{"type":"v1","addr":"172.21.15.98:6805","nonce":90627}]},"public_addr":"172.21.15.98:6801/90627","cluster_addr":"172.21.15.98:6803/90627","heartbeat_back_addr":"172.21.15.98:6807/90627","heartbeat_front_addr":"172.21.15.98:6805/90627","state":["exists","up"]},{"osd":1,"uuid":"f25ba5be-d65e-4707-a134-154092deb754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":90628},{"type":"v1","addr":"172.21.15.98:6809","nonce":90628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":90628},{"type":"v1","addr":"172.21.15.98:6811","nonce":90628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":90628},{"type":"v1","addr":"172.21.15.98:6815","nonce":90628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":90628},{"type":"v1","addr":"172.21.15.98:6813","nonce":90628}]},"public_addr":"172.21.15.98:6809/90628","cluster_addr":"172.21.15.98:6811/90628","heartbeat_back_addr":"172.21.15.98:6815/90628","heartbeat_front_addr":"172.21.15.98:6813/90628","state":["exists","up"]},{"osd":2,"uuid":"5ab9e028-945c-43e2-8c95-3b0b4cf38e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":90662},{"type":"v1","addr":"172.21.15.98:6818","nonce":90662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":90662},{"type":"v1","addr":"172.21.15.98:6822","nonce":90662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":90662},{"type":"v1","addr":"172.21.15.98:6830","nonce":90662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":90662},{"type":"v1","addr":"172.21.15.98:6826","nonce":90662}]},"public_addr":"172.21.15.98:6818/90662","cluster_addr":"172.21.15.98:6822/90662","heartbeat_back_addr":"172.21.15.98:6830/90662","heartbeat_front_addr":"172.21.15.98:6826/90662","state":["exists","up"]},{"osd":3,"uuid":"3bb803ff-1738-46bf-be62-2267bf378822","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6817","nonce":90663},{"type":"v1","addr":"172.21.15.98:6819","nonce":90663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6821","nonce":90663},{"type":"v1","addr":"172.21.15.98:6823","nonce":90663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6829","nonce":90663},{"type":"v1","addr":"172.21.15.98:6831","nonce":90663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6825","nonce":90663},{"type":"v1","addr":"172.21.15.98:6827","nonce":90663}]},"public_addr":"172.21.15.98:6819/90663","cluster_addr":"172.21.15.98:6823/90663","heartbeat_back_addr":"172.21.15.98:6831/90663","heartbeat_front_addr":"172.21.15.98:6827/90663","state":["exists","up"]},{"osd":4,"uuid":"7d57a4c9-6462-43f6-b64a-7f9e9b4353d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":90847},{"type":"v1","addr":"172.21.15.174:6801","nonce":90847}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":90847},{"type":"v1","addr":"172.21.15.174:6803","nonce":90847}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":90847},{"type":"v1","addr":"172.21.15.174:6807","nonce":90847}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":90847},{"type":"v1","addr":"172.21.15.174:6805","nonce":90847}]},"public_addr":"172.21.15.174:6801/90847","cluster_addr":"172.21.15.174:6803/90847","heartbeat_back_addr":"172.21.15.174:6807/90847","heartbeat_front_addr":"172.21.15.174:6805/90847","state":["exists","up"]},{"osd":5,"uuid":"83287ea4-e44d-4af9-bc77-36d47c60d4a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":90848},{"type":"v1","addr":"172.21.15.174:6809","nonce":90848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":90848},{"type":"v1","addr":"172.21.15.174:6811","nonce":90848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":90848},{"type":"v1","addr":"172.21.15.174:6815","nonce":90848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":90848},{"type":"v1","addr":"172.21.15.174:6813","nonce":90848}]},"public_addr":"172.21.15.174:6809/90848","cluster_addr":"172.21.15.174:6811/90848","heartbeat_back_addr":"172.21.15.174:6815/90848","heartbeat_front_addr":"172.21.15.174:6813/90848","state":["exists","up"]},{"osd":6,"uuid":"496a6272-892e-43c4-a145-206623c22f99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":90877},{"type":"v1","addr":"172.21.15.174:6817","nonce":90877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":90877},{"type":"v1","addr":"172.21.15.174:6819","nonce":90877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":90877},{"type":"v1","addr":"172.21.15.174:6823","nonce":90877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":90877},{"type":"v1","addr":"172.21.15.174:6821","nonce":90877}]},"public_addr":"172.21.15.174:6817/90877","cluster_addr":"172.21.15.174:6819/90877","heartbeat_back_addr":"172.21.15.174:6823/90877","heartbeat_front_addr":"172.21.15.174:6821/90877","state":["exists","up"]},{"osd":7,"uuid":"49967e34-8b07-4bab-a13e-0c16d6a0f3a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":90878},{"type":"v1","addr":"172.21.15.174:6825","nonce":90878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":90878},{"type":"v1","addr":"172.21.15.174:6827","nonce":90878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":90878},{"type":"v1","addr":"172.21.15.174:6831","nonce":90878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":90878},{"type":"v1","addr":"172.21.15.174:6829","nonce":90878}]},"public_addr":"172.21.15.174:6825/90878","cluster_addr":"172.21.15.174:6827/90878","heartbeat_back_addr":"172.21.15.174:6831/90878","heartbeat_front_addr":"172.21.15.174:6829/90878","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.446479+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.514266+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.794108+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.810894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.731919+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.717756+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.737435+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.508682+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T05:06:52.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.935+0000 7fc3935f2700 1 -- 172.21.15.98:0/173452643 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fc370066f30 msgr2=0x7fc3700693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:52.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.935+0000 7fc3935f2700 1 --2- 172.21.15.98:0/173452643 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fc370066f30 0x7fc3700693e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc388006f00 tx=0x7fc3880091a0 comp rx=0 tx=0).stop 2023-10-06T05:06:52.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.935+0000 7fc3935f2700 1 -- 172.21.15.98:0/173452643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc38c0c23a0 msgr2=0x7fc38c0c2810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:52.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.935+0000 7fc3935f2700 1 --2- 172.21.15.98:0/173452643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc38c0c23a0 0x7fc38c0c2810 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc37c002920 tx=0x7fc37c007d30 comp rx=0 tx=0).stop 2023-10-06T05:06:52.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.935+0000 7fc3935f2700 1 -- 172.21.15.98:0/173452643 shutdown_connections 2023-10-06T05:06:52.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.935+0000 7fc3935f2700 1 --2- 172.21.15.98:0/173452643 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc38c0a62c0 0x7fc38c0a6760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:52.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.935+0000 7fc3935f2700 1 --2- 172.21.15.98:0/173452643 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc378004d50 0x7fc38c151ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:52.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.935+0000 7fc3935f2700 1 --2- 172.21.15.98:0/173452643 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fc370066f30 0x7fc3700693e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:52.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.935+0000 7fc3935f2700 1 --2- 172.21.15.98:0/173452643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc38c0c23a0 0x7fc38c0c2810 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:52.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.935+0000 7fc3935f2700 1 -- 172.21.15.98:0/173452643 >> 172.21.15.98:0/173452643 conn(0x7fc38c0cdeb0 msgr2=0x7fc38c155160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:06:52.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.935+0000 7fc3935f2700 1 -- 172.21.15.98:0/173452643 shutdown_connections 2023-10-06T05:06:52.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:52.935+0000 7fc3935f2700 1 -- 172.21.15.98:0/173452643 wait complete. 2023-10-06T05:06:52.949 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-06T05:06:44.559671+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-10-06T05:06:51.432161+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2023-10-06T05:06:52.949 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-06T05:06:53.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.064+0000 7f639f66c700 1 Processor -- start 2023-10-06T05:06:53.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.064+0000 7f639f66c700 1 -- start start 2023-10-06T05:06:53.068 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.065+0000 7f639f66c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f63980cd940 0x7f6398159bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:53.068 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.065+0000 7f639f66c700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f63980b4a20 0x7f63980d02d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:53.068 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.065+0000 7f639f66c700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63980b5ef0 con 0x7f63980b4e70 2023-10-06T05:06:53.068 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.065+0000 7f639f66c700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63980b3600 con 0x7f63980cd940 2023-10-06T05:06:53.068 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.065+0000 7f639f66c700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63980b6380 con 0x7f63980b4a20 2023-10-06T05:06:53.068 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.065+0000 7f639d408700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f63980cd940 0x7f6398159bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:53.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.065+0000 7f639dc09700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f63980b4e70 0x7f639815ada0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:49342/0 (socket says 172.21.15.98:49342) 2023-10-06T05:06:53.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.065+0000 7f639dc09700 1 -- 172.21.15.98:0/3925252344 learned_addr learned my addr 172.21.15.98:0/3925252344 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:06:53.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.065+0000 7f639cc07700 1 --2- 172.21.15.98:0/3925252344 >> v2:172.21.15.174:3300/0 conn(0x7f63980b4a20 0x7f63980d02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:53.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.065+0000 7f639d408700 1 -- 172.21.15.98:0/3925252344 >> v2:172.21.15.174:3300/0 conn(0x7f63980b4a20 msgr2=0x7f63980d02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:06:53.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.065+0000 7f639d408700 1 --2- 172.21.15.98:0/3925252344 >> v2:172.21.15.174:3300/0 conn(0x7f63980b4a20 0x7f63980d02d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:53.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.065+0000 7f639d408700 1 -- 172.21.15.98:0/3925252344 >> v1:172.21.15.98:6789/0 conn(0x7f63980b4e70 legacy=0x7f639815ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:53.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.065+0000 7f639d408700 1 -- 172.21.15.98:0/3925252344 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63980b52e0 con 0x7f63980cd940 2023-10-06T05:06:53.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.065+0000 7f638ffff700 1 -- 172.21.15.98:0/3925252344 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2230711305 0 0) 0x7f63980b5ef0 con 0x7f63980b4e70 2023-10-06T05:06:53.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.065+0000 7f639cc07700 1 --2- 172.21.15.98:0/3925252344 >> v2:172.21.15.174:3300/0 conn(0x7f63980b4a20 0x7f63980d02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:06:53.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.065+0000 7f639d408700 1 --2- 172.21.15.98:0/3925252344 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63980cd940 0x7f6398159bd0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f63980af140 tx=0x7f6394009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9b75bac62ed873c server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:06:53.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.065+0000 7f638ffff700 1 -- 172.21.15.98:0/3925252344 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f639400e070 con 0x7f63980cd940 2023-10-06T05:06:53.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.065+0000 7f638ffff700 1 -- 172.21.15.98:0/3925252344 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f639400bc30 con 0x7f63980cd940 2023-10-06T05:06:53.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.065+0000 7f638ffff700 1 -- 172.21.15.98:0/3925252344 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6394012430 con 0x7f63980cd940 2023-10-06T05:06:53.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.066+0000 7f639f66c700 1 -- 172.21.15.98:0/3925252344 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63980cd940 msgr2=0x7f6398159bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:53.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.066+0000 7f639f66c700 1 --2- 172.21.15.98:0/3925252344 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63980cd940 0x7f6398159bd0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f63980af140 tx=0x7f6394009780 comp rx=0 tx=0).stop 2023-10-06T05:06:53.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.066+0000 7f639f66c700 1 -- 172.21.15.98:0/3925252344 shutdown_connections 2023-10-06T05:06:53.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.066+0000 7f639f66c700 1 --2- 172.21.15.98:0/3925252344 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63980cd940 0x7f6398159bd0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:53.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.066+0000 7f639f66c700 1 --2- 172.21.15.98:0/3925252344 >> v2:172.21.15.174:3300/0 conn(0x7f63980b4a20 0x7f63980d02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:53.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.066+0000 7f639f66c700 1 -- 172.21.15.98:0/3925252344 >> 172.21.15.98:0/3925252344 conn(0x7f639806ece0 msgr2=0x7f639806f0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:06:53.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.066+0000 7f639f66c700 1 -- 172.21.15.98:0/3925252344 shutdown_connections 2023-10-06T05:06:53.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.066+0000 7f639f66c700 1 -- 172.21.15.98:0/3925252344 wait complete. 2023-10-06T05:06:53.073 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.067+0000 7f639f66c700 1 Processor -- start 2023-10-06T05:06:53.073 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.067+0000 7f639f66c700 1 -- start start 2023-10-06T05:06:53.073 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.068+0000 7f639f66c700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63980b4a20 0x7f639814bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:53.073 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.068+0000 7f639f66c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63980b4e70 0x7f639814e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:53.073 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.068+0000 7f639f66c700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f63980cd940 0x7f639814fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:53.073 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.068+0000 7f639f66c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f639814ecc0 con 0x7f63980b4e70 2023-10-06T05:06:53.073 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.068+0000 7f639f66c700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f639814efa0 con 0x7f63980b4a20 2023-10-06T05:06:53.073 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.068+0000 7f639f66c700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f639814f280 con 0x7f63980cd940 2023-10-06T05:06:53.073 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.068+0000 7f639cc07700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63980b4e70 0x7f639814e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:53.073 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.068+0000 7f639cc07700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63980b4e70 0x7f639814e370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54102/0 (socket says 172.21.15.98:54102) 2023-10-06T05:06:53.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.068+0000 7f639cc07700 1 -- 172.21.15.98:0/2233569445 learned_addr learned my addr 172.21.15.98:0/2233569445 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:06:53.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.068+0000 7f639d408700 1 --2- 172.21.15.98:0/2233569445 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63980b4a20 0x7f639814bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:53.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.068+0000 7f639dc09700 1 --2- 172.21.15.98:0/2233569445 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f63980cd940 0x7f639814fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:53.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.068+0000 7f639cc07700 1 -- 172.21.15.98:0/2233569445 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f63980cd940 msgr2=0x7f639814fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:53.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.068+0000 7f639cc07700 1 --2- 172.21.15.98:0/2233569445 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f63980cd940 0x7f639814fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:53.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.068+0000 7f639cc07700 1 -- 172.21.15.98:0/2233569445 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63980b4a20 msgr2=0x7f639814bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:53.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.068+0000 7f639cc07700 1 --2- 172.21.15.98:0/2233569445 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63980b4a20 0x7f639814bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:53.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.068+0000 7f639cc07700 1 -- 172.21.15.98:0/2233569445 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6394009a20 con 0x7f63980b4e70 2023-10-06T05:06:53.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.068+0000 7f639d408700 1 --2- 172.21.15.98:0/2233569445 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63980b4a20 0x7f639814bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:06:53.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.068+0000 7f639cc07700 1 --2- 172.21.15.98:0/2233569445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63980b4e70 0x7f639814e370 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f638800c830 tx=0x7f6388008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:06:53.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.068+0000 7f638d7fa700 1 -- 172.21.15.98:0/2233569445 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6388011600 con 0x7f63980b4e70 2023-10-06T05:06:53.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.069+0000 7f638d7fa700 1 -- 172.21.15.98:0/2233569445 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6388011760 con 0x7f63980b4e70 2023-10-06T05:06:53.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.069+0000 7f639f66c700 1 -- 172.21.15.98:0/2233569445 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f639814f560 con 0x7f63980b4e70 2023-10-06T05:06:53.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.069+0000 7f638d7fa700 1 -- 172.21.15.98:0/2233569445 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6388011b20 con 0x7f63980b4e70 2023-10-06T05:06:53.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.069+0000 7f639f66c700 1 -- 172.21.15.98:0/2233569445 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f639814e8b0 con 0x7f63980b4e70 2023-10-06T05:06:53.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.069+0000 7f639f66c700 1 -- 172.21.15.98:0/2233569445 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6398064a90 con 0x7f63980b4e70 2023-10-06T05:06:53.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.070+0000 7f638d7fa700 1 -- 172.21.15.98:0/2233569445 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f638800a3e0 con 0x7f63980b4e70 2023-10-06T05:06:53.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.070+0000 7f638d7fa700 1 --2- 172.21.15.98:0/2233569445 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f637c066f30 0x7f637c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:53.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.070+0000 7f638d7fa700 1 -- 172.21.15.98:0/2233569445 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f6388084810 con 0x7f63980b4e70 2023-10-06T05:06:53.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.070+0000 7f639d408700 1 --2- 172.21.15.98:0/2233569445 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f637c066f30 0x7f637c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:53.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.071+0000 7f639d408700 1 --2- 172.21.15.98:0/2233569445 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f637c066f30 0x7f637c0693e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f63980af140 tx=0x7f6394019000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:06:53.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.074+0000 7f638d7fa700 1 -- 172.21.15.98:0/2233569445 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6388051c70 con 0x7f63980b4e70 2023-10-06T05:06:53.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.256+0000 7f639f66c700 1 -- 172.21.15.98:0/2233569445 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f63980b55c0 con 0x7f63980b4e70 2023-10-06T05:06:53.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.256+0000 7f638d7fa700 1 -- 172.21.15.98:0/2233569445 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f63880150a0 con 0x7f63980b4e70 2023-10-06T05:06:53.259 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2023-10-06T05:06:53.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.258+0000 7f639f66c700 1 -- 172.21.15.98:0/2233569445 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f637c066f30 msgr2=0x7f637c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:53.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.258+0000 7f639f66c700 1 --2- 172.21.15.98:0/2233569445 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f637c066f30 0x7f637c0693e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f63980af140 tx=0x7f6394019000 comp rx=0 tx=0).stop 2023-10-06T05:06:53.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.258+0000 7f639f66c700 1 -- 172.21.15.98:0/2233569445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63980b4e70 msgr2=0x7f639814e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:53.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.258+0000 7f639f66c700 1 --2- 172.21.15.98:0/2233569445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63980b4e70 0x7f639814e370 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f638800c830 tx=0x7f6388008040 comp rx=0 tx=0).stop 2023-10-06T05:06:53.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.258+0000 7f639f66c700 1 -- 172.21.15.98:0/2233569445 shutdown_connections 2023-10-06T05:06:53.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.258+0000 7f639f66c700 1 --2- 172.21.15.98:0/2233569445 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f63980cd940 0x7f639814fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:53.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.258+0000 7f639f66c700 1 --2- 172.21.15.98:0/2233569445 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63980b4a20 0x7f639814bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:53.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.258+0000 7f639f66c700 1 --2- 172.21.15.98:0/2233569445 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f637c066f30 0x7f637c0693e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:53.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.258+0000 7f639f66c700 1 --2- 172.21.15.98:0/2233569445 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63980b4e70 0x7f639814e370 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:53.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.258+0000 7f639f66c700 1 -- 172.21.15.98:0/2233569445 >> 172.21.15.98:0/2233569445 conn(0x7f639806ece0 msgr2=0x7f639815c930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:06:53.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.258+0000 7f639f66c700 1 -- 172.21.15.98:0/2233569445 shutdown_connections 2023-10-06T05:06:53.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.258+0000 7f639f66c700 1 -- 172.21.15.98:0/2233569445 wait complete. 2023-10-06T05:06:53.272 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-10-06T05:06:53.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.386+0000 7f96c81a0700 1 Processor -- start 2023-10-06T05:06:53.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.386+0000 7f96c81a0700 1 -- start start 2023-10-06T05:06:53.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.387+0000 7f96c81a0700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f96c01594d0 0x7f96c00cb560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:53.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.387+0000 7f96c81a0700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f96c01598a0 0x7f96c00cbaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:53.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.387+0000 7f96c81a0700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96c00cf310 con 0x7f96c00ce7e0 2023-10-06T05:06:53.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.387+0000 7f96c81a0700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96c00cf5d0 con 0x7f96c01598a0 2023-10-06T05:06:53.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.387+0000 7f96c81a0700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96c00cf8b0 con 0x7f96c01594d0 2023-10-06T05:06:53.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.387+0000 7f96c573b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f96c01598a0 0x7f96c00cbaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:53.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.387+0000 7f96c573b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f96c01598a0 0x7f96c00cbaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54112/0 (socket says 172.21.15.98:54112) 2023-10-06T05:06:53.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.387+0000 7f96c573b700 1 -- 172.21.15.98:0/2928049303 learned_addr learned my addr 172.21.15.98:0/2928049303 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:06:53.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.387+0000 7f96c5f3c700 1 --2- 172.21.15.98:0/2928049303 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f96c01594d0 0x7f96c00cb560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:53.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.387+0000 7f96c573b700 1 -- 172.21.15.98:0/2928049303 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f96c01594d0 msgr2=0x7f96c00cb560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:06:53.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.387+0000 7f96c573b700 1 --2- 172.21.15.98:0/2928049303 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f96c01594d0 0x7f96c00cb560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:53.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.387+0000 7f96c573b700 1 -- 172.21.15.98:0/2928049303 >> v1:172.21.15.174:6789/0 conn(0x7f96c00ce7e0 legacy=0x7f96c00cec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:53.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.387+0000 7f96c573b700 1 -- 172.21.15.98:0/2928049303 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96c00cfb90 con 0x7f96c01598a0 2023-10-06T05:06:53.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.388+0000 7f96c573b700 1 --2- 172.21.15.98:0/2928049303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96c01598a0 0x7f96c00cbaa0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f96b00099b0 tx=0x7f96b0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b9f5dc9090f5516 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:06:53.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.388+0000 7f96c4f3a700 1 -- 172.21.15.98:0/2928049303 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96b000b860 con 0x7f96c01598a0 2023-10-06T05:06:53.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.388+0000 7f96c4f3a700 1 -- 172.21.15.98:0/2928049303 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96b000b9c0 con 0x7f96c01598a0 2023-10-06T05:06:53.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.388+0000 7f96c4f3a700 1 -- 172.21.15.98:0/2928049303 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96b000bd60 con 0x7f96c01598a0 2023-10-06T05:06:53.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.388+0000 7f96c81a0700 1 -- 172.21.15.98:0/2928049303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96c01598a0 msgr2=0x7f96c00cbaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:53.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.388+0000 7f96c81a0700 1 --2- 172.21.15.98:0/2928049303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96c01598a0 0x7f96c00cbaa0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f96b00099b0 tx=0x7f96b0009250 comp rx=0 tx=0).stop 2023-10-06T05:06:53.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.388+0000 7f96c81a0700 1 -- 172.21.15.98:0/2928049303 shutdown_connections 2023-10-06T05:06:53.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.388+0000 7f96c81a0700 1 --2- 172.21.15.98:0/2928049303 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f96c01594d0 0x7f96c00cb560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:53.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.388+0000 7f96c81a0700 1 --2- 172.21.15.98:0/2928049303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96c01598a0 0x7f96c00cbaa0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:53.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.388+0000 7f96c81a0700 1 -- 172.21.15.98:0/2928049303 >> 172.21.15.98:0/2928049303 conn(0x7f96c0089e50 msgr2=0x7f96c0099770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:06:53.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.388+0000 7f96c81a0700 1 -- 172.21.15.98:0/2928049303 shutdown_connections 2023-10-06T05:06:53.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.388+0000 7f96c81a0700 1 -- 172.21.15.98:0/2928049303 wait complete. 2023-10-06T05:06:53.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.390+0000 7f96c81a0700 1 Processor -- start 2023-10-06T05:06:53.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.390+0000 7f96c81a0700 1 -- start start 2023-10-06T05:06:53.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.390+0000 7f96c81a0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96c00ce7e0 0x7f96c015c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:53.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.390+0000 7f96c81a0700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f96c01594d0 0x7f96c0146f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:53.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.390+0000 7f96c81a0700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f96c01598a0 0x7f96c00b3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:53.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.390+0000 7f96c81a0700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96c0158970 con 0x7f96c00ce7e0 2023-10-06T05:06:53.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.390+0000 7f96c81a0700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96c00c8390 con 0x7f96c01594d0 2023-10-06T05:06:53.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.390+0000 7f96c81a0700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96c00c8670 con 0x7f96c01598a0 2023-10-06T05:06:53.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.390+0000 7f96c5f3c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96c00ce7e0 0x7f96c015c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:53.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.390+0000 7f96c5f3c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96c00ce7e0 0x7f96c015c480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54122/0 (socket says 172.21.15.98:54122) 2023-10-06T05:06:53.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.390+0000 7f96c5f3c700 1 -- 172.21.15.98:0/3092815077 learned_addr learned my addr 172.21.15.98:0/3092815077 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:06:53.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.390+0000 7f96c573b700 1 --2- 172.21.15.98:0/3092815077 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f96c01594d0 0x7f96c0146f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:53.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.390+0000 7f96c673d700 1 --2- 172.21.15.98:0/3092815077 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f96c01598a0 0x7f96c00b3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:53.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.391+0000 7f96c5f3c700 1 -- 172.21.15.98:0/3092815077 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f96c01598a0 msgr2=0x7f96c00b3770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:53.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.391+0000 7f96c5f3c700 1 --2- 172.21.15.98:0/3092815077 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f96c01598a0 0x7f96c00b3770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:53.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.391+0000 7f96c5f3c700 1 -- 172.21.15.98:0/3092815077 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f96c01594d0 msgr2=0x7f96c0146f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:53.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.391+0000 7f96c5f3c700 1 --2- 172.21.15.98:0/3092815077 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f96c01594d0 0x7f96c0146f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:53.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.391+0000 7f96c5f3c700 1 -- 172.21.15.98:0/3092815077 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96b00094f0 con 0x7f96c00ce7e0 2023-10-06T05:06:53.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.391+0000 7f96c5f3c700 1 --2- 172.21.15.98:0/3092815077 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96c00ce7e0 0x7f96c015c480 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f96bc00e190 tx=0x7f96bc009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:06:53.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.391+0000 7f96b5ffb700 1 -- 172.21.15.98:0/3092815077 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96bc00ed80 con 0x7f96c00ce7e0 2023-10-06T05:06:53.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.391+0000 7f96b5ffb700 1 -- 172.21.15.98:0/3092815077 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96bc014420 con 0x7f96c00ce7e0 2023-10-06T05:06:53.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.391+0000 7f96c81a0700 1 -- 172.21.15.98:0/3092815077 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96c00c8950 con 0x7f96c00ce7e0 2023-10-06T05:06:53.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.391+0000 7f96b5ffb700 1 -- 172.21.15.98:0/3092815077 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96bc0147e0 con 0x7f96c00ce7e0 2023-10-06T05:06:53.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.391+0000 7f96c81a0700 1 -- 172.21.15.98:0/3092815077 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96c00c8f10 con 0x7f96c00ce7e0 2023-10-06T05:06:53.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.392+0000 7f96c81a0700 1 -- 172.21.15.98:0/3092815077 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96c00c90a0 con 0x7f96c00ce7e0 2023-10-06T05:06:53.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.392+0000 7f96b5ffb700 1 -- 172.21.15.98:0/3092815077 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f96bc014b90 con 0x7f96c00ce7e0 2023-10-06T05:06:53.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.393+0000 7f96b5ffb700 1 --2- 172.21.15.98:0/3092815077 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f96a4066f30 0x7f96a40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:53.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.393+0000 7f96b5ffb700 1 -- 172.21.15.98:0/3092815077 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f96bc084af0 con 0x7f96c00ce7e0 2023-10-06T05:06:53.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.393+0000 7f96c573b700 1 --2- 172.21.15.98:0/3092815077 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f96a4066f30 0x7f96a40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:53.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.393+0000 7f96c573b700 1 --2- 172.21.15.98:0/3092815077 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f96a4066f30 0x7f96a40693e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f96b00099b0 tx=0x7f96b0017000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:06:53.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.397+0000 7f96b5ffb700 1 -- 172.21.15.98:0/3092815077 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f96bc051f50 con 0x7f96c00ce7e0 2023-10-06T05:06:53.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.578+0000 7f96c81a0700 1 -- 172.21.15.98:0/3092815077 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f96c0064ac0 con 0x7f96c00ce7e0 2023-10-06T05:06:53.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.579+0000 7f96b5ffb700 1 -- 172.21.15.98:0/3092815077 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v20) v1 ==== 92+0+10 (secure 0 0 0) 0x7f96bc009a60 con 0x7f96c00ce7e0 2023-10-06T05:06:53.581 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 8 2023-10-06T05:06:53.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.581+0000 7f96c81a0700 1 -- 172.21.15.98:0/3092815077 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f96a4066f30 msgr2=0x7f96a40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:53.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.581+0000 7f96c81a0700 1 --2- 172.21.15.98:0/3092815077 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f96a4066f30 0x7f96a40693e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f96b00099b0 tx=0x7f96b0017000 comp rx=0 tx=0).stop 2023-10-06T05:06:53.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.581+0000 7f96c81a0700 1 -- 172.21.15.98:0/3092815077 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96c00ce7e0 msgr2=0x7f96c015c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:53.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.581+0000 7f96c81a0700 1 --2- 172.21.15.98:0/3092815077 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96c00ce7e0 0x7f96c015c480 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f96bc00e190 tx=0x7f96bc009920 comp rx=0 tx=0).stop 2023-10-06T05:06:53.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.581+0000 7f96c81a0700 1 -- 172.21.15.98:0/3092815077 shutdown_connections 2023-10-06T05:06:53.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.581+0000 7f96c81a0700 1 --2- 172.21.15.98:0/3092815077 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f96c01598a0 0x7f96c00b3770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:53.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.581+0000 7f96c81a0700 1 --2- 172.21.15.98:0/3092815077 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f96a4066f30 0x7f96a40693e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:53.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.581+0000 7f96c81a0700 1 --2- 172.21.15.98:0/3092815077 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96c00ce7e0 0x7f96c015c480 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:53.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.581+0000 7f96c81a0700 1 --2- 172.21.15.98:0/3092815077 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f96c01594d0 0x7f96c0146f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:53.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.581+0000 7f96c81a0700 1 -- 172.21.15.98:0/3092815077 >> 172.21.15.98:0/3092815077 conn(0x7f96c0089e50 msgr2=0x7f96c00d04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:06:53.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.581+0000 7f96c81a0700 1 -- 172.21.15.98:0/3092815077 shutdown_connections 2023-10-06T05:06:53.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.581+0000 7f96c81a0700 1 -- 172.21.15.98:0/3092815077 wait complete. 2023-10-06T05:06:53.596 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-06T05:06:53.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.709+0000 7f3daa873700 1 Processor -- start 2023-10-06T05:06:53.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.709+0000 7f3daa873700 1 -- start start 2023-10-06T05:06:53.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.710+0000 7f3daa873700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f3da415a870 0x7f3da40ce980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:53.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.710+0000 7f3daa873700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3da40cdb40 con 0x7f3da40c8f20 2023-10-06T05:06:53.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.710+0000 7f3daa873700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3da40ccf30 con 0x7f3da415ac40 2023-10-06T05:06:53.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.710+0000 7f3daa873700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3da40cd210 con 0x7f3da415a870 2023-10-06T05:06:53.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.710+0000 7f3da8e10700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f3da40c8f20 0x7f3da40ab3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:49348/0 (socket says 172.21.15.98:49348) 2023-10-06T05:06:53.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.710+0000 7f3da8e10700 1 -- 172.21.15.98:0/3989966162 learned_addr learned my addr 172.21.15.98:0/3989966162 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:06:53.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.710+0000 7f3da2ffd700 1 -- 172.21.15.98:0/3989966162 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 290225293 0 0) 0x7f3da40cdb40 con 0x7f3da40c8f20 2023-10-06T05:06:53.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.710+0000 7f3da2ffd700 1 -- 172.21.15.98:0/3989966162 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d8c003570 con 0x7f3da40c8f20 2023-10-06T05:06:53.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.710+0000 7f3da3fff700 1 --2- 172.21.15.98:0/3989966162 >> v2:172.21.15.174:3300/0 conn(0x7f3da415a870 0x7f3da40ce980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:53.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.711+0000 7f3da2ffd700 1 -- 172.21.15.98:0/3989966162 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2237615284 0 0) 0x7f3d8c003570 con 0x7f3da40c8f20 2023-10-06T05:06:53.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.711+0000 7f3da2ffd700 1 -- 172.21.15.98:0/3989966162 >> v2:172.21.15.174:3300/0 conn(0x7f3da415a870 msgr2=0x7f3da40ce980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:06:53.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.711+0000 7f3da2ffd700 1 --2- 172.21.15.98:0/3989966162 >> v2:172.21.15.174:3300/0 conn(0x7f3da415a870 0x7f3da40ce980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:53.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.711+0000 7f3da2ffd700 1 -- 172.21.15.98:0/3989966162 >> v1:172.21.15.174:6789/0 conn(0x7f3da415ac40 legacy=0x7f3da40c8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:53.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.711+0000 7f3da2ffd700 1 -- 172.21.15.98:0/3989966162 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3da40cd4f0 con 0x7f3da40c8f20 2023-10-06T05:06:53.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.711+0000 7f3da2ffd700 1 -- 172.21.15.98:0/3989966162 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1543991876 0 0) 0x7f3d9c0017f0 con 0x7f3da40c8f20 2023-10-06T05:06:53.719 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.711+0000 7f3da2ffd700 1 -- 172.21.15.98:0/3989966162 >> v1:172.21.15.98:6789/0 conn(0x7f3da40c8f20 legacy=0x7f3da40ab3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:53.719 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.711+0000 7f3da2ffd700 1 --2- 172.21.15.98:0/3989966162 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3d8c003a60 0x7f3d8c003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:53.719 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.711+0000 7f3da2ffd700 1 --2- 172.21.15.98:0/3989966162 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3d8c004810 0x7f3d8c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:53.719 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.711+0000 7f3da2ffd700 1 --2- 172.21.15.98:0/3989966162 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d8c0051c0 0x7f3d8c00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:53.719 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.711+0000 7f3da2ffd700 1 -- 172.21.15.98:0/3989966162 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3da40cd210 con 0x7f3d8c0051c0 2023-10-06T05:06:53.719 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.711+0000 7f3da2ffd700 1 -- 172.21.15.98:0/3989966162 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d8c00c0d0 con 0x7f3d8c004810 2023-10-06T05:06:53.719 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.711+0000 7f3da2ffd700 1 -- 172.21.15.98:0/3989966162 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d8c00c290 con 0x7f3d8c003a60 2023-10-06T05:06:53.719 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.711+0000 7f3da8e10700 1 --2- 172.21.15.98:0/3989966162 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d8c0051c0 0x7f3d8c00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:53.719 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.711+0000 7f3da3fff700 1 --2- 172.21.15.98:0/3989966162 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3d8c003a60 0x7f3d8c003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:53.719 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.711+0000 7f3da37fe700 1 --2- 172.21.15.98:0/3989966162 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3d8c004810 0x7f3d8c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:53.719 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.711+0000 7f3da8e10700 1 -- 172.21.15.98:0/3989966162 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3d8c003a60 msgr2=0x7f3d8c003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:06:53.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.711+0000 7f3da8e10700 1 --2- 172.21.15.98:0/3989966162 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3d8c003a60 0x7f3d8c003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:53.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.711+0000 7f3da8e10700 1 -- 172.21.15.98:0/3989966162 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3d8c004810 msgr2=0x7f3d8c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:06:53.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.711+0000 7f3da8e10700 1 --2- 172.21.15.98:0/3989966162 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3d8c004810 0x7f3d8c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:53.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.711+0000 7f3da8e10700 1 -- 172.21.15.98:0/3989966162 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d8c003570 con 0x7f3d8c0051c0 2023-10-06T05:06:53.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.711+0000 7f3da8e10700 1 -- 172.21.15.98:0/3989966162 reap_dead start 2023-10-06T05:06:53.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.711+0000 7f3da8e10700 1 --2- 172.21.15.98:0/3989966162 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d8c0051c0 0x7f3d8c00ba60 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c00a930 tx=0x7f3d9c0021d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=552178863c5b084d server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:06:53.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.711+0000 7f3da2ffd700 1 -- 172.21.15.98:0/3989966162 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d9c0070d0 con 0x7f3d8c0051c0 2023-10-06T05:06:53.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.711+0000 7f3da2ffd700 1 -- 172.21.15.98:0/3989966162 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d9c007230 con 0x7f3d8c0051c0 2023-10-06T05:06:53.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.711+0000 7f3da2ffd700 1 -- 172.21.15.98:0/3989966162 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d9c0075d0 con 0x7f3d8c0051c0 2023-10-06T05:06:53.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.712+0000 7f3daa873700 1 -- 172.21.15.98:0/3989966162 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d8c0051c0 msgr2=0x7f3d8c00ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:53.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.712+0000 7f3daa873700 1 --2- 172.21.15.98:0/3989966162 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d8c0051c0 0x7f3d8c00ba60 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c00a930 tx=0x7f3d9c0021d0 comp rx=0 tx=0).stop 2023-10-06T05:06:53.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.712+0000 7f3daa873700 1 -- 172.21.15.98:0/3989966162 shutdown_connections 2023-10-06T05:06:53.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.712+0000 7f3daa873700 1 --2- 172.21.15.98:0/3989966162 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d8c0051c0 0x7f3d8c00ba60 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:53.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.712+0000 7f3daa873700 1 -- 172.21.15.98:0/3989966162 >> 172.21.15.98:0/3989966162 conn(0x7f3da4067d90 msgr2=0x7f3da40ce270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:06:53.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.712+0000 7f3daa873700 1 -- 172.21.15.98:0/3989966162 shutdown_connections 2023-10-06T05:06:53.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.712+0000 7f3daa873700 1 -- 172.21.15.98:0/3989966162 wait complete. 2023-10-06T05:06:53.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.713+0000 7f3daa873700 1 Processor -- start 2023-10-06T05:06:53.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.713+0000 7f3daa873700 1 -- start start 2023-10-06T05:06:53.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.714+0000 7f3daa873700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3d8c0051c0 0x7f3da406b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:53.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.714+0000 7f3daa873700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3da40af910 0x7f3da40afd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:53.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.714+0000 7f3daa873700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3da40b26e0 0x7f3da40b2b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:53.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.714+0000 7f3daa873700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3da40b30a0 con 0x7f3da40af910 2023-10-06T05:06:53.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.714+0000 7f3daa873700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3da40af480 con 0x7f3d8c0051c0 2023-10-06T05:06:53.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.714+0000 7f3daa873700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3da40ae870 con 0x7f3da40b26e0 2023-10-06T05:06:53.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.714+0000 7f3da37fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3da40af910 0x7f3da40afd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:53.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.714+0000 7f3da37fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3da40af910 0x7f3da40afd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54146/0 (socket says 172.21.15.98:54146) 2023-10-06T05:06:53.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.714+0000 7f3da37fe700 1 -- 172.21.15.98:0/2305606303 learned_addr learned my addr 172.21.15.98:0/2305606303 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:06:53.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.714+0000 7f3da3fff700 1 --2- 172.21.15.98:0/2305606303 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3d8c0051c0 0x7f3da406b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:53.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.714+0000 7f3da8e10700 1 --2- 172.21.15.98:0/2305606303 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3da40b26e0 0x7f3da40b2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:53.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.714+0000 7f3da37fe700 1 -- 172.21.15.98:0/2305606303 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3da40b26e0 msgr2=0x7f3da40b2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:53.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.714+0000 7f3da37fe700 1 --2- 172.21.15.98:0/2305606303 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3da40b26e0 0x7f3da40b2b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:53.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.714+0000 7f3da37fe700 1 -- 172.21.15.98:0/2305606303 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3d8c0051c0 msgr2=0x7f3da406b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:53.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.714+0000 7f3da37fe700 1 --2- 172.21.15.98:0/2305606303 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3d8c0051c0 0x7f3da406b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:53.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.714+0000 7f3da37fe700 1 -- 172.21.15.98:0/2305606303 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3da40aeb50 con 0x7f3da40af910 2023-10-06T05:06:53.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.714+0000 7f3da3fff700 1 --2- 172.21.15.98:0/2305606303 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3d8c0051c0 0x7f3da406b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:06:53.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.714+0000 7f3da37fe700 1 --2- 172.21.15.98:0/2305606303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3da40af910 0x7f3da40afd80 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f3d98002920 tx=0x7f3d98007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:06:53.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.714+0000 7f3d8bfff700 1 -- 172.21.15.98:0/2305606303 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d98002e60 con 0x7f3da40af910 2023-10-06T05:06:53.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.714+0000 7f3d8bfff700 1 -- 172.21.15.98:0/2305606303 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d9800f8d0 con 0x7f3da40af910 2023-10-06T05:06:53.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.714+0000 7f3daa873700 1 -- 172.21.15.98:0/2305606303 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3da40aee30 con 0x7f3da40af910 2023-10-06T05:06:53.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.714+0000 7f3d8bfff700 1 -- 172.21.15.98:0/2305606303 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d9800fc70 con 0x7f3da40af910 2023-10-06T05:06:53.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.715+0000 7f3daa873700 1 -- 172.21.15.98:0/2305606303 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3da406bc30 con 0x7f3da40af910 2023-10-06T05:06:53.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.715+0000 7f3daa873700 1 -- 172.21.15.98:0/2305606303 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3da40af610 con 0x7f3da40af910 2023-10-06T05:06:53.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.716+0000 7f3d8bfff700 1 -- 172.21.15.98:0/2305606303 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f3d9801c400 con 0x7f3da40af910 2023-10-06T05:06:53.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.716+0000 7f3d8bfff700 1 --2- 172.21.15.98:0/2305606303 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f3d90066f30 0x7f3d900693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:53.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.716+0000 7f3d8bfff700 1 -- 172.21.15.98:0/2305606303 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f3d98084420 con 0x7f3da40af910 2023-10-06T05:06:53.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.716+0000 7f3da3fff700 1 --2- 172.21.15.98:0/2305606303 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f3d90066f30 0x7f3d900693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:53.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.716+0000 7f3da3fff700 1 --2- 172.21.15.98:0/2305606303 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f3d90066f30 0x7f3d900693e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3d94006ea0 tx=0x7f3d94008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:06:53.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.720+0000 7f3d8bfff700 1 -- 172.21.15.98:0/2305606303 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3d98051880 con 0x7f3da40af910 2023-10-06T05:06:53.947 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.945+0000 7f3daa873700 1 -- 172.21.15.98:0/2305606303 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3da4064a90 con 0x7f3da40af910 2023-10-06T05:06:53.947 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.945+0000 7f3d8bfff700 1 -- 172.21.15.98:0/2305606303 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3527 (secure 0 0 0) 0x7f3d98054ea0 con 0x7f3da40af910 2023-10-06T05:06:53.947 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:06:53.948 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4285,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6834/925250632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":925250632},{"type":"v1","addr":"172.21.15.174:6834","nonce":925250632}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4288,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/1629449810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6833","nonce":1629449810},{"type":"v1","addr":"172.21.15.174:6835","nonce":1629449810}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4299,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/3961327990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3961327990},{"type":"v1","addr":"172.21.15.98:6836","nonce":3961327990}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4302,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/2683239094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":2683239094},{"type":"v1","addr":"172.21.15.98:6837","nonce":2683239094}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2023-10-06T05:06:53.949 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.947+0000 7f3daa873700 1 -- 172.21.15.98:0/2305606303 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f3d90066f30 msgr2=0x7f3d900693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:53.949 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.947+0000 7f3daa873700 1 --2- 172.21.15.98:0/2305606303 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f3d90066f30 0x7f3d900693e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3d94006ea0 tx=0x7f3d94008040 comp rx=0 tx=0).stop 2023-10-06T05:06:53.950 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.947+0000 7f3daa873700 1 -- 172.21.15.98:0/2305606303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3da40af910 msgr2=0x7f3da40afd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:53.950 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.947+0000 7f3daa873700 1 --2- 172.21.15.98:0/2305606303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3da40af910 0x7f3da40afd80 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f3d98002920 tx=0x7f3d98007d30 comp rx=0 tx=0).stop 2023-10-06T05:06:53.950 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.947+0000 7f3daa873700 1 -- 172.21.15.98:0/2305606303 shutdown_connections 2023-10-06T05:06:53.950 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.947+0000 7f3daa873700 1 --2- 172.21.15.98:0/2305606303 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3da40b26e0 0x7f3da40b2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:53.950 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.947+0000 7f3daa873700 1 --2- 172.21.15.98:0/2305606303 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3d8c0051c0 0x7f3da406b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:53.950 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.947+0000 7f3daa873700 1 --2- 172.21.15.98:0/2305606303 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f3d90066f30 0x7f3d900693e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:53.950 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.947+0000 7f3daa873700 1 --2- 172.21.15.98:0/2305606303 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3da40af910 0x7f3da40afd80 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:53.950 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.947+0000 7f3daa873700 1 -- 172.21.15.98:0/2305606303 >> 172.21.15.98:0/2305606303 conn(0x7f3da4067d90 msgr2=0x7f3da4156740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:06:53.950 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.947+0000 7f3daa873700 1 -- 172.21.15.98:0/2305606303 shutdown_connections 2023-10-06T05:06:53.951 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:53.947+0000 7f3daa873700 1 -- 172.21.15.98:0/2305606303 wait complete. 2023-10-06T05:06:53.951 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 3 2023-10-06T05:06:53.961 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T05:06:54.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.073+0000 7f0b92c65700 1 Processor -- start 2023-10-06T05:06:54.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.073+0000 7f0b92c65700 1 -- start start 2023-10-06T05:06:54.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.074+0000 7f0b92c65700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0b8c0adb60 0x7f0b8c0ce190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:54.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.074+0000 7f0b92c65700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0b8c0ce6d0 0x7f0b8c15aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:54.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.074+0000 7f0b92c65700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b8c157de0 con 0x7f0b8c0ca370 2023-10-06T05:06:54.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.074+0000 7f0b92c65700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b8c0cbb90 con 0x7f0b8c0adb60 2023-10-06T05:06:54.080 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.074+0000 7f0b92c65700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b8c0ceb10 con 0x7f0b8c0ce6d0 2023-10-06T05:06:54.080 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.074+0000 7f0b83fff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0b8c0adb60 0x7f0b8c0ce190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:54.080 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.074+0000 7f0b83fff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0b8c0adb60 0x7f0b8c0ce190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54156/0 (socket says 172.21.15.98:54156) 2023-10-06T05:06:54.080 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.074+0000 7f0b83fff700 1 -- 172.21.15.98:0/1971848992 learned_addr learned my addr 172.21.15.98:0/1971848992 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:06:54.080 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.074+0000 7f0b91202700 1 --2- 172.21.15.98:0/1971848992 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0b8c0ce6d0 0x7f0b8c15aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:54.080 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.075+0000 7f0b83fff700 1 -- 172.21.15.98:0/1971848992 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0b8c0ce6d0 msgr2=0x7f0b8c15aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:06:54.080 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.075+0000 7f0b83fff700 1 --2- 172.21.15.98:0/1971848992 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0b8c0ce6d0 0x7f0b8c15aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:54.080 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.075+0000 7f0b83fff700 1 -- 172.21.15.98:0/1971848992 >> v1:172.21.15.98:6789/0 conn(0x7f0b8c0ca370 legacy=0x7f0b8c158270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:54.081 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.075+0000 7f0b83fff700 1 -- 172.21.15.98:0/1971848992 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b8c0cb5d0 con 0x7f0b8c0adb60 2023-10-06T05:06:54.081 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.075+0000 7f0b837fe700 1 -- 172.21.15.98:0/1971848992 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1593498129 0 0) 0x7f0b8c157de0 con 0x7f0b8c0ca370 2023-10-06T05:06:54.081 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.075+0000 7f0b83fff700 1 --2- 172.21.15.98:0/1971848992 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b8c0adb60 0x7f0b8c0ce190 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0b7c0099b0 tx=0x7f0b7c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b21e2780eb0325a server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:06:54.081 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.075+0000 7f0b837fe700 1 -- 172.21.15.98:0/1971848992 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b7c00b860 con 0x7f0b8c0adb60 2023-10-06T05:06:54.081 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.075+0000 7f0b837fe700 1 -- 172.21.15.98:0/1971848992 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b7c00b9c0 con 0x7f0b8c0adb60 2023-10-06T05:06:54.081 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.075+0000 7f0b837fe700 1 -- 172.21.15.98:0/1971848992 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b7c00bd60 con 0x7f0b8c0adb60 2023-10-06T05:06:54.081 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.075+0000 7f0b92c65700 1 -- 172.21.15.98:0/1971848992 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b8c0adb60 msgr2=0x7f0b8c0ce190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:54.081 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.075+0000 7f0b92c65700 1 --2- 172.21.15.98:0/1971848992 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b8c0adb60 0x7f0b8c0ce190 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0b7c0099b0 tx=0x7f0b7c009250 comp rx=0 tx=0).stop 2023-10-06T05:06:54.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.075+0000 7f0b92c65700 1 -- 172.21.15.98:0/1971848992 shutdown_connections 2023-10-06T05:06:54.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.076+0000 7f0b92c65700 1 --2- 172.21.15.98:0/1971848992 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0b8c0ce6d0 0x7f0b8c15aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:54.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.076+0000 7f0b92c65700 1 --2- 172.21.15.98:0/1971848992 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b8c0adb60 0x7f0b8c0ce190 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:54.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.076+0000 7f0b92c65700 1 -- 172.21.15.98:0/1971848992 >> 172.21.15.98:0/1971848992 conn(0x7f0b8c0707d0 msgr2=0x7f0b8c070bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:06:54.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.076+0000 7f0b92c65700 1 -- 172.21.15.98:0/1971848992 shutdown_connections 2023-10-06T05:06:54.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.076+0000 7f0b92c65700 1 -- 172.21.15.98:0/1971848992 wait complete. 2023-10-06T05:06:54.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.077+0000 7f0b92c65700 1 Processor -- start 2023-10-06T05:06:54.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.077+0000 7f0b92c65700 1 -- start start 2023-10-06T05:06:54.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.077+0000 7f0b92c65700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0b8c0adb60 0x7f0b8c14be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:54.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.078+0000 7f0b92c65700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b8c0ca370 0x7f0b8c0c96f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:54.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.078+0000 7f0b92c65700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0b8c0ce6d0 0x7f0b8c0c9c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:54.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.078+0000 7f0b92c65700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b8c155860 con 0x7f0b8c0ca370 2023-10-06T05:06:54.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.078+0000 7f0b92c65700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b8c16afa0 con 0x7f0b8c0ce6d0 2023-10-06T05:06:54.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.078+0000 7f0b92c65700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b8c14c3b0 con 0x7f0b8c0adb60 2023-10-06T05:06:54.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.078+0000 7f0b83fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b8c0ca370 0x7f0b8c0c96f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:54.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.078+0000 7f0b83fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b8c0ca370 0x7f0b8c0c96f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54170/0 (socket says 172.21.15.98:54170) 2023-10-06T05:06:54.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.078+0000 7f0b83fff700 1 -- 172.21.15.98:0/829958544 learned_addr learned my addr 172.21.15.98:0/829958544 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:06:54.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.078+0000 7f0b90a01700 1 --2- 172.21.15.98:0/829958544 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0b8c0adb60 0x7f0b8c14be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:54.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.078+0000 7f0b91202700 1 --2- 172.21.15.98:0/829958544 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0b8c0ce6d0 0x7f0b8c0c9c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:54.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.078+0000 7f0b83fff700 1 -- 172.21.15.98:0/829958544 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0b8c0adb60 msgr2=0x7f0b8c14be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:54.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.078+0000 7f0b83fff700 1 --2- 172.21.15.98:0/829958544 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0b8c0adb60 0x7f0b8c14be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:54.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.078+0000 7f0b83fff700 1 -- 172.21.15.98:0/829958544 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0b8c0ce6d0 msgr2=0x7f0b8c0c9c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:54.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.078+0000 7f0b83fff700 1 --2- 172.21.15.98:0/829958544 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0b8c0ce6d0 0x7f0b8c0c9c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:54.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.078+0000 7f0b83fff700 1 -- 172.21.15.98:0/829958544 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b7c0094f0 con 0x7f0b8c0ca370 2023-10-06T05:06:54.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.078+0000 7f0b91202700 1 --2- 172.21.15.98:0/829958544 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0b8c0ce6d0 0x7f0b8c0c9c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:06:54.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.078+0000 7f0b83fff700 1 --2- 172.21.15.98:0/829958544 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b8c0ca370 0x7f0b8c0c96f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c0cdfa0 tx=0x7f0b7c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:06:54.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.078+0000 7f0b80ff9700 1 -- 172.21.15.98:0/829958544 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b7c024440 con 0x7f0b8c0ca370 2023-10-06T05:06:54.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.078+0000 7f0b80ff9700 1 -- 172.21.15.98:0/829958544 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b7c0245a0 con 0x7f0b8c0ca370 2023-10-06T05:06:54.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.078+0000 7f0b80ff9700 1 -- 172.21.15.98:0/829958544 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b7c024960 con 0x7f0b8c0ca370 2023-10-06T05:06:54.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.078+0000 7f0b92c65700 1 -- 172.21.15.98:0/829958544 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b8c16acc0 con 0x7f0b8c0ca370 2023-10-06T05:06:54.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.078+0000 7f0b92c65700 1 -- 172.21.15.98:0/829958544 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b8c16b5b0 con 0x7f0b8c0ca370 2023-10-06T05:06:54.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.079+0000 7f0b92c65700 1 -- 172.21.15.98:0/829958544 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b8c157720 con 0x7f0b8c0ca370 2023-10-06T05:06:54.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.079+0000 7f0b80ff9700 1 -- 172.21.15.98:0/829958544 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f0b7c024d50 con 0x7f0b8c0ca370 2023-10-06T05:06:54.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.080+0000 7f0b80ff9700 1 --2- 172.21.15.98:0/829958544 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f0b78066f30 0x7f0b780693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:54.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.080+0000 7f0b80ff9700 1 -- 172.21.15.98:0/829958544 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f0b7c085d80 con 0x7f0b8c0ca370 2023-10-06T05:06:54.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.080+0000 7f0b90a01700 1 --2- 172.21.15.98:0/829958544 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f0b78066f30 0x7f0b780693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:54.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.080+0000 7f0b90a01700 1 --2- 172.21.15.98:0/829958544 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f0b78066f30 0x7f0b780693e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c15a890 tx=0x7f0b88009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:06:54.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.084+0000 7f0b80ff9700 1 -- 172.21.15.98:0/829958544 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0b7c053320 con 0x7f0b8c0ca370 2023-10-06T05:06:54.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.268+0000 7f0b92c65700 1 -- 172.21.15.98:0/829958544 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0b8c064a90 con 0x7f0b8c0ca370 2023-10-06T05:06:54.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.269+0000 7f0b80ff9700 1 -- 172.21.15.98:0/829958544 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+13647 (secure 0 0 0) 0x7f0b7c02b340 con 0x7f0b8c0ca370 2023-10-06T05:06:54.271 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:06:54.272 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":21,"fsid":"f50c5df5-cad5-47a4-9bb1-8c791d4311ec","created":"2023-10-06T05:06:32.971175+0000","modified":"2023-10-06T05:06:53.580888+0000","last_up_change":"2023-10-06T05:06:44.016489+0000","last_in_change":"2023-10-06T05:06:35.544511+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-06T05:06:44.559671+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-06T05:06:51.432161+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d9495d95-da75-4c3b-8927-e9bae6a8b093","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":90627},{"type":"v1","addr":"172.21.15.98:6801","nonce":90627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":90627},{"type":"v1","addr":"172.21.15.98:6803","nonce":90627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":90627},{"type":"v1","addr":"172.21.15.98:6807","nonce":90627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":90627},{"type":"v1","addr":"172.21.15.98:6805","nonce":90627}]},"public_addr":"172.21.15.98:6801/90627","cluster_addr":"172.21.15.98:6803/90627","heartbeat_back_addr":"172.21.15.98:6807/90627","heartbeat_front_addr":"172.21.15.98:6805/90627","state":["exists","up"]},{"osd":1,"uuid":"f25ba5be-d65e-4707-a134-154092deb754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":90628},{"type":"v1","addr":"172.21.15.98:6809","nonce":90628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":90628},{"type":"v1","addr":"172.21.15.98:6811","nonce":90628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":90628},{"type":"v1","addr":"172.21.15.98:6815","nonce":90628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":90628},{"type":"v1","addr":"172.21.15.98:6813","nonce":90628}]},"public_addr":"172.21.15.98:6809/90628","cluster_addr":"172.21.15.98:6811/90628","heartbeat_back_addr":"172.21.15.98:6815/90628","heartbeat_front_addr":"172.21.15.98:6813/90628","state":["exists","up"]},{"osd":2,"uuid":"5ab9e028-945c-43e2-8c95-3b0b4cf38e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":90662},{"type":"v1","addr":"172.21.15.98:6818","nonce":90662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":90662},{"type":"v1","addr":"172.21.15.98:6822","nonce":90662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":90662},{"type":"v1","addr":"172.21.15.98:6830","nonce":90662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":90662},{"type":"v1","addr":"172.21.15.98:6826","nonce":90662}]},"public_addr":"172.21.15.98:6818/90662","cluster_addr":"172.21.15.98:6822/90662","heartbeat_back_addr":"172.21.15.98:6830/90662","heartbeat_front_addr":"172.21.15.98:6826/90662","state":["exists","up"]},{"osd":3,"uuid":"3bb803ff-1738-46bf-be62-2267bf378822","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6817","nonce":90663},{"type":"v1","addr":"172.21.15.98:6819","nonce":90663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6821","nonce":90663},{"type":"v1","addr":"172.21.15.98:6823","nonce":90663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6829","nonce":90663},{"type":"v1","addr":"172.21.15.98:6831","nonce":90663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6825","nonce":90663},{"type":"v1","addr":"172.21.15.98:6827","nonce":90663}]},"public_addr":"172.21.15.98:6819/90663","cluster_addr":"172.21.15.98:6823/90663","heartbeat_back_addr":"172.21.15.98:6831/90663","heartbeat_front_addr":"172.21.15.98:6827/90663","state":["exists","up"]},{"osd":4,"uuid":"7d57a4c9-6462-43f6-b64a-7f9e9b4353d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":90847},{"type":"v1","addr":"172.21.15.174:6801","nonce":90847}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":90847},{"type":"v1","addr":"172.21.15.174:6803","nonce":90847}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":90847},{"type":"v1","addr":"172.21.15.174:6807","nonce":90847}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":90847},{"type":"v1","addr":"172.21.15.174:6805","nonce":90847}]},"public_addr":"172.21.15.174:6801/90847","cluster_addr":"172.21.15.174:6803/90847","heartbeat_back_addr":"172.21.15.174:6807/90847","heartbeat_front_addr":"172.21.15.174:6805/90847","state":["exists","up"]},{"osd":5,"uuid":"83287ea4-e44d-4af9-bc77-36d47c60d4a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":90848},{"type":"v1","addr":"172.21.15.174:6809","nonce":90848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":90848},{"type":"v1","addr":"172.21.15.174:6811","nonce":90848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":90848},{"type":"v1","addr":"172.21.15.174:6815","nonce":90848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":90848},{"type":"v1","addr":"172.21.15.174:6813","nonce":90848}]},"public_addr":"172.21.15.174:6809/90848","cluster_addr":"172.21.15.174:6811/90848","heartbeat_back_addr":"172.21.15.174:6815/90848","heartbeat_front_addr":"172.21.15.174:6813/90848","state":["exists","up"]},{"osd":6,"uuid":"496a6272-892e-43c4-a145-206623c22f99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":90877},{"type":"v1","addr":"172.21.15.174:6817","nonce":90877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":90877},{"type":"v1","addr":"172.21.15.174:6819","nonce":90877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":90877},{"type":"v1","addr":"172.21.15.174:6823","nonce":90877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":90877},{"type":"v1","addr":"172.21.15.174:6821","nonce":90877}]},"public_addr":"172.21.15.174:6817/90877","cluster_addr":"172.21.15.174:6819/90877","heartbeat_back_addr":"172.21.15.174:6823/90877","heartbeat_front_addr":"172.21.15.174:6821/90877","state":["exists","up"]},{"osd":7,"uuid":"49967e34-8b07-4bab-a13e-0c16d6a0f3a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":90878},{"type":"v1","addr":"172.21.15.174:6825","nonce":90878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":90878},{"type":"v1","addr":"172.21.15.174:6827","nonce":90878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":90878},{"type":"v1","addr":"172.21.15.174:6831","nonce":90878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":90878},{"type":"v1","addr":"172.21.15.174:6829","nonce":90878}]},"public_addr":"172.21.15.174:6825/90878","cluster_addr":"172.21.15.174:6827/90878","heartbeat_back_addr":"172.21.15.174:6831/90878","heartbeat_front_addr":"172.21.15.174:6829/90878","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.446479+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.514266+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.794108+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.810894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.731919+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.717756+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.737435+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.508682+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T05:06:54.274 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.271+0000 7f0b92c65700 1 -- 172.21.15.98:0/829958544 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f0b78066f30 msgr2=0x7f0b780693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:54.274 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.271+0000 7f0b92c65700 1 --2- 172.21.15.98:0/829958544 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f0b78066f30 0x7f0b780693e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c15a890 tx=0x7f0b88009040 comp rx=0 tx=0).stop 2023-10-06T05:06:54.274 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.271+0000 7f0b92c65700 1 -- 172.21.15.98:0/829958544 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b8c0ca370 msgr2=0x7f0b8c0c96f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:54.274 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.271+0000 7f0b92c65700 1 --2- 172.21.15.98:0/829958544 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b8c0ca370 0x7f0b8c0c96f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c0cdfa0 tx=0x7f0b7c017000 comp rx=0 tx=0).stop 2023-10-06T05:06:54.274 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.271+0000 7f0b92c65700 1 -- 172.21.15.98:0/829958544 shutdown_connections 2023-10-06T05:06:54.274 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.271+0000 7f0b92c65700 1 --2- 172.21.15.98:0/829958544 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0b8c0ce6d0 0x7f0b8c0c9c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:54.274 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.271+0000 7f0b92c65700 1 --2- 172.21.15.98:0/829958544 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0b8c0adb60 0x7f0b8c14be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:54.274 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.271+0000 7f0b92c65700 1 --2- 172.21.15.98:0/829958544 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f0b78066f30 0x7f0b780693e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:54.274 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.271+0000 7f0b92c65700 1 --2- 172.21.15.98:0/829958544 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b8c0ca370 0x7f0b8c0c96f0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:54.274 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.271+0000 7f0b92c65700 1 -- 172.21.15.98:0/829958544 >> 172.21.15.98:0/829958544 conn(0x7f0b8c0707d0 msgr2=0x7f0b8c14dc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:06:54.274 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.271+0000 7f0b92c65700 1 -- 172.21.15.98:0/829958544 shutdown_connections 2023-10-06T05:06:54.275 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.271+0000 7f0b92c65700 1 -- 172.21.15.98:0/829958544 wait complete. 2023-10-06T05:06:54.286 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-06T05:06:44.559671+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-10-06T05:06:51.432161+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2023-10-06T05:06:54.287 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-06T05:06:54.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.401+0000 7f32353b4700 1 Processor -- start 2023-10-06T05:06:54.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.401+0000 7f32353b4700 1 -- start start 2023-10-06T05:06:54.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.402+0000 7f32353b4700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3230153970 0x7f32300b1120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:54.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.402+0000 7f32353b4700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f32300b07c0 0x7f32300b1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:54.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.402+0000 7f32353b4700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f323015a600 con 0x7f32300af180 2023-10-06T05:06:54.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.402+0000 7f32353b4700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3230157600 con 0x7f3230153970 2023-10-06T05:06:54.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.402+0000 7f32353b4700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32300b0cd0 con 0x7f32300b07c0 2023-10-06T05:06:54.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.402+0000 7f322ffff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f3230153970 0x7f32300b1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:54.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.402+0000 7f3234bb3700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f32300af180 0x7f323006a290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:49374/0 (socket says 172.21.15.98:49374) 2023-10-06T05:06:54.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.402+0000 7f3234bb3700 1 -- 172.21.15.98:0/3020492359 learned_addr learned my addr 172.21.15.98:0/3020492359 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:06:54.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.402+0000 7f322f7fe700 1 --2- 172.21.15.98:0/3020492359 >> v2:172.21.15.174:3300/0 conn(0x7f32300b07c0 0x7f32300b1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:54.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.402+0000 7f322ffff700 1 -- 172.21.15.98:0/3020492359 >> v2:172.21.15.174:3300/0 conn(0x7f32300b07c0 msgr2=0x7f32300b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:06:54.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.402+0000 7f322ffff700 1 --2- 172.21.15.98:0/3020492359 >> v2:172.21.15.174:3300/0 conn(0x7f32300b07c0 0x7f32300b1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:54.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.402+0000 7f322ffff700 1 -- 172.21.15.98:0/3020492359 >> v1:172.21.15.98:6789/0 conn(0x7f32300af180 legacy=0x7f323006a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:54.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.402+0000 7f322ffff700 1 -- 172.21.15.98:0/3020492359 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3230159cd0 con 0x7f3230153970 2023-10-06T05:06:54.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.402+0000 7f322effd700 1 -- 172.21.15.98:0/3020492359 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4283928081 0 0) 0x7f323015a600 con 0x7f32300af180 2023-10-06T05:06:54.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.402+0000 7f322f7fe700 1 --2- 172.21.15.98:0/3020492359 >> v2:172.21.15.174:3300/0 conn(0x7f32300b07c0 0x7f32300b1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:06:54.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.402+0000 7f322ffff700 1 --2- 172.21.15.98:0/3020492359 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3230153970 0x7f32300b1120 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f323006d9b0 tx=0x7f322000b430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28f5ede72fd4a716 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:06:54.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.402+0000 7f322effd700 1 -- 172.21.15.98:0/3020492359 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f322000db70 con 0x7f3230153970 2023-10-06T05:06:54.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.403+0000 7f322effd700 1 -- 172.21.15.98:0/3020492359 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f322000dcd0 con 0x7f3230153970 2023-10-06T05:06:54.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.403+0000 7f322effd700 1 -- 172.21.15.98:0/3020492359 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32200196b0 con 0x7f3230153970 2023-10-06T05:06:54.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.403+0000 7f32353b4700 1 -- 172.21.15.98:0/3020492359 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3230153970 msgr2=0x7f32300b1120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:54.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.403+0000 7f32353b4700 1 --2- 172.21.15.98:0/3020492359 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3230153970 0x7f32300b1120 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f323006d9b0 tx=0x7f322000b430 comp rx=0 tx=0).stop 2023-10-06T05:06:54.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.403+0000 7f32353b4700 1 -- 172.21.15.98:0/3020492359 shutdown_connections 2023-10-06T05:06:54.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.403+0000 7f32353b4700 1 --2- 172.21.15.98:0/3020492359 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3230153970 0x7f32300b1120 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:54.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.403+0000 7f32353b4700 1 --2- 172.21.15.98:0/3020492359 >> v2:172.21.15.174:3300/0 conn(0x7f32300b07c0 0x7f32300b1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:54.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.403+0000 7f32353b4700 1 -- 172.21.15.98:0/3020492359 >> 172.21.15.98:0/3020492359 conn(0x7f3230069140 msgr2=0x7f32300aea70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:06:54.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.403+0000 7f32353b4700 1 -- 172.21.15.98:0/3020492359 shutdown_connections 2023-10-06T05:06:54.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.403+0000 7f32353b4700 1 -- 172.21.15.98:0/3020492359 wait complete. 2023-10-06T05:06:54.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.404+0000 7f32353b4700 1 Processor -- start 2023-10-06T05:06:54.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.404+0000 7f32353b4700 1 -- start start 2023-10-06T05:06:54.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.405+0000 7f32353b4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32300af180 0x7f3230157ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:54.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.405+0000 7f32353b4700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f32300b07c0 0x7f32300cc0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:54.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.405+0000 7f32353b4700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3230153970 0x7f32300cc630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:54.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.405+0000 7f32353b4700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32300ccb70 con 0x7f32300af180 2023-10-06T05:06:54.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.405+0000 7f32353b4700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32301689b0 con 0x7f32300b07c0 2023-10-06T05:06:54.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.405+0000 7f32353b4700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3230168c90 con 0x7f3230153970 2023-10-06T05:06:54.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.405+0000 7f322ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32300af180 0x7f3230157ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:54.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.405+0000 7f322ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32300af180 0x7f3230157ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54176/0 (socket says 172.21.15.98:54176) 2023-10-06T05:06:54.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.405+0000 7f322ffff700 1 -- 172.21.15.98:0/3731226015 learned_addr learned my addr 172.21.15.98:0/3731226015 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:06:54.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.405+0000 7f322ffff700 1 -- 172.21.15.98:0/3731226015 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3230153970 msgr2=0x7f32300cc630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:54.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.405+0000 7f322ffff700 1 --2- 172.21.15.98:0/3731226015 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3230153970 0x7f32300cc630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:54.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.405+0000 7f322ffff700 1 -- 172.21.15.98:0/3731226015 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f32300b07c0 msgr2=0x7f32300cc0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:54.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.405+0000 7f322ffff700 1 --2- 172.21.15.98:0/3731226015 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f32300b07c0 0x7f32300cc0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:54.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.405+0000 7f322ffff700 1 -- 172.21.15.98:0/3731226015 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f322000b6d0 con 0x7f32300af180 2023-10-06T05:06:54.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.405+0000 7f322ffff700 1 --2- 172.21.15.98:0/3731226015 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32300af180 0x7f3230157ff0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f322000b470 tx=0x7f322001b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:06:54.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.406+0000 7f3217fff700 1 -- 172.21.15.98:0/3731226015 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f322001d050 con 0x7f32300af180 2023-10-06T05:06:54.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.406+0000 7f3217fff700 1 -- 172.21.15.98:0/3731226015 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f322001e040 con 0x7f32300af180 2023-10-06T05:06:54.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.406+0000 7f32353b4700 1 -- 172.21.15.98:0/3731226015 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3230168f70 con 0x7f32300af180 2023-10-06T05:06:54.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.406+0000 7f3217fff700 1 -- 172.21.15.98:0/3731226015 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3220010ca0 con 0x7f32300af180 2023-10-06T05:06:54.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.406+0000 7f32353b4700 1 -- 172.21.15.98:0/3731226015 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3230169530 con 0x7f32300af180 2023-10-06T05:06:54.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.407+0000 7f32353b4700 1 -- 172.21.15.98:0/3731226015 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f323014d750 con 0x7f32300af180 2023-10-06T05:06:54.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.407+0000 7f3217fff700 1 -- 172.21.15.98:0/3731226015 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f3220024620 con 0x7f32300af180 2023-10-06T05:06:54.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.407+0000 7f3217fff700 1 --2- 172.21.15.98:0/3731226015 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f321c066f30 0x7f321c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:54.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.407+0000 7f3217fff700 1 -- 172.21.15.98:0/3731226015 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f3220014070 con 0x7f32300af180 2023-10-06T05:06:54.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.407+0000 7f322f7fe700 1 --2- 172.21.15.98:0/3731226015 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f321c066f30 0x7f321c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:54.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.408+0000 7f322f7fe700 1 --2- 172.21.15.98:0/3731226015 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f321c066f30 0x7f321c0693e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3224007aa0 tx=0x7f3224008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:06:54.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.412+0000 7f3217fff700 1 -- 172.21.15.98:0/3731226015 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f32200563a0 con 0x7f32300af180 2023-10-06T05:06:54.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.595+0000 7f32353b4700 1 -- 172.21.15.98:0/3731226015 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f323005f230 con 0x7f32300af180 2023-10-06T05:06:54.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.595+0000 7f3217fff700 1 -- 172.21.15.98:0/3731226015 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f32200599c0 con 0x7f32300af180 2023-10-06T05:06:54.597 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2023-10-06T05:06:54.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.597+0000 7f32353b4700 1 -- 172.21.15.98:0/3731226015 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f321c066f30 msgr2=0x7f321c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:54.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.597+0000 7f32353b4700 1 --2- 172.21.15.98:0/3731226015 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f321c066f30 0x7f321c0693e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3224007aa0 tx=0x7f3224008040 comp rx=0 tx=0).stop 2023-10-06T05:06:54.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.597+0000 7f32353b4700 1 -- 172.21.15.98:0/3731226015 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32300af180 msgr2=0x7f3230157ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:54.600 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.597+0000 7f32353b4700 1 --2- 172.21.15.98:0/3731226015 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32300af180 0x7f3230157ff0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f322000b470 tx=0x7f322001b000 comp rx=0 tx=0).stop 2023-10-06T05:06:54.600 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.597+0000 7f32353b4700 1 -- 172.21.15.98:0/3731226015 shutdown_connections 2023-10-06T05:06:54.600 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.597+0000 7f32353b4700 1 --2- 172.21.15.98:0/3731226015 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3230153970 0x7f32300cc630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:54.600 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.597+0000 7f32353b4700 1 --2- 172.21.15.98:0/3731226015 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f321c066f30 0x7f321c0693e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:54.600 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.597+0000 7f32353b4700 1 --2- 172.21.15.98:0/3731226015 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f32300af180 0x7f3230157ff0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:54.600 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.597+0000 7f32353b4700 1 --2- 172.21.15.98:0/3731226015 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f32300b07c0 0x7f32300cc0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:54.600 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.597+0000 7f32353b4700 1 -- 172.21.15.98:0/3731226015 >> 172.21.15.98:0/3731226015 conn(0x7f3230069140 msgr2=0x7f32300ae8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:06:54.600 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.597+0000 7f32353b4700 1 -- 172.21.15.98:0/3731226015 shutdown_connections 2023-10-06T05:06:54.600 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.597+0000 7f32353b4700 1 -- 172.21.15.98:0/3731226015 wait complete. 2023-10-06T05:06:54.611 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-10-06T05:06:54.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.727+0000 7f64e3c15700 1 Processor -- start 2023-10-06T05:06:54.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.727+0000 7f64e3c15700 1 -- start start 2023-10-06T05:06:54.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.728+0000 7f64e3c15700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f64dc1594d0 0x7f64dc0cb560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:54.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.728+0000 7f64e3c15700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64dc0cbee0 con 0x7f64dc0cbaa0 2023-10-06T05:06:54.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.728+0000 7f64e3c15700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64dc0cf5d0 con 0x7f64dc1598a0 2023-10-06T05:06:54.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.728+0000 7f64e3c15700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64dc0cf8b0 con 0x7f64dc1594d0 2023-10-06T05:06:54.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.728+0000 7f64e19b1700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f64dc1594d0 0x7f64dc0cb560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:54.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.728+0000 7f64e21b2700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f64dc0cbaa0 0x7f64dc0c5690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:49378/0 (socket says 172.21.15.98:49378) 2023-10-06T05:06:54.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.728+0000 7f64e21b2700 1 -- 172.21.15.98:0/3462993657 learned_addr learned my addr 172.21.15.98:0/3462993657 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:06:54.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.728+0000 7f64e19b1700 1 -- 172.21.15.98:0/3462993657 >> v1:172.21.15.174:6789/0 conn(0x7f64dc1598a0 legacy=0x7f64dc0ce7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:06:54.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.728+0000 7f64e19b1700 1 -- 172.21.15.98:0/3462993657 >> v1:172.21.15.98:6789/0 conn(0x7f64dc0cbaa0 legacy=0x7f64dc0c5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:54.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.728+0000 7f64e19b1700 1 -- 172.21.15.98:0/3462993657 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64dc0cfb90 con 0x7f64dc1594d0 2023-10-06T05:06:54.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.728+0000 7f64e09af700 1 -- 172.21.15.98:0/3462993657 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1583426176 0 0) 0x7f64dc0cbee0 con 0x7f64dc0cbaa0 2023-10-06T05:06:54.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.728+0000 7f64e19b1700 1 --2- 172.21.15.98:0/3462993657 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64dc1594d0 0x7f64dc0cb560 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f64d40098e0 tx=0x7f64d40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b5c88e8a2d2f47 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:06:54.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.729+0000 7f64e09af700 1 -- 172.21.15.98:0/3462993657 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64d4009e80 con 0x7f64dc1594d0 2023-10-06T05:06:54.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.729+0000 7f64e09af700 1 -- 172.21.15.98:0/3462993657 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64d400b820 con 0x7f64dc1594d0 2023-10-06T05:06:54.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.729+0000 7f64e09af700 1 -- 172.21.15.98:0/3462993657 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64d400bba0 con 0x7f64dc1594d0 2023-10-06T05:06:54.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.729+0000 7f64e3c15700 1 -- 172.21.15.98:0/3462993657 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64dc1594d0 msgr2=0x7f64dc0cb560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:54.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.729+0000 7f64e3c15700 1 --2- 172.21.15.98:0/3462993657 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64dc1594d0 0x7f64dc0cb560 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f64d40098e0 tx=0x7f64d40091a0 comp rx=0 tx=0).stop 2023-10-06T05:06:54.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.729+0000 7f64e3c15700 1 -- 172.21.15.98:0/3462993657 shutdown_connections 2023-10-06T05:06:54.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.729+0000 7f64e3c15700 1 --2- 172.21.15.98:0/3462993657 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64dc1594d0 0x7f64dc0cb560 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:54.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.729+0000 7f64e3c15700 1 -- 172.21.15.98:0/3462993657 >> 172.21.15.98:0/3462993657 conn(0x7f64dc089e50 msgr2=0x7f64dc099770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:06:54.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.729+0000 7f64e3c15700 1 -- 172.21.15.98:0/3462993657 shutdown_connections 2023-10-06T05:06:54.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.729+0000 7f64e3c15700 1 -- 172.21.15.98:0/3462993657 wait complete. 2023-10-06T05:06:54.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.730+0000 7f64e3c15700 1 Processor -- start 2023-10-06T05:06:54.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.730+0000 7f64e3c15700 1 -- start start 2023-10-06T05:06:54.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.731+0000 7f64e3c15700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f64dc0cbaa0 0x7f64dc0c8b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:54.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.731+0000 7f64e3c15700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f64dc1594d0 0x7f64dc16aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:54.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.731+0000 7f64e3c15700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64dc1598a0 0x7f64dc0b5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:54.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.731+0000 7f64e3c15700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64dc16b020 con 0x7f64dc1598a0 2023-10-06T05:06:54.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.731+0000 7f64e3c15700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64dc0b4e10 con 0x7f64dc1594d0 2023-10-06T05:06:54.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.731+0000 7f64e3c15700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64dc0b50b0 con 0x7f64dc0cbaa0 2023-10-06T05:06:54.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.731+0000 7f64e21b2700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64dc1598a0 0x7f64dc0b5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:54.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.731+0000 7f64e21b2700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64dc1598a0 0x7f64dc0b5b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54204/0 (socket says 172.21.15.98:54204) 2023-10-06T05:06:54.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.731+0000 7f64e21b2700 1 -- 172.21.15.98:0/1770055986 learned_addr learned my addr 172.21.15.98:0/1770055986 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:06:54.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.731+0000 7f64e19b1700 1 --2- 172.21.15.98:0/1770055986 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f64dc0cbaa0 0x7f64dc0c8b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:54.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.731+0000 7f64e11b0700 1 --2- 172.21.15.98:0/1770055986 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f64dc1594d0 0x7f64dc16aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:54.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.731+0000 7f64e21b2700 1 -- 172.21.15.98:0/1770055986 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f64dc0cbaa0 msgr2=0x7f64dc0c8b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:54.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.731+0000 7f64e21b2700 1 --2- 172.21.15.98:0/1770055986 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f64dc0cbaa0 0x7f64dc0c8b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:54.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.731+0000 7f64e21b2700 1 -- 172.21.15.98:0/1770055986 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f64dc1594d0 msgr2=0x7f64dc16aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:54.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.731+0000 7f64e21b2700 1 --2- 172.21.15.98:0/1770055986 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f64dc1594d0 0x7f64dc16aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:54.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.731+0000 7f64e21b2700 1 -- 172.21.15.98:0/1770055986 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64d4009420 con 0x7f64dc1598a0 2023-10-06T05:06:54.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.732+0000 7f64e21b2700 1 --2- 172.21.15.98:0/1770055986 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64dc1598a0 0x7f64dc0b5b50 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f64d800cc90 tx=0x7f64d8001a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:06:54.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.732+0000 7f64d1ffb700 1 -- 172.21.15.98:0/1770055986 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64d800cea0 con 0x7f64dc1598a0 2023-10-06T05:06:54.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.732+0000 7f64d1ffb700 1 -- 172.21.15.98:0/1770055986 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64d800ab70 con 0x7f64dc1598a0 2023-10-06T05:06:54.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.732+0000 7f64e3c15700 1 -- 172.21.15.98:0/1770055986 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64dc0b3970 con 0x7f64dc1598a0 2023-10-06T05:06:54.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.732+0000 7f64d1ffb700 1 -- 172.21.15.98:0/1770055986 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64d80070f0 con 0x7f64dc1598a0 2023-10-06T05:06:54.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.732+0000 7f64e3c15700 1 -- 172.21.15.98:0/1770055986 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64dc0b5350 con 0x7f64dc1598a0 2023-10-06T05:06:54.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.733+0000 7f64e3c15700 1 -- 172.21.15.98:0/1770055986 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64dc064a90 con 0x7f64dc1598a0 2023-10-06T05:06:54.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.733+0000 7f64d1ffb700 1 -- 172.21.15.98:0/1770055986 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f64d80074a0 con 0x7f64dc1598a0 2023-10-06T05:06:54.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.733+0000 7f64d1ffb700 1 --2- 172.21.15.98:0/1770055986 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f64c0066f30 0x7f64c00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:54.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.734+0000 7f64d1ffb700 1 -- 172.21.15.98:0/1770055986 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f64d80847c0 con 0x7f64dc1598a0 2023-10-06T05:06:54.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.734+0000 7f64e19b1700 1 --2- 172.21.15.98:0/1770055986 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f64c0066f30 0x7f64c00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:54.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.734+0000 7f64e19b1700 1 --2- 172.21.15.98:0/1770055986 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f64c0066f30 0x7f64c00693e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f64d40030c0 tx=0x7f64d4016000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:06:54.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.738+0000 7f64d1ffb700 1 -- 172.21.15.98:0/1770055986 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f64d8051ca0 con 0x7f64dc1598a0 2023-10-06T05:06:54.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.921+0000 7f64e3c15700 1 -- 172.21.15.98:0/1770055986 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f64dc0cfe70 con 0x7f64dc1598a0 2023-10-06T05:06:54.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.922+0000 7f64d1ffb700 1 -- 172.21.15.98:0/1770055986 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v21) v1 ==== 92+0+10 (secure 0 0 0) 0x7f64d80552c0 con 0x7f64dc1598a0 2023-10-06T05:06:54.924 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 8 2023-10-06T05:06:54.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.924+0000 7f64e3c15700 1 -- 172.21.15.98:0/1770055986 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f64c0066f30 msgr2=0x7f64c00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:54.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.924+0000 7f64e3c15700 1 --2- 172.21.15.98:0/1770055986 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f64c0066f30 0x7f64c00693e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f64d40030c0 tx=0x7f64d4016000 comp rx=0 tx=0).stop 2023-10-06T05:06:54.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.924+0000 7f64e3c15700 1 -- 172.21.15.98:0/1770055986 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64dc1598a0 msgr2=0x7f64dc0b5b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:54.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.924+0000 7f64e3c15700 1 --2- 172.21.15.98:0/1770055986 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64dc1598a0 0x7f64dc0b5b50 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f64d800cc90 tx=0x7f64d8001a80 comp rx=0 tx=0).stop 2023-10-06T05:06:54.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.924+0000 7f64e3c15700 1 -- 172.21.15.98:0/1770055986 shutdown_connections 2023-10-06T05:06:54.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.924+0000 7f64e3c15700 1 --2- 172.21.15.98:0/1770055986 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f64c0066f30 0x7f64c00693e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:54.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.924+0000 7f64e3c15700 1 --2- 172.21.15.98:0/1770055986 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64dc1598a0 0x7f64dc0b5b50 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:54.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.924+0000 7f64e3c15700 1 --2- 172.21.15.98:0/1770055986 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f64dc0cbaa0 0x7f64dc0c8b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:54.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.924+0000 7f64e3c15700 1 --2- 172.21.15.98:0/1770055986 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f64dc1594d0 0x7f64dc16aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:54.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.924+0000 7f64e3c15700 1 -- 172.21.15.98:0/1770055986 >> 172.21.15.98:0/1770055986 conn(0x7f64dc089e50 msgr2=0x7f64dc068370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:06:54.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.924+0000 7f64e3c15700 1 -- 172.21.15.98:0/1770055986 shutdown_connections 2023-10-06T05:06:54.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:54.924+0000 7f64e3c15700 1 -- 172.21.15.98:0/1770055986 wait complete. 2023-10-06T05:06:54.938 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-06T05:06:55.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.050+0000 7f7eb3933700 1 Processor -- start 2023-10-06T05:06:55.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.050+0000 7f7eb3933700 1 -- start start 2023-10-06T05:06:55.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.051+0000 7f7eb3933700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f7eac15cab0 0x7f7eac06f160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:55.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.051+0000 7f7eb3933700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7eac0caa90 0x7f7eac0cafb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:55.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.051+0000 7f7eb3933700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7eac0675b0 0x7f7eac0679d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:55.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.051+0000 7f7eb3933700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7eac06ecd0 con 0x7f7eac0675b0 2023-10-06T05:06:55.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.051+0000 7f7eb3933700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7eac067f10 con 0x7f7eac15cab0 2023-10-06T05:06:55.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.051+0000 7f7eb3933700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7eac0cb870 con 0x7f7eac0caa90 2023-10-06T05:06:55.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.051+0000 7f7eb1ed0700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7eac0675b0 0x7f7eac0679d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:55.056 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.051+0000 7f7eb1ed0700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7eac0675b0 0x7f7eac0679d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54216/0 (socket says 172.21.15.98:54216) 2023-10-06T05:06:55.056 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.051+0000 7f7eb1ed0700 1 -- 172.21.15.98:0/4206143385 learned_addr learned my addr 172.21.15.98:0/4206143385 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:06:55.056 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.052+0000 7f7eb0ece700 1 --2- 172.21.15.98:0/4206143385 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7eac0caa90 0x7f7eac0cafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:55.056 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.052+0000 7f7eb16cf700 1 --2- 172.21.15.98:0/4206143385 >> v2:172.21.15.174:3300/0 conn(0x7f7eac15cab0 0x7f7eac06f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:55.056 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.052+0000 7f7eb1ed0700 1 -- 172.21.15.98:0/4206143385 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7eac0caa90 msgr2=0x7f7eac0cafb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:06:55.056 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.052+0000 7f7eb1ed0700 1 --2- 172.21.15.98:0/4206143385 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7eac0caa90 0x7f7eac0cafb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:55.056 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.052+0000 7f7eb1ed0700 1 -- 172.21.15.98:0/4206143385 >> v2:172.21.15.174:3300/0 conn(0x7f7eac15cab0 msgr2=0x7f7eac06f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:06:55.056 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.052+0000 7f7eb1ed0700 1 --2- 172.21.15.98:0/4206143385 >> v2:172.21.15.174:3300/0 conn(0x7f7eac15cab0 0x7f7eac06f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:55.056 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.052+0000 7f7eb1ed0700 1 -- 172.21.15.98:0/4206143385 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eac0cbb50 con 0x7f7eac0675b0 2023-10-06T05:06:55.056 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.052+0000 7f7eb0ece700 1 --2- 172.21.15.98:0/4206143385 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7eac0caa90 0x7f7eac0cafb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:06:55.056 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.052+0000 7f7eb16cf700 1 --2- 172.21.15.98:0/4206143385 >> v2:172.21.15.174:3300/0 conn(0x7f7eac15cab0 0x7f7eac06f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:06:55.057 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.052+0000 7f7eb1ed0700 1 --2- 172.21.15.98:0/4206143385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7eac0675b0 0x7f7eac0679d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7eac06e480 tx=0x7f7ea4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e5b02a598aa793f server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:06:55.057 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.052+0000 7f7ea3fff700 1 -- 172.21.15.98:0/4206143385 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ea400e070 con 0x7f7eac0675b0 2023-10-06T05:06:55.057 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.052+0000 7f7ea3fff700 1 -- 172.21.15.98:0/4206143385 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ea400ba50 con 0x7f7eac0675b0 2023-10-06T05:06:55.057 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.052+0000 7f7ea3fff700 1 -- 172.21.15.98:0/4206143385 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ea40092e0 con 0x7f7eac0675b0 2023-10-06T05:06:55.057 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.052+0000 7f7eb3933700 1 -- 172.21.15.98:0/4206143385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7eac0675b0 msgr2=0x7f7eac0679d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:55.057 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.052+0000 7f7eb3933700 1 --2- 172.21.15.98:0/4206143385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7eac0675b0 0x7f7eac0679d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7eac06e480 tx=0x7f7ea4009840 comp rx=0 tx=0).stop 2023-10-06T05:06:55.057 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.053+0000 7f7eb3933700 1 -- 172.21.15.98:0/4206143385 shutdown_connections 2023-10-06T05:06:55.057 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.053+0000 7f7eb3933700 1 --2- 172.21.15.98:0/4206143385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7eac0675b0 0x7f7eac0679d0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:55.057 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.053+0000 7f7eb3933700 1 --2- 172.21.15.98:0/4206143385 >> v2:172.21.15.174:3300/0 conn(0x7f7eac15cab0 0x7f7eac06f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:55.057 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.053+0000 7f7eb3933700 1 --2- 172.21.15.98:0/4206143385 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7eac0caa90 0x7f7eac0cafb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:55.057 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.053+0000 7f7eb3933700 1 -- 172.21.15.98:0/4206143385 >> 172.21.15.98:0/4206143385 conn(0x7f7eac0cdeb0 msgr2=0x7f7eac0c9e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:06:55.058 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.053+0000 7f7eb3933700 1 -- 172.21.15.98:0/4206143385 shutdown_connections 2023-10-06T05:06:55.058 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.053+0000 7f7eb3933700 1 -- 172.21.15.98:0/4206143385 wait complete. 2023-10-06T05:06:55.061 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.054+0000 7f7eb3933700 1 Processor -- start 2023-10-06T05:06:55.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.054+0000 7f7eb3933700 1 -- start start 2023-10-06T05:06:55.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.055+0000 7f7eb3933700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7eac0675b0 0x7f7eac0b2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:55.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.055+0000 7f7eb3933700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7eac0caa90 0x7f7eac0a9d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:55.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.055+0000 7f7eb3933700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7eac15cab0 0x7f7eac0aa260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:55.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.055+0000 7f7eb3933700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eac0ab4a0 con 0x7f7eac0675b0 2023-10-06T05:06:55.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.055+0000 7f7eb3933700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eac0b30b0 con 0x7f7eac0caa90 2023-10-06T05:06:55.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.055+0000 7f7eb3933700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7eac156180 con 0x7f7eac15cab0 2023-10-06T05:06:55.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.055+0000 7f7eb16cf700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7eac0675b0 0x7f7eac0b2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:55.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.055+0000 7f7eb16cf700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7eac0675b0 0x7f7eac0b2b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54218/0 (socket says 172.21.15.98:54218) 2023-10-06T05:06:55.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.055+0000 7f7eb16cf700 1 -- 172.21.15.98:0/2909017265 learned_addr learned my addr 172.21.15.98:0/2909017265 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:06:55.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.055+0000 7f7eb0ece700 1 --2- 172.21.15.98:0/2909017265 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7eac0caa90 0x7f7eac0a9d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:55.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.055+0000 7f7eb16cf700 1 -- 172.21.15.98:0/2909017265 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7eac15cab0 msgr2=0x7f7eac0aa260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:55.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.055+0000 7f7eb16cf700 1 --2- 172.21.15.98:0/2909017265 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7eac15cab0 0x7f7eac0aa260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:55.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.055+0000 7f7eb16cf700 1 -- 172.21.15.98:0/2909017265 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7eac0caa90 msgr2=0x7f7eac0a9d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:55.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.055+0000 7f7eb16cf700 1 --2- 172.21.15.98:0/2909017265 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7eac0caa90 0x7f7eac0a9d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:55.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.055+0000 7f7eb16cf700 1 -- 172.21.15.98:0/2909017265 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ea4009ae0 con 0x7f7eac0675b0 2023-10-06T05:06:55.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.055+0000 7f7eb0ece700 1 --2- 172.21.15.98:0/2909017265 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7eac0caa90 0x7f7eac0a9d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:06:55.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.055+0000 7f7eb16cf700 1 --2- 172.21.15.98:0/2909017265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7eac0675b0 0x7f7eac0b2b70 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7ea800d860 tx=0x7f7ea8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:06:55.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.055+0000 7f7ea17fa700 1 -- 172.21.15.98:0/2909017265 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ea8009610 con 0x7f7eac0675b0 2023-10-06T05:06:55.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.055+0000 7f7ea17fa700 1 -- 172.21.15.98:0/2909017265 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ea8009770 con 0x7f7eac0675b0 2023-10-06T05:06:55.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.055+0000 7f7eb3933700 1 -- 172.21.15.98:0/2909017265 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7eac0aa890 con 0x7f7eac0675b0 2023-10-06T05:06:55.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.055+0000 7f7ea17fa700 1 -- 172.21.15.98:0/2909017265 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ea8009b10 con 0x7f7eac0675b0 2023-10-06T05:06:55.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.055+0000 7f7eb3933700 1 -- 172.21.15.98:0/2909017265 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7eac0aae50 con 0x7f7eac0675b0 2023-10-06T05:06:55.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.056+0000 7f7eb3933700 1 -- 172.21.15.98:0/2909017265 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7eac0ab780 con 0x7f7eac0675b0 2023-10-06T05:06:55.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.056+0000 7f7ea17fa700 1 -- 172.21.15.98:0/2909017265 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f7ea8014430 con 0x7f7eac0675b0 2023-10-06T05:06:55.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.057+0000 7f7ea17fa700 1 --2- 172.21.15.98:0/2909017265 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f7e90066f30 0x7f7e900693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:55.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.057+0000 7f7ea17fa700 1 -- 172.21.15.98:0/2909017265 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f7ea8084710 con 0x7f7eac0675b0 2023-10-06T05:06:55.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.057+0000 7f7eb0ece700 1 --2- 172.21.15.98:0/2909017265 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f7e90066f30 0x7f7e900693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:55.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.057+0000 7f7eb0ece700 1 --2- 172.21.15.98:0/2909017265 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f7e90066f30 0x7f7e900693e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7e9c006f00 tx=0x7f7e9c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:06:55.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.061+0000 7f7ea17fa700 1 -- 172.21.15.98:0/2909017265 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7ea8051b70 con 0x7f7eac0675b0 2023-10-06T05:06:55.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.245+0000 7f7eb3933700 1 -- 172.21.15.98:0/2909017265 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f7eac064a90 con 0x7f7eac0675b0 2023-10-06T05:06:55.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.245+0000 7f7ea17fa700 1 -- 172.21.15.98:0/2909017265 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v21) v1 ==== 84+0+126 (secure 0 0 0) 0x7f7ea8055190 con 0x7f7eac0675b0 2023-10-06T05:06:55.248 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:06:55.248 INFO:teuthology.orchestra.run.smithi098.stdout:[ 2023-10-06T05:06:55.248 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-10-06T05:06:55.248 INFO:teuthology.orchestra.run.smithi098.stdout: "poolnum": 1, 2023-10-06T05:06:55.248 INFO:teuthology.orchestra.run.smithi098.stdout: "poolname": ".mgr" 2023-10-06T05:06:55.248 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-10-06T05:06:55.248 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-10-06T05:06:55.248 INFO:teuthology.orchestra.run.smithi098.stdout: "poolnum": 2, 2023-10-06T05:06:55.248 INFO:teuthology.orchestra.run.smithi098.stdout: "poolname": "rbd" 2023-10-06T05:06:55.249 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-10-06T05:06:55.249 INFO:teuthology.orchestra.run.smithi098.stdout:] 2023-10-06T05:06:55.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.248+0000 7f7eb3933700 1 -- 172.21.15.98:0/2909017265 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f7e90066f30 msgr2=0x7f7e900693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:55.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.248+0000 7f7eb3933700 1 --2- 172.21.15.98:0/2909017265 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f7e90066f30 0x7f7e900693e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7e9c006f00 tx=0x7f7e9c008040 comp rx=0 tx=0).stop 2023-10-06T05:06:55.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.248+0000 7f7eb3933700 1 -- 172.21.15.98:0/2909017265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7eac0675b0 msgr2=0x7f7eac0b2b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:55.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.248+0000 7f7eb3933700 1 --2- 172.21.15.98:0/2909017265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7eac0675b0 0x7f7eac0b2b70 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7ea800d860 tx=0x7f7ea8008040 comp rx=0 tx=0).stop 2023-10-06T05:06:55.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.248+0000 7f7eb3933700 1 -- 172.21.15.98:0/2909017265 shutdown_connections 2023-10-06T05:06:55.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.248+0000 7f7eb3933700 1 --2- 172.21.15.98:0/2909017265 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7eac15cab0 0x7f7eac0aa260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:55.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.248+0000 7f7eb3933700 1 --2- 172.21.15.98:0/2909017265 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f7e90066f30 0x7f7e900693e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:55.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.248+0000 7f7eb3933700 1 --2- 172.21.15.98:0/2909017265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7eac0675b0 0x7f7eac0b2b70 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:55.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.248+0000 7f7eb3933700 1 --2- 172.21.15.98:0/2909017265 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7eac0caa90 0x7f7eac0a9d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:55.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.248+0000 7f7eb3933700 1 -- 172.21.15.98:0/2909017265 >> 172.21.15.98:0/2909017265 conn(0x7f7eac0cdeb0 msgr2=0x7f7eac15ce80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:06:55.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.248+0000 7f7eb3933700 1 -- 172.21.15.98:0/2909017265 shutdown_connections 2023-10-06T05:06:55.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.248+0000 7f7eb3933700 1 -- 172.21.15.98:0/2909017265 wait complete. 2023-10-06T05:06:55.262 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-06T05:06:55.262 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-06T05:06:55.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.378+0000 7f246a31a700 1 Processor -- start 2023-10-06T05:06:55.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.378+0000 7f246a31a700 1 -- start start 2023-10-06T05:06:55.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.379+0000 7f246a31a700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f246415a690 0x7f2464157740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:55.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.379+0000 7f246a31a700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24640cdd40 con 0x7f246415d720 2023-10-06T05:06:55.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.379+0000 7f246a31a700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f246415b0b0 con 0x7f246415d350 2023-10-06T05:06:55.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.379+0000 7f246a31a700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24640b2c80 con 0x7f246415a690 2023-10-06T05:06:55.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.379+0000 7f24637fe700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f246415d720 0x7f2464159f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:49380/0 (socket says 172.21.15.98:49380) 2023-10-06T05:06:55.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.379+0000 7f24637fe700 1 -- 172.21.15.98:0/2438287157 learned_addr learned my addr 172.21.15.98:0/2438287157 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:06:55.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.379+0000 7f24688b7700 1 --2- 172.21.15.98:0/2438287157 >> v2:172.21.15.98:3300/0 conn(0x7f246415a690 0x7f2464157740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:55.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.379+0000 7f2462ffd700 1 -- 172.21.15.98:0/2438287157 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2469317753 0 0) 0x7f24640cdd40 con 0x7f246415d720 2023-10-06T05:06:55.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.379+0000 7f2462ffd700 1 -- 172.21.15.98:0/2438287157 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f244c003570 con 0x7f246415d720 2023-10-06T05:06:55.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.379+0000 7f2462ffd700 1 -- 172.21.15.98:0/2438287157 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 52691626 0 0) 0x7f246415b0b0 con 0x7f246415d350 2023-10-06T05:06:55.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.379+0000 7f24688b7700 1 -- 172.21.15.98:0/2438287157 >> v1:172.21.15.174:6789/0 conn(0x7f246415d350 legacy=0x7f24640cd630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:55.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.379+0000 7f24688b7700 1 -- 172.21.15.98:0/2438287157 >> v1:172.21.15.98:6789/0 conn(0x7f246415d720 legacy=0x7f2464159f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:55.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.379+0000 7f24688b7700 1 -- 172.21.15.98:0/2438287157 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f246415b390 con 0x7f246415a690 2023-10-06T05:06:55.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.379+0000 7f24688b7700 1 --2- 172.21.15.98:0/2438287157 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f246415a690 0x7f2464157740 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f245c009e90 tx=0x7f245c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f2dd8f499478e76 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:06:55.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.379+0000 7f2462ffd700 1 -- 172.21.15.98:0/2438287157 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f245c00e070 con 0x7f246415a690 2023-10-06T05:06:55.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.380+0000 7f2462ffd700 1 -- 172.21.15.98:0/2438287157 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f245c00bc40 con 0x7f246415a690 2023-10-06T05:06:55.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.380+0000 7f2462ffd700 1 -- 172.21.15.98:0/2438287157 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f245c012430 con 0x7f246415a690 2023-10-06T05:06:55.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.380+0000 7f246a31a700 1 -- 172.21.15.98:0/2438287157 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f246415a690 msgr2=0x7f2464157740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:55.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.380+0000 7f246a31a700 1 --2- 172.21.15.98:0/2438287157 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f246415a690 0x7f2464157740 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f245c009e90 tx=0x7f245c009780 comp rx=0 tx=0).stop 2023-10-06T05:06:55.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.380+0000 7f246a31a700 1 -- 172.21.15.98:0/2438287157 shutdown_connections 2023-10-06T05:06:55.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.380+0000 7f246a31a700 1 --2- 172.21.15.98:0/2438287157 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f246415a690 0x7f2464157740 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:55.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.380+0000 7f246a31a700 1 -- 172.21.15.98:0/2438287157 >> 172.21.15.98:0/2438287157 conn(0x7f2464068310 msgr2=0x7f2464068710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:06:55.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.380+0000 7f246a31a700 1 -- 172.21.15.98:0/2438287157 shutdown_connections 2023-10-06T05:06:55.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.380+0000 7f246a31a700 1 -- 172.21.15.98:0/2438287157 wait complete. 2023-10-06T05:06:55.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.381+0000 7f246a31a700 1 Processor -- start 2023-10-06T05:06:55.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.381+0000 7f246a31a700 1 -- start start 2023-10-06T05:06:55.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.382+0000 7f246a31a700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f246415a690 0x7f2464156560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:55.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.382+0000 7f246a31a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f246415d350 0x7f24640b5a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:55.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.382+0000 7f246a31a700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f246415d720 0x7f246414af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:55.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.382+0000 7f246a31a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24640c7570 con 0x7f246415d350 2023-10-06T05:06:55.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.382+0000 7f246a31a700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24640c7e10 con 0x7f246415d720 2023-10-06T05:06:55.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.382+0000 7f246a31a700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f246414b470 con 0x7f246415a690 2023-10-06T05:06:55.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.382+0000 7f24637fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f246415d350 0x7f24640b5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:55.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.382+0000 7f24637fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f246415d350 0x7f24640b5a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54242/0 (socket says 172.21.15.98:54242) 2023-10-06T05:06:55.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.382+0000 7f24637fe700 1 -- 172.21.15.98:0/3848339048 learned_addr learned my addr 172.21.15.98:0/3848339048 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:06:55.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.382+0000 7f24688b7700 1 --2- 172.21.15.98:0/3848339048 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f246415d720 0x7f246414af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:55.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.382+0000 7f2463fff700 1 --2- 172.21.15.98:0/3848339048 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f246415a690 0x7f2464156560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:55.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.382+0000 7f24637fe700 1 -- 172.21.15.98:0/3848339048 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f246415a690 msgr2=0x7f2464156560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:55.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.382+0000 7f24637fe700 1 --2- 172.21.15.98:0/3848339048 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f246415a690 0x7f2464156560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:55.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.382+0000 7f24637fe700 1 -- 172.21.15.98:0/3848339048 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f246415d720 msgr2=0x7f246414af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:55.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.382+0000 7f24637fe700 1 --2- 172.21.15.98:0/3848339048 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f246415d720 0x7f246414af30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:55.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.382+0000 7f24637fe700 1 -- 172.21.15.98:0/3848339048 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f245c009a20 con 0x7f246415d350 2023-10-06T05:06:55.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.382+0000 7f24637fe700 1 --2- 172.21.15.98:0/3848339048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f246415d350 0x7f24640b5a70 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f245400c580 tx=0x7f2454007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:06:55.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.383+0000 7f244bfff700 1 -- 172.21.15.98:0/3848339048 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2454005b30 con 0x7f246415d350 2023-10-06T05:06:55.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.383+0000 7f244bfff700 1 -- 172.21.15.98:0/3848339048 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2454005c90 con 0x7f246415d350 2023-10-06T05:06:55.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.383+0000 7f244bfff700 1 -- 172.21.15.98:0/3848339048 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2454007320 con 0x7f246415d350 2023-10-06T05:06:55.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.383+0000 7f246a31a700 1 -- 172.21.15.98:0/3848339048 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24640b6150 con 0x7f246415d350 2023-10-06T05:06:55.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.383+0000 7f246a31a700 1 -- 172.21.15.98:0/3848339048 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24640c7aa0 con 0x7f246415d350 2023-10-06T05:06:55.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.384+0000 7f246a31a700 1 -- 172.21.15.98:0/3848339048 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2464064a90 con 0x7f246415d350 2023-10-06T05:06:55.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.384+0000 7f244bfff700 1 -- 172.21.15.98:0/3848339048 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f2454007690 con 0x7f246415d350 2023-10-06T05:06:55.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.384+0000 7f244bfff700 1 --2- 172.21.15.98:0/3848339048 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f2450066f30 0x7f24500693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:55.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.384+0000 7f244bfff700 1 -- 172.21.15.98:0/3848339048 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f2454083810 con 0x7f246415d350 2023-10-06T05:06:55.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.384+0000 7f2463fff700 1 --2- 172.21.15.98:0/3848339048 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f2450066f30 0x7f24500693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:55.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.385+0000 7f2463fff700 1 --2- 172.21.15.98:0/3848339048 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f2450066f30 0x7f24500693e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2458001800 tx=0x7f2458009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:06:55.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.389+0000 7f244bfff700 1 -- 172.21.15.98:0/3848339048 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2454050c70 con 0x7f246415d350 2023-10-06T05:06:55.573 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.570+0000 7f246a31a700 1 -- 172.21.15.98:0/3848339048 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f24640c7f50 con 0x7f246415d350 2023-10-06T05:06:55.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.690+0000 7f244bfff700 1 -- 172.21.15.98:0/3848339048 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v22) v1 ==== 138+0+0 (secure 0 0 0) 0x7f2454017020 con 0x7f246415d350 2023-10-06T05:06:55.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.693+0000 7f246a31a700 1 -- 172.21.15.98:0/3848339048 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f2450066f30 msgr2=0x7f24500693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:55.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.693+0000 7f246a31a700 1 --2- 172.21.15.98:0/3848339048 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f2450066f30 0x7f24500693e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2458001800 tx=0x7f2458009040 comp rx=0 tx=0).stop 2023-10-06T05:06:55.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.693+0000 7f246a31a700 1 -- 172.21.15.98:0/3848339048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f246415d350 msgr2=0x7f24640b5a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:55.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.693+0000 7f246a31a700 1 --2- 172.21.15.98:0/3848339048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f246415d350 0x7f24640b5a70 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f245400c580 tx=0x7f2454007df0 comp rx=0 tx=0).stop 2023-10-06T05:06:55.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.693+0000 7f246a31a700 1 -- 172.21.15.98:0/3848339048 shutdown_connections 2023-10-06T05:06:55.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.693+0000 7f246a31a700 1 --2- 172.21.15.98:0/3848339048 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f246415d720 0x7f246414af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:55.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.693+0000 7f246a31a700 1 --2- 172.21.15.98:0/3848339048 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f246415a690 0x7f2464156560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:55.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.693+0000 7f246a31a700 1 --2- 172.21.15.98:0/3848339048 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f2450066f30 0x7f24500693e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:55.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.693+0000 7f246a31a700 1 --2- 172.21.15.98:0/3848339048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f246415d350 0x7f24640b5a70 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:55.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.693+0000 7f246a31a700 1 -- 172.21.15.98:0/3848339048 >> 172.21.15.98:0/3848339048 conn(0x7f2464068310 msgr2=0x7f2464157130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:06:55.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.693+0000 7f246a31a700 1 -- 172.21.15.98:0/3848339048 shutdown_connections 2023-10-06T05:06:55.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.693+0000 7f246a31a700 1 -- 172.21.15.98:0/3848339048 wait complete. 2023-10-06T05:06:55.701 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'cephfs_metadata' created 2023-10-06T05:06:55.710 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2023-10-06T05:06:55.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.828+0000 7f55045fb700 1 Processor -- start 2023-10-06T05:06:55.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.828+0000 7f55045fb700 1 -- start start 2023-10-06T05:06:55.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.829+0000 7f55045fb700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f54fc1594d0 0x7f54fc0cb560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:55.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.829+0000 7f55045fb700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54fc0cbee0 con 0x7f54fc0cbaa0 2023-10-06T05:06:55.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.829+0000 7f55045fb700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54fc0cf5d0 con 0x7f54fc1598a0 2023-10-06T05:06:55.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.829+0000 7f55045fb700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54fc0cf8b0 con 0x7f54fc1594d0 2023-10-06T05:06:55.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.829+0000 7f5502b98700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f54fc0cbaa0 0x7f54fc0c5690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:49382/0 (socket says 172.21.15.98:49382) 2023-10-06T05:06:55.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.829+0000 7f5502b98700 1 -- 172.21.15.98:0/2692416694 learned_addr learned my addr 172.21.15.98:0/2692416694 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:06:55.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.829+0000 7f5502397700 1 --2- 172.21.15.98:0/2692416694 >> v2:172.21.15.174:3300/0 conn(0x7f54fc1594d0 0x7f54fc0cb560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:55.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.829+0000 7f5501395700 1 -- 172.21.15.98:0/2692416694 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2376190053 0 0) 0x7f54fc0cbee0 con 0x7f54fc0cbaa0 2023-10-06T05:06:55.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.829+0000 7f5501395700 1 -- 172.21.15.98:0/2692416694 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54e8003570 con 0x7f54fc0cbaa0 2023-10-06T05:06:55.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.830+0000 7f5501395700 1 -- 172.21.15.98:0/2692416694 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1727225049 0 0) 0x7f54e8003570 con 0x7f54fc0cbaa0 2023-10-06T05:06:55.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.830+0000 7f5501395700 1 -- 172.21.15.98:0/2692416694 >> v2:172.21.15.174:3300/0 conn(0x7f54fc1594d0 msgr2=0x7f54fc0cb560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:06:55.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.830+0000 7f5501395700 1 --2- 172.21.15.98:0/2692416694 >> v2:172.21.15.174:3300/0 conn(0x7f54fc1594d0 0x7f54fc0cb560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:55.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.830+0000 7f5501395700 1 -- 172.21.15.98:0/2692416694 >> v1:172.21.15.174:6789/0 conn(0x7f54fc1598a0 legacy=0x7f54fc0ce7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:55.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.830+0000 7f5501395700 1 -- 172.21.15.98:0/2692416694 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54fc0cfb90 con 0x7f54fc0cbaa0 2023-10-06T05:06:55.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.830+0000 7f5501395700 1 -- 172.21.15.98:0/2692416694 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1543991876 0 0) 0x7f54f4001470 con 0x7f54fc0cbaa0 2023-10-06T05:06:55.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.830+0000 7f5501395700 1 -- 172.21.15.98:0/2692416694 >> v1:172.21.15.98:6789/0 conn(0x7f54fc0cbaa0 legacy=0x7f54fc0c5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:55.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.830+0000 7f5501395700 1 --2- 172.21.15.98:0/2692416694 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f54e8003a60 0x7f54e8003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:55.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.830+0000 7f5501395700 1 --2- 172.21.15.98:0/2692416694 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54e8004810 0x7f54e8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:55.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.830+0000 7f5501395700 1 --2- 172.21.15.98:0/2692416694 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f54e80051c0 0x7f54e800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:55.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.830+0000 7f5501395700 1 -- 172.21.15.98:0/2692416694 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54fc0cf8b0 con 0x7f54e8004810 2023-10-06T05:06:55.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.830+0000 7f5501395700 1 -- 172.21.15.98:0/2692416694 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54e800c0d0 con 0x7f54e80051c0 2023-10-06T05:06:55.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.830+0000 7f5501395700 1 -- 172.21.15.98:0/2692416694 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54e800c290 con 0x7f54e8003a60 2023-10-06T05:06:55.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.830+0000 7f5501b96700 1 --2- 172.21.15.98:0/2692416694 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54e8004810 0x7f54e8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:55.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.830+0000 7f5502b98700 1 --2- 172.21.15.98:0/2692416694 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f54e80051c0 0x7f54e800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:55.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.830+0000 7f5502397700 1 --2- 172.21.15.98:0/2692416694 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f54e8003a60 0x7f54e8003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:55.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.830+0000 7f5501b96700 1 -- 172.21.15.98:0/2692416694 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f54e8003a60 msgr2=0x7f54e8003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:06:55.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.830+0000 7f5501b96700 1 --2- 172.21.15.98:0/2692416694 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f54e8003a60 0x7f54e8003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:55.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.830+0000 7f5501b96700 1 -- 172.21.15.98:0/2692416694 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f54e80051c0 msgr2=0x7f54e800ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:06:55.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.830+0000 7f5501b96700 1 --2- 172.21.15.98:0/2692416694 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f54e80051c0 0x7f54e800ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:55.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.830+0000 7f5502b98700 1 -- 172.21.15.98:0/2692416694 reap_dead start 2023-10-06T05:06:55.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.830+0000 7f5501b96700 1 -- 172.21.15.98:0/2692416694 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54e8003570 con 0x7f54e8004810 2023-10-06T05:06:55.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.830+0000 7f5502397700 1 --2- 172.21.15.98:0/2692416694 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f54e8003a60 0x7f54e8003e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:06:55.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.830+0000 7f5501b96700 1 --2- 172.21.15.98:0/2692416694 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54e8004810 0x7f54e8004c80 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f54fc067cf0 tx=0x7f54ec007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e13d31c76c2cf1b2 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:06:55.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.830+0000 7f5501395700 1 -- 172.21.15.98:0/2692416694 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54ec012420 con 0x7f54e8004810 2023-10-06T05:06:55.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.830+0000 7f5501395700 1 -- 172.21.15.98:0/2692416694 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54ec0125a0 con 0x7f54e8004810 2023-10-06T05:06:55.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.830+0000 7f5501395700 1 -- 172.21.15.98:0/2692416694 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54ec012960 con 0x7f54e8004810 2023-10-06T05:06:55.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.831+0000 7f55045fb700 1 -- 172.21.15.98:0/2692416694 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54e8004810 msgr2=0x7f54e8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:55.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.831+0000 7f55045fb700 1 --2- 172.21.15.98:0/2692416694 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54e8004810 0x7f54e8004c80 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f54fc067cf0 tx=0x7f54ec007d30 comp rx=0 tx=0).stop 2023-10-06T05:06:55.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.831+0000 7f55045fb700 1 -- 172.21.15.98:0/2692416694 shutdown_connections 2023-10-06T05:06:55.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.831+0000 7f55045fb700 1 --2- 172.21.15.98:0/2692416694 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54e8004810 0x7f54e8004c80 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:55.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.831+0000 7f55045fb700 1 -- 172.21.15.98:0/2692416694 >> 172.21.15.98:0/2692416694 conn(0x7f54fc089e50 msgr2=0x7f54fc099770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:06:55.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.831+0000 7f55045fb700 1 -- 172.21.15.98:0/2692416694 shutdown_connections 2023-10-06T05:06:55.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.831+0000 7f55045fb700 1 -- 172.21.15.98:0/2692416694 wait complete. 2023-10-06T05:06:55.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.832+0000 7f55045fb700 1 Processor -- start 2023-10-06T05:06:55.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.832+0000 7f55045fb700 1 -- start start 2023-10-06T05:06:55.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.833+0000 7f55045fb700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f54e8004810 0x7f54fc0c9c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:55.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.833+0000 7f55045fb700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54fc16e320 0x7f54fc0b4100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:55.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.833+0000 7f55045fb700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f54fc16d6c0 0x7f54fc16db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:55.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.833+0000 7f55045fb700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54fc16e080 con 0x7f54fc16e320 2023-10-06T05:06:55.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.833+0000 7f55045fb700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54fc0b4a90 con 0x7f54e8004810 2023-10-06T05:06:55.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.833+0000 7f55045fb700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54fc0b58f0 con 0x7f54fc16d6c0 2023-10-06T05:06:55.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.833+0000 7f5501b96700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54fc16e320 0x7f54fc0b4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:55.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.833+0000 7f5501b96700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54fc16e320 0x7f54fc0b4100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54268/0 (socket says 172.21.15.98:54268) 2023-10-06T05:06:55.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.833+0000 7f5501b96700 1 -- 172.21.15.98:0/2762394586 learned_addr learned my addr 172.21.15.98:0/2762394586 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:06:55.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.833+0000 7f5502b98700 1 --2- 172.21.15.98:0/2762394586 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f54fc16d6c0 0x7f54fc16db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:55.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.833+0000 7f5502397700 1 --2- 172.21.15.98:0/2762394586 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f54e8004810 0x7f54fc0c9c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:55.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.833+0000 7f5501b96700 1 -- 172.21.15.98:0/2762394586 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f54fc16d6c0 msgr2=0x7f54fc16db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:55.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.833+0000 7f5501b96700 1 --2- 172.21.15.98:0/2762394586 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f54fc16d6c0 0x7f54fc16db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:55.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.833+0000 7f5501b96700 1 -- 172.21.15.98:0/2762394586 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f54e8004810 msgr2=0x7f54fc0c9c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:55.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.833+0000 7f5501b96700 1 --2- 172.21.15.98:0/2762394586 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f54e8004810 0x7f54fc0c9c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:55.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.833+0000 7f5501b96700 1 -- 172.21.15.98:0/2762394586 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54fc0b4640 con 0x7f54fc16e320 2023-10-06T05:06:55.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.833+0000 7f5502397700 1 --2- 172.21.15.98:0/2762394586 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f54e8004810 0x7f54fc0c9c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:06:55.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.833+0000 7f5502b98700 1 --2- 172.21.15.98:0/2762394586 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f54fc16d6c0 0x7f54fc16db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:06:55.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.834+0000 7f5501b96700 1 --2- 172.21.15.98:0/2762394586 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54fc16e320 0x7f54fc0b4100 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f54ec012ad0 tx=0x7f54ec015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:06:55.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.834+0000 7f54f27fc700 1 -- 172.21.15.98:0/2762394586 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54ec006820 con 0x7f54fc16e320 2023-10-06T05:06:55.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.834+0000 7f54f27fc700 1 -- 172.21.15.98:0/2762394586 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54ec0069a0 con 0x7f54fc16e320 2023-10-06T05:06:55.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.834+0000 7f55045fb700 1 -- 172.21.15.98:0/2762394586 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54fc0b4ce0 con 0x7f54fc16e320 2023-10-06T05:06:55.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.834+0000 7f54f27fc700 1 -- 172.21.15.98:0/2762394586 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54ec006d60 con 0x7f54fc16e320 2023-10-06T05:06:55.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.834+0000 7f55045fb700 1 -- 172.21.15.98:0/2762394586 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54fc0b52a0 con 0x7f54fc16e320 2023-10-06T05:06:55.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.835+0000 7f55045fb700 1 -- 172.21.15.98:0/2762394586 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54fc0b5a80 con 0x7f54fc16e320 2023-10-06T05:06:55.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.835+0000 7f54f27fc700 1 -- 172.21.15.98:0/2762394586 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f54ec019030 con 0x7f54fc16e320 2023-10-06T05:06:55.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.835+0000 7f54f27fc700 1 --2- 172.21.15.98:0/2762394586 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f54e0066f30 0x7f54e00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:55.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.836+0000 7f54f27fc700 1 -- 172.21.15.98:0/2762394586 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5576+0+0 (secure 0 0 0) 0x7f54ec085520 con 0x7f54fc16e320 2023-10-06T05:06:55.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.836+0000 7f5502397700 1 --2- 172.21.15.98:0/2762394586 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f54e0066f30 0x7f54e00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:55.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.836+0000 7f5502397700 1 --2- 172.21.15.98:0/2762394586 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f54e0066f30 0x7f54e00693e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f54fc04cc00 tx=0x7f54f800b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:06:55.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:55.840+0000 7f54f27fc700 1 -- 172.21.15.98:0/2762394586 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f54ec052810 con 0x7f54fc16e320 2023-10-06T05:06:56.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:56.023+0000 7f55045fb700 1 -- 172.21.15.98:0/2762394586 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) v1 -- 0x7f54fc064a90 con 0x7f54fc16e320 2023-10-06T05:06:56.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:56.698+0000 7f54f27fc700 1 -- 172.21.15.98:0/2762394586 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v23) v1 ==== 171+0+0 (secure 0 0 0) 0x7f54ec055e30 con 0x7f54fc16e320 2023-10-06T05:06:56.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:56.708+0000 7f55045fb700 1 -- 172.21.15.98:0/2762394586 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f54e0066f30 msgr2=0x7f54e00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:56.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:56.708+0000 7f55045fb700 1 --2- 172.21.15.98:0/2762394586 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f54e0066f30 0x7f54e00693e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f54fc04cc00 tx=0x7f54f800b040 comp rx=0 tx=0).stop 2023-10-06T05:06:56.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:56.708+0000 7f55045fb700 1 -- 172.21.15.98:0/2762394586 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54fc16e320 msgr2=0x7f54fc0b4100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:56.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:56.708+0000 7f55045fb700 1 --2- 172.21.15.98:0/2762394586 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54fc16e320 0x7f54fc0b4100 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f54ec012ad0 tx=0x7f54ec015000 comp rx=0 tx=0).stop 2023-10-06T05:06:56.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:56.708+0000 7f55045fb700 1 -- 172.21.15.98:0/2762394586 shutdown_connections 2023-10-06T05:06:56.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:56.708+0000 7f55045fb700 1 --2- 172.21.15.98:0/2762394586 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f54fc16d6c0 0x7f54fc16db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:56.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:56.708+0000 7f55045fb700 1 --2- 172.21.15.98:0/2762394586 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f54e8004810 0x7f54fc0c9c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:56.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:56.708+0000 7f55045fb700 1 --2- 172.21.15.98:0/2762394586 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f54e0066f30 0x7f54e00693e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:56.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:56.708+0000 7f55045fb700 1 --2- 172.21.15.98:0/2762394586 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f54fc16e320 0x7f54fc0b4100 secure :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f54ec012ad0 tx=0x7f54ec015000 comp rx=0 tx=0).stop 2023-10-06T05:06:56.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:56.708+0000 7f55045fb700 1 -- 172.21.15.98:0/2762394586 >> 172.21.15.98:0/2762394586 conn(0x7f54fc089e50 msgr2=0x7f54fc15d470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:06:56.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:56.710+0000 7f55045fb700 1 -- 172.21.15.98:0/2762394586 shutdown_connections 2023-10-06T05:06:56.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:56.710+0000 7f55045fb700 1 -- 172.21.15.98:0/2762394586 wait complete. 2023-10-06T05:06:56.713 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'cephfs_data' created 2023-10-06T05:06:56.728 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-06T05:06:56.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:56.844+0000 7f3ecae21700 1 Processor -- start 2023-10-06T05:06:56.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:56.844+0000 7f3ecae21700 1 -- start start 2023-10-06T05:06:56.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:56.845+0000 7f3ecae21700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f3ec41598a0 0x7f3ec40ce7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:56.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:56.845+0000 7f3ecae21700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3ec40cbc70 0x7f3ec40ced20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:56.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:56.845+0000 7f3ecae21700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ec40cf5d0 con 0x7f3ec41594d0 2023-10-06T05:06:56.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:56.845+0000 7f3ecae21700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ec40cf8b0 con 0x7f3ec41598a0 2023-10-06T05:06:56.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:56.845+0000 7f3ecae21700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec40cfb90 con 0x7f3ec40cbc70 2023-10-06T05:06:56.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:56.845+0000 7f3ec93be700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3ec40cbc70 0x7f3ec40ced20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:56.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:56.845+0000 7f3ec3fff700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f3ec41598a0 0x7f3ec40ce7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:56.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:56.845+0000 7f3ec93be700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3ec40cbc70 0x7f3ec40ced20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.98:51598/0 (socket says 172.21.15.98:51598) 2023-10-06T05:06:56.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:56.845+0000 7f3ec93be700 1 -- 172.21.15.98:0/361838112 learned_addr learned my addr 172.21.15.98:0/361838112 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:06:56.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:56.845+0000 7f3ec3fff700 1 -- 172.21.15.98:0/361838112 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3ec40cbc70 msgr2=0x7f3ec40ced20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:06:56.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:56.845+0000 7f3ec3fff700 1 --2- 172.21.15.98:0/361838112 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3ec40cbc70 0x7f3ec40ced20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:56.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:56.845+0000 7f3ec3fff700 1 -- 172.21.15.98:0/361838112 >> v1:172.21.15.174:6789/0 conn(0x7f3ec41594d0 legacy=0x7f3ec40cb560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:56.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:56.845+0000 7f3ec3fff700 1 -- 172.21.15.98:0/361838112 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ec40cfe70 con 0x7f3ec41598a0 2023-10-06T05:06:56.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:56.845+0000 7f3ec37fe700 1 -- 172.21.15.98:0/361838112 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2172385272 0 0) 0x7f3ec40cf5d0 con 0x7f3ec41594d0 2023-10-06T05:06:56.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:56.845+0000 7f3ec93be700 1 --2- 172.21.15.98:0/361838112 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3ec40cbc70 0x7f3ec40ced20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T05:06:56.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:56.846+0000 7f3ec3fff700 1 --2- 172.21.15.98:0/361838112 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3ec41598a0 0x7f3ec40ce7e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3eb00098e0 tx=0x7f3eb00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=96b102c3a3973d20 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:06:56.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:56.846+0000 7f3ec37fe700 1 -- 172.21.15.98:0/361838112 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eb000b640 con 0x7f3ec41598a0 2023-10-06T05:06:56.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:56.846+0000 7f3ec37fe700 1 -- 172.21.15.98:0/361838112 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3eb000b7a0 con 0x7f3ec41598a0 2023-10-06T05:06:56.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:56.846+0000 7f3ec37fe700 1 -- 172.21.15.98:0/361838112 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eb000bb40 con 0x7f3ec41598a0 2023-10-06T05:06:56.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:56.846+0000 7f3ecae21700 1 -- 172.21.15.98:0/361838112 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3ec41598a0 msgr2=0x7f3ec40ce7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:56.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:56.846+0000 7f3ecae21700 1 --2- 172.21.15.98:0/361838112 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3ec41598a0 0x7f3ec40ce7e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3eb00098e0 tx=0x7f3eb00091a0 comp rx=0 tx=0).stop 2023-10-06T05:06:56.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:56.846+0000 7f3ecae21700 1 -- 172.21.15.98:0/361838112 shutdown_connections 2023-10-06T05:06:56.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:56.846+0000 7f3ecae21700 1 --2- 172.21.15.98:0/361838112 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3ec40cbc70 0x7f3ec40ced20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:56.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:56.846+0000 7f3ecae21700 1 --2- 172.21.15.98:0/361838112 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3ec41598a0 0x7f3ec40ce7e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:56.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:56.846+0000 7f3ecae21700 1 -- 172.21.15.98:0/361838112 >> 172.21.15.98:0/361838112 conn(0x7f3ec4089e50 msgr2=0x7f3ec4099770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:06:56.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:56.846+0000 7f3ecae21700 1 -- 172.21.15.98:0/361838112 shutdown_connections 2023-10-06T05:06:56.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:56.846+0000 7f3ecae21700 1 -- 172.21.15.98:0/361838112 wait complete. 2023-10-06T05:06:56.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:56.847+0000 7f3ecae21700 1 Processor -- start 2023-10-06T05:06:56.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:56.848+0000 7f3ecae21700 1 -- start start 2023-10-06T05:06:56.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:56.848+0000 7f3ecae21700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3ec40cbc70 0x7f3ec40c75c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:56.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:56.848+0000 7f3ecae21700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3ec41594d0 0x7f3ec415c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:56.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:56.848+0000 7f3ecae21700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ec41598a0 0x7f3ec40c9830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:56.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:56.848+0000 7f3ecae21700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec4149910 con 0x7f3ec41598a0 2023-10-06T05:06:56.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:56.848+0000 7f3ecae21700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec40c7ba0 con 0x7f3ec41594d0 2023-10-06T05:06:56.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:56.848+0000 7f3ecae21700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec40c7e80 con 0x7f3ec40cbc70 2023-10-06T05:06:56.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:56.848+0000 7f3ec93be700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ec41598a0 0x7f3ec40c9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:56.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:56.848+0000 7f3ec93be700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ec41598a0 0x7f3ec40c9830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54274/0 (socket says 172.21.15.98:54274) 2023-10-06T05:06:56.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:56.848+0000 7f3ec93be700 1 -- 172.21.15.98:0/3276566011 learned_addr learned my addr 172.21.15.98:0/3276566011 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:06:56.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:56.849+0000 7f3ec8bbd700 1 --2- 172.21.15.98:0/3276566011 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3ec40cbc70 0x7f3ec40c75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:56.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:56.849+0000 7f3ec3fff700 1 --2- 172.21.15.98:0/3276566011 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3ec41594d0 0x7f3ec415c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:56.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:56.849+0000 7f3ec93be700 1 -- 172.21.15.98:0/3276566011 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3ec40cbc70 msgr2=0x7f3ec40c75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:56.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:56.849+0000 7f3ec93be700 1 --2- 172.21.15.98:0/3276566011 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3ec40cbc70 0x7f3ec40c75c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:56.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:56.849+0000 7f3ec93be700 1 -- 172.21.15.98:0/3276566011 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3ec41594d0 msgr2=0x7f3ec415c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:56.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:56.849+0000 7f3ec93be700 1 --2- 172.21.15.98:0/3276566011 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3ec41594d0 0x7f3ec415c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:56.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:56.849+0000 7f3ec93be700 1 -- 172.21.15.98:0/3276566011 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ebc009ae0 con 0x7f3ec41598a0 2023-10-06T05:06:56.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:56.849+0000 7f3ec93be700 1 --2- 172.21.15.98:0/3276566011 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ec41598a0 0x7f3ec40c9830 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3ebc009fd0 tx=0x7f3ebc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:06:56.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:56.849+0000 7f3ec0ff9700 1 -- 172.21.15.98:0/3276566011 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ebc018070 con 0x7f3ec41598a0 2023-10-06T05:06:56.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:56.849+0000 7f3ec0ff9700 1 -- 172.21.15.98:0/3276566011 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ebc00f040 con 0x7f3ec41598a0 2023-10-06T05:06:56.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:56.849+0000 7f3ecae21700 1 -- 172.21.15.98:0/3276566011 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3eb0009420 con 0x7f3ec41598a0 2023-10-06T05:06:56.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:56.849+0000 7f3ec0ff9700 1 -- 172.21.15.98:0/3276566011 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ebc00cc80 con 0x7f3ec41598a0 2023-10-06T05:06:56.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:56.849+0000 7f3ecae21700 1 -- 172.21.15.98:0/3276566011 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ec40c8440 con 0x7f3ec41598a0 2023-10-06T05:06:56.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:56.850+0000 7f3ecae21700 1 -- 172.21.15.98:0/3276566011 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ec4064a90 con 0x7f3ec41598a0 2023-10-06T05:06:56.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:56.851+0000 7f3ec0ff9700 1 -- 172.21.15.98:0/3276566011 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f3ebc003020 con 0x7f3ec41598a0 2023-10-06T05:06:56.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:56.851+0000 7f3ec0ff9700 1 --2- 172.21.15.98:0/3276566011 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f3ea8066f30 0x7f3ea80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:56.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:56.851+0000 7f3ec0ff9700 1 -- 172.21.15.98:0/3276566011 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5958+0+0 (secure 0 0 0) 0x7f3ebc084b50 con 0x7f3ec41598a0 2023-10-06T05:06:56.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:56.851+0000 7f3ec8bbd700 1 --2- 172.21.15.98:0/3276566011 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f3ea8066f30 0x7f3ea80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:56.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:56.852+0000 7f3ec8bbd700 1 --2- 172.21.15.98:0/3276566011 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f3ea8066f30 0x7f3ea80693e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3ec415b5d0 tx=0x7f3eb8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:06:56.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:56.856+0000 7f3ec0ff9700 1 -- 172.21.15.98:0/3276566011 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3ebc051cc0 con 0x7f3ec41598a0 2023-10-06T05:06:56.873 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-06T05:06:56.875 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-06T05:06:57.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:57.084+0000 7f3ecae21700 1 -- 172.21.15.98:0/3276566011 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f3ec40cf260 con 0x7f3ec41598a0 2023-10-06T05:06:57.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:57.698+0000 7f3ec0ff9700 1 -- 172.21.15.98:0/3276566011 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 3 and data pool 4 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f3ebc0552e0 con 0x7f3ec41598a0 2023-10-06T05:06:57.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:57.702+0000 7f3ecae21700 1 -- 172.21.15.98:0/3276566011 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f3ea8066f30 msgr2=0x7f3ea80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:57.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:57.702+0000 7f3ecae21700 1 --2- 172.21.15.98:0/3276566011 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f3ea8066f30 0x7f3ea80693e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3ec415b5d0 tx=0x7f3eb8009040 comp rx=0 tx=0).stop 2023-10-06T05:06:57.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:57.702+0000 7f3ecae21700 1 -- 172.21.15.98:0/3276566011 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ec41598a0 msgr2=0x7f3ec40c9830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:57.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:57.702+0000 7f3ecae21700 1 --2- 172.21.15.98:0/3276566011 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ec41598a0 0x7f3ec40c9830 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3ebc009fd0 tx=0x7f3ebc00b040 comp rx=0 tx=0).stop 2023-10-06T05:06:57.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:57.703+0000 7f3ecae21700 1 -- 172.21.15.98:0/3276566011 shutdown_connections 2023-10-06T05:06:57.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:57.703+0000 7f3ecae21700 1 --2- 172.21.15.98:0/3276566011 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f3ea8066f30 0x7f3ea80693e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:57.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:57.703+0000 7f3ecae21700 1 --2- 172.21.15.98:0/3276566011 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3ec41598a0 0x7f3ec40c9830 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:57.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:57.703+0000 7f3ecae21700 1 --2- 172.21.15.98:0/3276566011 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3ec40cbc70 0x7f3ec40c75c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:57.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:57.703+0000 7f3ecae21700 1 --2- 172.21.15.98:0/3276566011 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3ec41594d0 0x7f3ec415c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:57.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:57.703+0000 7f3ecae21700 1 -- 172.21.15.98:0/3276566011 >> 172.21.15.98:0/3276566011 conn(0x7f3ec4089e50 msgr2=0x7f3ec414bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:06:57.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:57.703+0000 7f3ecae21700 1 -- 172.21.15.98:0/3276566011 shutdown_connections 2023-10-06T05:06:57.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:57.703+0000 7f3ecae21700 1 -- 172.21.15.98:0/3276566011 wait complete. 2023-10-06T05:06:57.706 INFO:teuthology.orchestra.run.smithi098.stderr:new fs with metadata pool 3 and data pool 4 2023-10-06T05:06:57.719 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2023-10-06T05:06:57.720 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2023-10-06T05:06:57.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:57.841+0000 7f8670990700 1 Processor -- start 2023-10-06T05:06:57.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:57.841+0000 7f8670990700 1 -- start start 2023-10-06T05:06:57.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:57.842+0000 7f8670990700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f866c06d5d0 0x7f866c0cbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:57.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:57.842+0000 7f8670990700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f866c157320 0x7f866c0b1380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:57.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:57.842+0000 7f8670990700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f866c0cd570 con 0x7f866c156f50 2023-10-06T05:06:57.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:57.842+0000 7f8670990700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f866c0cd0e0 con 0x7f866c157320 2023-10-06T05:06:57.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:57.842+0000 7f8670990700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f866c0cc4d0 con 0x7f866c06d5d0 2023-10-06T05:06:57.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:57.842+0000 7f866b59e700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f866c157320 0x7f866c0b1380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:57.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:57.842+0000 7f866b59e700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f866c157320 0x7f866c0b1380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54278/0 (socket says 172.21.15.98:54278) 2023-10-06T05:06:57.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:57.842+0000 7f866b59e700 1 -- 172.21.15.98:0/3517521167 learned_addr learned my addr 172.21.15.98:0/3517521167 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:06:57.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:57.842+0000 7f866ad9d700 1 --1- 172.21.15.98:0/3517521167 >> v1:172.21.15.98:6789/0 conn(0x7f866c156f50 0x7f866c06cec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:49394/0 (socket says 172.21.15.98:49394) 2023-10-06T05:06:57.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:57.842+0000 7f866a59c700 1 --2- 172.21.15.98:0/3517521167 >> v2:172.21.15.174:3300/0 conn(0x7f866c06d5d0 0x7f866c0cbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:57.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:57.842+0000 7f866b59e700 1 -- 172.21.15.98:0/3517521167 >> v2:172.21.15.174:3300/0 conn(0x7f866c06d5d0 msgr2=0x7f866c0cbdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:06:57.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:57.842+0000 7f866b59e700 1 --2- 172.21.15.98:0/3517521167 >> v2:172.21.15.174:3300/0 conn(0x7f866c06d5d0 0x7f866c0cbdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:57.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:57.842+0000 7f866b59e700 1 -- 172.21.15.98:0/3517521167 >> v1:172.21.15.98:6789/0 conn(0x7f866c156f50 legacy=0x7f866c06cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:57.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:57.842+0000 7f866b59e700 1 -- 172.21.15.98:0/3517521167 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f866c0cc7b0 con 0x7f866c157320 2023-10-06T05:06:57.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:57.842+0000 7f8669d9b700 1 -- 172.21.15.98:0/3517521167 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1739793703 0 0) 0x7f865c001110 con 0x7f866c156f50 2023-10-06T05:06:57.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:57.842+0000 7f866b59e700 1 --2- 172.21.15.98:0/3517521167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f866c157320 0x7f866c0b1380 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f866c0cf1f0 tx=0x7f8664009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73819fddb8488ee6 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:06:57.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:57.842+0000 7f8669d9b700 1 -- 172.21.15.98:0/3517521167 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f866400e070 con 0x7f866c157320 2023-10-06T05:06:57.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:57.842+0000 7f8669d9b700 1 -- 172.21.15.98:0/3517521167 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f866400ba20 con 0x7f866c157320 2023-10-06T05:06:57.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:57.842+0000 7f8669d9b700 1 -- 172.21.15.98:0/3517521167 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86640092e0 con 0x7f866c157320 2023-10-06T05:06:57.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:57.843+0000 7f8670990700 1 -- 172.21.15.98:0/3517521167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f866c157320 msgr2=0x7f866c0b1380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:57.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:57.843+0000 7f8670990700 1 --2- 172.21.15.98:0/3517521167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f866c157320 0x7f866c0b1380 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f866c0cf1f0 tx=0x7f8664009840 comp rx=0 tx=0).stop 2023-10-06T05:06:57.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:57.843+0000 7f8670990700 1 -- 172.21.15.98:0/3517521167 shutdown_connections 2023-10-06T05:06:57.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:57.843+0000 7f8670990700 1 --2- 172.21.15.98:0/3517521167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f866c157320 0x7f866c0b1380 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:57.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:57.843+0000 7f8670990700 1 --2- 172.21.15.98:0/3517521167 >> v2:172.21.15.174:3300/0 conn(0x7f866c06d5d0 0x7f866c0cbdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:57.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:57.843+0000 7f8670990700 1 -- 172.21.15.98:0/3517521167 >> 172.21.15.98:0/3517521167 conn(0x7f866c0675b0 msgr2=0x7f866c0679b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:06:57.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:57.843+0000 7f8670990700 1 -- 172.21.15.98:0/3517521167 shutdown_connections 2023-10-06T05:06:57.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:57.843+0000 7f8670990700 1 -- 172.21.15.98:0/3517521167 wait complete. 2023-10-06T05:06:57.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:57.844+0000 7f8670990700 1 Processor -- start 2023-10-06T05:06:57.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:57.844+0000 7f8670990700 1 -- start start 2023-10-06T05:06:57.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:57.845+0000 7f8670990700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f866c06d5d0 0x7f866c154ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:57.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:57.845+0000 7f8670990700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f866c156f50 0x7f866c1682a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:57.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:57.845+0000 7f8670990700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f866c157320 0x7f866c0c5a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:57.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:57.845+0000 7f8670990700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f866c0c4c30 con 0x7f866c06d5d0 2023-10-06T05:06:57.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:57.845+0000 7f8670990700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f866c0c4020 con 0x7f866c156f50 2023-10-06T05:06:57.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:57.845+0000 7f8670990700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f866c0c4300 con 0x7f866c157320 2023-10-06T05:06:57.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:57.845+0000 7f866ad9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f866c06d5d0 0x7f866c154ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:57.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:57.845+0000 7f866ad9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f866c06d5d0 0x7f866c154ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54284/0 (socket says 172.21.15.98:54284) 2023-10-06T05:06:57.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:57.845+0000 7f866ad9d700 1 -- 172.21.15.98:0/3910524622 learned_addr learned my addr 172.21.15.98:0/3910524622 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:06:57.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:57.845+0000 7f866ad9d700 1 -- 172.21.15.98:0/3910524622 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f866c157320 msgr2=0x7f866c0c5a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:57.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:57.845+0000 7f866a59c700 1 --2- 172.21.15.98:0/3910524622 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f866c156f50 0x7f866c1682a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:57.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:57.845+0000 7f866ad9d700 1 --2- 172.21.15.98:0/3910524622 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f866c157320 0x7f866c0c5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:57.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:57.845+0000 7f866ad9d700 1 -- 172.21.15.98:0/3910524622 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f866c156f50 msgr2=0x7f866c1682a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:57.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:57.845+0000 7f866ad9d700 1 --2- 172.21.15.98:0/3910524622 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f866c156f50 0x7f866c1682a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:57.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:57.845+0000 7f866ad9d700 1 -- 172.21.15.98:0/3910524622 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8664009ae0 con 0x7f866c06d5d0 2023-10-06T05:06:57.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:57.845+0000 7f866a59c700 1 --2- 172.21.15.98:0/3910524622 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f866c156f50 0x7f866c1682a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:06:57.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:57.845+0000 7f866ad9d700 1 --2- 172.21.15.98:0/3910524622 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f866c06d5d0 0x7f866c154ff0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f865c00c810 tx=0x7f865c007c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:06:57.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:57.845+0000 7f8652ffd700 1 -- 172.21.15.98:0/3910524622 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f865c007300 con 0x7f866c06d5d0 2023-10-06T05:06:57.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:57.846+0000 7f8652ffd700 1 -- 172.21.15.98:0/3910524622 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f865c007480 con 0x7f866c06d5d0 2023-10-06T05:06:57.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:57.846+0000 7f8670990700 1 -- 172.21.15.98:0/3910524622 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f866c0c45e0 con 0x7f866c06d5d0 2023-10-06T05:06:57.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:57.846+0000 7f8652ffd700 1 -- 172.21.15.98:0/3910524622 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f865c004830 con 0x7f866c06d5d0 2023-10-06T05:06:57.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:57.846+0000 7f8670990700 1 -- 172.21.15.98:0/3910524622 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f866c1687e0 con 0x7f866c06d5d0 2023-10-06T05:06:57.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:57.846+0000 7f8670990700 1 -- 172.21.15.98:0/3910524622 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f866c0cb740 con 0x7f866c06d5d0 2023-10-06T05:06:57.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:57.847+0000 7f8652ffd700 1 -- 172.21.15.98:0/3910524622 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f865c004be0 con 0x7f866c06d5d0 2023-10-06T05:06:57.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:57.847+0000 7f8652ffd700 1 --2- 172.21.15.98:0/3910524622 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f8658066f30 0x7f86580693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:57.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:57.847+0000 7f8652ffd700 1 -- 172.21.15.98:0/3910524622 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f865c083970 con 0x7f866c06d5d0 2023-10-06T05:06:57.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:57.847+0000 7f866a59c700 1 --2- 172.21.15.98:0/3910524622 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f8658066f30 0x7f86580693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:57.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:57.848+0000 7f866a59c700 1 --2- 172.21.15.98:0/3910524622 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f8658066f30 0x7f86580693e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8660006f00 tx=0x7f8660008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:06:57.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:57.852+0000 7f8652ffd700 1 -- 172.21.15.98:0/3910524622 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f865c050a80 con 0x7f866c06d5d0 2023-10-06T05:06:58.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.046+0000 7f8670990700 1 -- 172.21.15.98:0/3910524622 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) v1 -- 0x7f866c0cdd20 con 0x7f866c06d5d0 2023-10-06T05:06:58.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.595+0000 7f8652ffd700 1 -- 172.21.15.98:0/3910524622 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v25) v1 ==== 157+0+0 (secure 0 0 0) 0x7f865c0180e0 con 0x7f866c06d5d0 2023-10-06T05:06:58.600 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.598+0000 7f8670990700 1 -- 172.21.15.98:0/3910524622 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f8658066f30 msgr2=0x7f86580693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:58.600 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.598+0000 7f8670990700 1 --2- 172.21.15.98:0/3910524622 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f8658066f30 0x7f86580693e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8660006f00 tx=0x7f8660008040 comp rx=0 tx=0).stop 2023-10-06T05:06:58.601 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.598+0000 7f8670990700 1 -- 172.21.15.98:0/3910524622 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f866c06d5d0 msgr2=0x7f866c154ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:58.601 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.598+0000 7f8670990700 1 --2- 172.21.15.98:0/3910524622 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f866c06d5d0 0x7f866c154ff0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f865c00c810 tx=0x7f865c007c50 comp rx=0 tx=0).stop 2023-10-06T05:06:58.601 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.598+0000 7f8670990700 1 -- 172.21.15.98:0/3910524622 shutdown_connections 2023-10-06T05:06:58.601 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.598+0000 7f8670990700 1 --2- 172.21.15.98:0/3910524622 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f866c157320 0x7f866c0c5a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:58.601 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.598+0000 7f8670990700 1 --2- 172.21.15.98:0/3910524622 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f8658066f30 0x7f86580693e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:58.601 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.598+0000 7f8670990700 1 --2- 172.21.15.98:0/3910524622 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f866c06d5d0 0x7f866c154ff0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:58.601 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.598+0000 7f8670990700 1 --2- 172.21.15.98:0/3910524622 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f866c156f50 0x7f866c1682a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:58.601 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.598+0000 7f8670990700 1 -- 172.21.15.98:0/3910524622 >> 172.21.15.98:0/3910524622 conn(0x7f866c0675b0 msgr2=0x7f866c0afe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:06:58.601 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.598+0000 7f8670990700 1 -- 172.21.15.98:0/3910524622 shutdown_connections 2023-10-06T05:06:58.601 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.598+0000 7f8670990700 1 -- 172.21.15.98:0/3910524622 wait complete. 2023-10-06T05:06:58.614 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2023-10-06T05:06:58.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.737+0000 7fbf4489c700 1 Processor -- start 2023-10-06T05:06:58.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.737+0000 7fbf4489c700 1 -- start start 2023-10-06T05:06:58.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.738+0000 7fbf4489c700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbf400b05d0 0x7fbf400cc710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:58.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.738+0000 7fbf4489c700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fbf400c9580 0x7fbf400c2f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:58.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.738+0000 7fbf4489c700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf400cd0c0 con 0x7fbf400ccc50 2023-10-06T05:06:58.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.738+0000 7fbf4489c700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf400cadd0 con 0x7fbf400c9580 2023-10-06T05:06:58.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.738+0000 7fbf4489c700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf401577d0 con 0x7fbf400b05d0 2023-10-06T05:06:58.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.738+0000 7fbf3ffff700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fbf400c9580 0x7fbf400c2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:58.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.738+0000 7fbf3f7fe700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbf400b05d0 0x7fbf400cc710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:58.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.738+0000 7fbf3effd700 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fbf400ccc50 0x7fbf400c8e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.98:37822/0 (socket says 172.21.15.98:37822) 2023-10-06T05:06:58.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.738+0000 7fbf3effd700 1 -- 172.21.15.98:0/2877344017 learned_addr learned my addr 172.21.15.98:0/2877344017 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:06:58.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.739+0000 7fbf3ffff700 1 -- 172.21.15.98:0/2877344017 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbf400b05d0 msgr2=0x7fbf400cc710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:06:58.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.739+0000 7fbf3ffff700 1 --2- 172.21.15.98:0/2877344017 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbf400b05d0 0x7fbf400cc710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:58.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.739+0000 7fbf3ffff700 1 -- 172.21.15.98:0/2877344017 >> v1:172.21.15.174:6789/0 conn(0x7fbf400ccc50 legacy=0x7fbf400c8e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:58.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.739+0000 7fbf3ffff700 1 -- 172.21.15.98:0/2877344017 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf400c9ae0 con 0x7fbf400c9580 2023-10-06T05:06:58.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.739+0000 7fbf3f7fe700 1 --2- 172.21.15.98:0/2877344017 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbf400b05d0 0x7fbf400cc710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T05:06:58.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.739+0000 7fbf3ffff700 1 --2- 172.21.15.98:0/2877344017 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf400c9580 0x7fbf400c2f40 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fbf380098e0 tx=0x7fbf380091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=878601463f6586ca server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:06:58.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.739+0000 7fbf3e7fc700 1 -- 172.21.15.98:0/2877344017 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf3800b640 con 0x7fbf400c9580 2023-10-06T05:06:58.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.739+0000 7fbf3e7fc700 1 -- 172.21.15.98:0/2877344017 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf3800b7a0 con 0x7fbf400c9580 2023-10-06T05:06:58.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.739+0000 7fbf3e7fc700 1 -- 172.21.15.98:0/2877344017 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf3800bb40 con 0x7fbf400c9580 2023-10-06T05:06:58.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.740+0000 7fbf4489c700 1 -- 172.21.15.98:0/2877344017 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf400c9580 msgr2=0x7fbf400c2f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:58.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.740+0000 7fbf4489c700 1 --2- 172.21.15.98:0/2877344017 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf400c9580 0x7fbf400c2f40 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fbf380098e0 tx=0x7fbf380091a0 comp rx=0 tx=0).stop 2023-10-06T05:06:58.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.740+0000 7fbf4489c700 1 -- 172.21.15.98:0/2877344017 shutdown_connections 2023-10-06T05:06:58.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.740+0000 7fbf4489c700 1 --2- 172.21.15.98:0/2877344017 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf400c9580 0x7fbf400c2f40 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:58.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.740+0000 7fbf4489c700 1 --2- 172.21.15.98:0/2877344017 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbf400b05d0 0x7fbf400cc710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:58.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.740+0000 7fbf4489c700 1 -- 172.21.15.98:0/2877344017 >> 172.21.15.98:0/2877344017 conn(0x7fbf40096a30 msgr2=0x7fbf40096e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:06:58.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.740+0000 7fbf4489c700 1 -- 172.21.15.98:0/2877344017 shutdown_connections 2023-10-06T05:06:58.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.740+0000 7fbf4489c700 1 -- 172.21.15.98:0/2877344017 wait complete. 2023-10-06T05:06:58.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.741+0000 7fbf4489c700 1 Processor -- start 2023-10-06T05:06:58.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.741+0000 7fbf4489c700 1 -- start start 2023-10-06T05:06:58.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.742+0000 7fbf4489c700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbf400b05d0 0x7fbf4013cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:58.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.742+0000 7fbf4489c700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf400c9580 0x7fbf400c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:58.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.742+0000 7fbf4489c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf400ccc50 0x7fbf400c4050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:58.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.742+0000 7fbf4489c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf400c8280 con 0x7fbf400ccc50 2023-10-06T05:06:58.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.742+0000 7fbf4489c700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf4013d310 con 0x7fbf400c9580 2023-10-06T05:06:58.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.742+0000 7fbf4489c700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf400c5720 con 0x7fbf400b05d0 2023-10-06T05:06:58.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.742+0000 7fbf3ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf400ccc50 0x7fbf400c4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:58.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.742+0000 7fbf3ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf400ccc50 0x7fbf400c4050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54292/0 (socket says 172.21.15.98:54292) 2023-10-06T05:06:58.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.742+0000 7fbf3ffff700 1 -- 172.21.15.98:0/3740914172 learned_addr learned my addr 172.21.15.98:0/3740914172 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:06:58.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.742+0000 7fbf3f7fe700 1 --2- 172.21.15.98:0/3740914172 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbf400b05d0 0x7fbf4013cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:58.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.742+0000 7fbf3effd700 1 --2- 172.21.15.98:0/3740914172 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf400c9580 0x7fbf400c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:58.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.742+0000 7fbf3ffff700 1 -- 172.21.15.98:0/3740914172 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbf400b05d0 msgr2=0x7fbf4013cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:58.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.742+0000 7fbf3ffff700 1 --2- 172.21.15.98:0/3740914172 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbf400b05d0 0x7fbf4013cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:58.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.742+0000 7fbf3ffff700 1 -- 172.21.15.98:0/3740914172 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf400c9580 msgr2=0x7fbf400c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:06:58.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.742+0000 7fbf3ffff700 1 --2- 172.21.15.98:0/3740914172 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf400c9580 0x7fbf400c51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:06:58.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.742+0000 7fbf3ffff700 1 -- 172.21.15.98:0/3740914172 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf30009a20 con 0x7fbf400ccc50 2023-10-06T05:06:58.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.742+0000 7fbf3effd700 1 --2- 172.21.15.98:0/3740914172 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf400c9580 0x7fbf400c51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:06:58.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.742+0000 7fbf3f7fe700 1 --2- 172.21.15.98:0/3740914172 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbf400b05d0 0x7fbf4013cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:06:58.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.742+0000 7fbf3ffff700 1 --2- 172.21.15.98:0/3740914172 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf400ccc50 0x7fbf400c4050 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fbf38009f80 tx=0x7fbf38017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:06:58.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.743+0000 7fbf277fe700 1 -- 172.21.15.98:0/3740914172 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf3801be30 con 0x7fbf400ccc50 2023-10-06T05:06:58.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.743+0000 7fbf277fe700 1 -- 172.21.15.98:0/3740914172 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf38011630 con 0x7fbf400ccc50 2023-10-06T05:06:58.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.743+0000 7fbf4489c700 1 -- 172.21.15.98:0/3740914172 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf38009420 con 0x7fbf400ccc50 2023-10-06T05:06:58.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.743+0000 7fbf277fe700 1 -- 172.21.15.98:0/3740914172 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf380119f0 con 0x7fbf400ccc50 2023-10-06T05:06:58.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.743+0000 7fbf4489c700 1 -- 172.21.15.98:0/3740914172 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf400c7fa0 con 0x7fbf400ccc50 2023-10-06T05:06:58.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.744+0000 7fbf4489c700 1 -- 172.21.15.98:0/3740914172 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf4005f230 con 0x7fbf400ccc50 2023-10-06T05:06:58.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.744+0000 7fbf277fe700 1 -- 172.21.15.98:0/3740914172 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fbf3802e440 con 0x7fbf400ccc50 2023-10-06T05:06:58.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.744+0000 7fbf277fe700 1 --2- 172.21.15.98:0/3740914172 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fbf2c066f30 0x7fbf2c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:06:58.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.744+0000 7fbf277fe700 1 -- 172.21.15.98:0/3740914172 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6284+0+0 (secure 0 0 0) 0x7fbf38086400 con 0x7fbf400ccc50 2023-10-06T05:06:58.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.744+0000 7fbf3f7fe700 1 --2- 172.21.15.98:0/3740914172 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fbf2c066f30 0x7fbf2c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:06:58.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.745+0000 7fbf3f7fe700 1 --2- 172.21.15.98:0/3740914172 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fbf2c066f30 0x7fbf2c0693e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fbf300099f0 tx=0x7fbf3000b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:06:58.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.749+0000 7fbf277fe700 1 -- 172.21.15.98:0/3740914172 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbf38053430 con 0x7fbf400ccc50 2023-10-06T05:06:58.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:06:58.932+0000 7fbf4489c700 1 -- 172.21.15.98:0/3740914172 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fbf400975d0 con 0x7fbf400ccc50 2023-10-06T05:07:00.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.747+0000 7fbf277fe700 1 -- 172.21.15.98:0/3740914172 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v27) v1 ==== 228+0+0 (secure 0 0 0) 0x7fbf38056a50 con 0x7fbf400ccc50 2023-10-06T05:07:00.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.752+0000 7fbf4489c700 1 -- 172.21.15.98:0/3740914172 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fbf2c066f30 msgr2=0x7fbf2c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:00.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.752+0000 7fbf4489c700 1 --2- 172.21.15.98:0/3740914172 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fbf2c066f30 0x7fbf2c0693e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fbf300099f0 tx=0x7fbf3000b040 comp rx=0 tx=0).stop 2023-10-06T05:07:00.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.752+0000 7fbf4489c700 1 -- 172.21.15.98:0/3740914172 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf400ccc50 msgr2=0x7fbf400c4050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:00.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.752+0000 7fbf4489c700 1 --2- 172.21.15.98:0/3740914172 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf400ccc50 0x7fbf400c4050 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fbf38009f80 tx=0x7fbf38017000 comp rx=0 tx=0).stop 2023-10-06T05:07:00.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.754+0000 7fbf4489c700 1 -- 172.21.15.98:0/3740914172 shutdown_connections 2023-10-06T05:07:00.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.754+0000 7fbf4489c700 1 --2- 172.21.15.98:0/3740914172 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fbf2c066f30 0x7fbf2c0693e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:00.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.754+0000 7fbf4489c700 1 --2- 172.21.15.98:0/3740914172 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbf400ccc50 0x7fbf400c4050 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:00.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.754+0000 7fbf4489c700 1 --2- 172.21.15.98:0/3740914172 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbf400b05d0 0x7fbf4013cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:00.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.754+0000 7fbf4489c700 1 --2- 172.21.15.98:0/3740914172 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf400c9580 0x7fbf400c51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:00.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.754+0000 7fbf4489c700 1 -- 172.21.15.98:0/3740914172 >> 172.21.15.98:0/3740914172 conn(0x7fbf40096a30 msgr2=0x7fbf40153a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:00.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.754+0000 7fbf4489c700 1 -- 172.21.15.98:0/3740914172 shutdown_connections 2023-10-06T05:07:00.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.754+0000 7fbf4489c700 1 -- 172.21.15.98:0/3740914172 wait complete. 2023-10-06T05:07:00.758 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'cephfs_data_ec' created 2023-10-06T05:07:00.771 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2023-10-06T05:07:00.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.892+0000 7f59e3d76700 1 Processor -- start 2023-10-06T05:07:00.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.893+0000 7f59e3d76700 1 -- start start 2023-10-06T05:07:00.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.893+0000 7f59e3d76700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f59dc15d350 0x7f59dc157740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:00.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.893+0000 7f59e3d76700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f59dc159f80 0x7f59dc15a400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:00.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.893+0000 7f59e3d76700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59dc0cdd40 con 0x7f59dc15d720 2023-10-06T05:07:00.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.893+0000 7f59e3d76700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59dc0b2c80 con 0x7f59dc15d350 2023-10-06T05:07:00.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.893+0000 7f59e3d76700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59dc15b0b0 con 0x7f59dc159f80 2023-10-06T05:07:00.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.893+0000 7f59e1311700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f59dc15d720 0x7f59dc0cd630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:57020/0 (socket says 172.21.15.98:57020) 2023-10-06T05:07:00.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.893+0000 7f59e1311700 1 -- 172.21.15.98:0/2937237779 learned_addr learned my addr 172.21.15.98:0/2937237779 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:07:00.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.894+0000 7f59e2313700 1 --2- 172.21.15.98:0/2937237779 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f59dc159f80 0x7f59dc15a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:00.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.894+0000 7f59e1b12700 1 --2- 172.21.15.98:0/2937237779 >> v2:172.21.15.174:3300/0 conn(0x7f59dc15d350 0x7f59dc157740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:00.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.894+0000 7f59e0b10700 1 -- 172.21.15.98:0/2937237779 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3912544371 0 0) 0x7f59dc0cdd40 con 0x7f59dc15d720 2023-10-06T05:07:00.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.894+0000 7f59e0b10700 1 -- 172.21.15.98:0/2937237779 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59c8003570 con 0x7f59dc15d720 2023-10-06T05:07:00.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.894+0000 7f59e0b10700 1 -- 172.21.15.98:0/2937237779 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 205958023 0 0) 0x7f59c8003570 con 0x7f59dc15d720 2023-10-06T05:07:00.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.894+0000 7f59e1b12700 1 -- 172.21.15.98:0/2937237779 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f59dc159f80 msgr2=0x7f59dc15a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:00.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.894+0000 7f59e1b12700 1 --2- 172.21.15.98:0/2937237779 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f59dc159f80 0x7f59dc15a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:00.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.894+0000 7f59e1b12700 1 -- 172.21.15.98:0/2937237779 >> v1:172.21.15.98:6789/0 conn(0x7f59dc15d720 legacy=0x7f59dc0cd630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:00.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.894+0000 7f59e1b12700 1 -- 172.21.15.98:0/2937237779 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59dc15b390 con 0x7f59dc15d350 2023-10-06T05:07:00.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.894+0000 7f59e2313700 1 --2- 172.21.15.98:0/2937237779 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f59dc159f80 0x7f59dc15a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T05:07:00.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.894+0000 7f59e1b12700 1 --2- 172.21.15.98:0/2937237779 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59dc15d350 0x7f59dc157740 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f59d80098d0 tx=0x7f59d8009120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c1d4643fb6d76ec server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:00.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.894+0000 7f59e0b10700 1 -- 172.21.15.98:0/2937237779 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59cc001470 con 0x7f59dc15d350 2023-10-06T05:07:00.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.894+0000 7f59e0b10700 1 -- 172.21.15.98:0/2937237779 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59d8011420 con 0x7f59dc15d350 2023-10-06T05:07:00.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.894+0000 7f59e0b10700 1 -- 172.21.15.98:0/2937237779 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59d80117c0 con 0x7f59dc15d350 2023-10-06T05:07:00.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.895+0000 7f59e3d76700 1 -- 172.21.15.98:0/2937237779 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59dc15d350 msgr2=0x7f59dc157740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:00.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.895+0000 7f59e3d76700 1 --2- 172.21.15.98:0/2937237779 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59dc15d350 0x7f59dc157740 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f59d80098d0 tx=0x7f59d8009120 comp rx=0 tx=0).stop 2023-10-06T05:07:00.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.895+0000 7f59e3d76700 1 -- 172.21.15.98:0/2937237779 shutdown_connections 2023-10-06T05:07:00.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.895+0000 7f59e3d76700 1 --2- 172.21.15.98:0/2937237779 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f59dc159f80 0x7f59dc15a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:00.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.895+0000 7f59e3d76700 1 --2- 172.21.15.98:0/2937237779 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59dc15d350 0x7f59dc157740 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:00.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.895+0000 7f59e3d76700 1 -- 172.21.15.98:0/2937237779 >> 172.21.15.98:0/2937237779 conn(0x7f59dc068310 msgr2=0x7f59dc068710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:00.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.895+0000 7f59e3d76700 1 -- 172.21.15.98:0/2937237779 shutdown_connections 2023-10-06T05:07:00.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.895+0000 7f59e3d76700 1 -- 172.21.15.98:0/2937237779 wait complete. 2023-10-06T05:07:00.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.896+0000 7f59e3d76700 1 Processor -- start 2023-10-06T05:07:00.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.896+0000 7f59e3d76700 1 -- start start 2023-10-06T05:07:00.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.897+0000 7f59e3d76700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f59dc159f80 0x7f59dc0cfe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:00.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.897+0000 7f59e3d76700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59dc15d350 0x7f59dc0c7570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:00.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.897+0000 7f59e3d76700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59dc15d720 0x7f59dc148ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:00.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.897+0000 7f59e3d76700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59dc0c8920 con 0x7f59dc15d350 2023-10-06T05:07:00.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.897+0000 7f59e3d76700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59dc0c7ab0 con 0x7f59dc15d720 2023-10-06T05:07:00.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.897+0000 7f59e3d76700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59dc0c7d10 con 0x7f59dc159f80 2023-10-06T05:07:00.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.897+0000 7f59e1311700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59dc15d350 0x7f59dc0c7570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:00.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.897+0000 7f59e1311700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59dc15d350 0x7f59dc0c7570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47726/0 (socket says 172.21.15.98:47726) 2023-10-06T05:07:00.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.897+0000 7f59e1311700 1 -- 172.21.15.98:0/3835930405 learned_addr learned my addr 172.21.15.98:0/3835930405 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:00.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.897+0000 7f59e1b12700 1 --2- 172.21.15.98:0/3835930405 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f59dc159f80 0x7f59dc0cfe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:00.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.897+0000 7f59e2313700 1 --2- 172.21.15.98:0/3835930405 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59dc15d720 0x7f59dc148ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:00.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.897+0000 7f59e1311700 1 -- 172.21.15.98:0/3835930405 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f59dc159f80 msgr2=0x7f59dc0cfe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:00.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.897+0000 7f59e1311700 1 --2- 172.21.15.98:0/3835930405 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f59dc159f80 0x7f59dc0cfe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:00.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.897+0000 7f59e1311700 1 -- 172.21.15.98:0/3835930405 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59dc15d720 msgr2=0x7f59dc148ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:00.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.897+0000 7f59e1311700 1 --2- 172.21.15.98:0/3835930405 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59dc15d720 0x7f59dc148ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:00.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.897+0000 7f59e1311700 1 -- 172.21.15.98:0/3835930405 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59d4009a20 con 0x7f59dc15d350 2023-10-06T05:07:00.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.897+0000 7f59e1311700 1 --2- 172.21.15.98:0/3835930405 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59dc15d350 0x7f59dc0c7570 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f59cc00cb90 tx=0x7f59cc001a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:00.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.897+0000 7f59d1ffb700 1 -- 172.21.15.98:0/3835930405 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59cc007150 con 0x7f59dc15d350 2023-10-06T05:07:00.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.898+0000 7f59d1ffb700 1 -- 172.21.15.98:0/3835930405 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59cc0072d0 con 0x7f59dc15d350 2023-10-06T05:07:00.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.898+0000 7f59d1ffb700 1 -- 172.21.15.98:0/3835930405 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59cc007690 con 0x7f59dc15d350 2023-10-06T05:07:00.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.898+0000 7f59e3d76700 1 -- 172.21.15.98:0/3835930405 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59d80093c0 con 0x7f59dc15d350 2023-10-06T05:07:00.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.898+0000 7f59e3d76700 1 -- 172.21.15.98:0/3835930405 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59dc0c82d0 con 0x7f59dc15d350 2023-10-06T05:07:00.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.899+0000 7f59e3d76700 1 -- 172.21.15.98:0/3835930405 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59dc064a90 con 0x7f59dc15d350 2023-10-06T05:07:00.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.899+0000 7f59d1ffb700 1 -- 172.21.15.98:0/3835930405 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f59cc004a40 con 0x7f59dc15d350 2023-10-06T05:07:00.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.899+0000 7f59d1ffb700 1 --2- 172.21.15.98:0/3835930405 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f59c0066f30 0x7f59c00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:00.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.899+0000 7f59d1ffb700 1 -- 172.21.15.98:0/3835930405 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6777+0+0 (secure 0 0 0) 0x7f59cc083d30 con 0x7f59dc15d350 2023-10-06T05:07:00.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.899+0000 7f59e1b12700 1 --2- 172.21.15.98:0/3835930405 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f59c0066f30 0x7f59c00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:00.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.900+0000 7f59e1b12700 1 --2- 172.21.15.98:0/3835930405 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f59c0066f30 0x7f59c00693e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f59d80030c0 tx=0x7f59d8016000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:00.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:00.904+0000 7f59d1ffb700 1 -- 172.21.15.98:0/3835930405 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f59cc050c10 con 0x7f59dc15d350 2023-10-06T05:07:01.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.086+0000 7f59e3d76700 1 -- 172.21.15.98:0/3835930405 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) v1 -- 0x7f59dc15b670 con 0x7f59dc15d350 2023-10-06T05:07:01.744 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.742+0000 7f59d1ffb700 1 -- 172.21.15.98:0/3835930405 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 5 allow_ec_overwrites to true v28) v1 ==== 169+0+0 (secure 0 0 0) 0x7f59cc018070 con 0x7f59dc15d350 2023-10-06T05:07:01.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.750+0000 7f59e3d76700 1 -- 172.21.15.98:0/3835930405 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f59c0066f30 msgr2=0x7f59c00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:01.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.750+0000 7f59e3d76700 1 --2- 172.21.15.98:0/3835930405 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f59c0066f30 0x7f59c00693e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f59d80030c0 tx=0x7f59d8016000 comp rx=0 tx=0).stop 2023-10-06T05:07:01.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.750+0000 7f59e3d76700 1 -- 172.21.15.98:0/3835930405 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59dc15d350 msgr2=0x7f59dc0c7570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:01.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.750+0000 7f59e3d76700 1 --2- 172.21.15.98:0/3835930405 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59dc15d350 0x7f59dc0c7570 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f59cc00cb90 tx=0x7f59cc001a30 comp rx=0 tx=0).stop 2023-10-06T05:07:01.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.753+0000 7f59e3d76700 1 -- 172.21.15.98:0/3835930405 shutdown_connections 2023-10-06T05:07:01.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.753+0000 7f59e3d76700 1 --2- 172.21.15.98:0/3835930405 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59dc15d720 0x7f59dc148ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:01.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.753+0000 7f59e3d76700 1 --2- 172.21.15.98:0/3835930405 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f59dc159f80 0x7f59dc0cfe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:01.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.753+0000 7f59e3d76700 1 --2- 172.21.15.98:0/3835930405 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f59c0066f30 0x7f59c00693e0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:01.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.753+0000 7f59e3d76700 1 --2- 172.21.15.98:0/3835930405 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59dc15d350 0x7f59dc0c7570 secure :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f59cc00cb90 tx=0x7f59cc001a30 comp rx=0 tx=0).stop 2023-10-06T05:07:01.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.753+0000 7f59e3d76700 1 -- 172.21.15.98:0/3835930405 >> 172.21.15.98:0/3835930405 conn(0x7f59dc068310 msgr2=0x7f59dc154f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:01.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.756+0000 7f59e3d76700 1 -- 172.21.15.98:0/3835930405 shutdown_connections 2023-10-06T05:07:01.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.757+0000 7f59e3d76700 1 -- 172.21.15.98:0/3835930405 wait complete. 2023-10-06T05:07:01.764 INFO:teuthology.orchestra.run.smithi098.stderr:set pool 5 allow_ec_overwrites to true 2023-10-06T05:07:01.777 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2023-10-06T05:07:01.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.891+0000 7ff0d7ef6700 1 Processor -- start 2023-10-06T05:07:01.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.891+0000 7ff0d7ef6700 1 -- start start 2023-10-06T05:07:01.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.892+0000 7ff0d7ef6700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff0d006a050 0x7ff0d006b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:01.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.892+0000 7ff0d7ef6700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0d015d130 con 0x7ff0d015a5b0 2023-10-06T05:07:01.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.892+0000 7ff0d7ef6700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0d015d380 con 0x7ff0d006a420 2023-10-06T05:07:01.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.892+0000 7ff0d7ef6700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d015d660 con 0x7ff0d006a050 2023-10-06T05:07:01.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.892+0000 7ff0d6493700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7ff0d015a5b0 0x7ff0d00686c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:57026/0 (socket says 172.21.15.98:57026) 2023-10-06T05:07:01.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.892+0000 7ff0d6493700 1 -- 172.21.15.98:0/1663188055 learned_addr learned my addr 172.21.15.98:0/1663188055 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:07:01.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.892+0000 7ff0d5c92700 1 --2- 172.21.15.98:0/1663188055 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff0d006a050 0x7ff0d006b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:01.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.893+0000 7ff0d4c90700 1 -- 172.21.15.98:0/1663188055 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 617932006 0 0) 0x7ff0d015d130 con 0x7ff0d015a5b0 2023-10-06T05:07:01.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.893+0000 7ff0d4c90700 1 -- 172.21.15.98:0/1663188055 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0bc003570 con 0x7ff0d015a5b0 2023-10-06T05:07:01.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.893+0000 7ff0d4c90700 1 -- 172.21.15.98:0/1663188055 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2311601105 0 0) 0x7ff0d015d380 con 0x7ff0d006a420 2023-10-06T05:07:01.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.893+0000 7ff0d4c90700 1 -- 172.21.15.98:0/1663188055 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0d015d130 con 0x7ff0d006a420 2023-10-06T05:07:01.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.893+0000 7ff0d4c90700 1 -- 172.21.15.98:0/1663188055 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1625429102 0 0) 0x7ff0bc003570 con 0x7ff0d015a5b0 2023-10-06T05:07:01.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.893+0000 7ff0d4c90700 1 -- 172.21.15.98:0/1663188055 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff0d006a050 msgr2=0x7ff0d006b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:01.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.893+0000 7ff0d4c90700 1 --2- 172.21.15.98:0/1663188055 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff0d006a050 0x7ff0d006b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:01.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.893+0000 7ff0d4c90700 1 -- 172.21.15.98:0/1663188055 >> v1:172.21.15.174:6789/0 conn(0x7ff0d006a420 legacy=0x7ff0d0159ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:01.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.893+0000 7ff0d4c90700 1 -- 172.21.15.98:0/1663188055 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0d015d940 con 0x7ff0d015a5b0 2023-10-06T05:07:01.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.893+0000 7ff0d4c90700 1 -- 172.21.15.98:0/1663188055 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1543991876 0 0) 0x7ff0c80017f0 con 0x7ff0d015a5b0 2023-10-06T05:07:01.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.893+0000 7ff0d4c90700 1 -- 172.21.15.98:0/1663188055 >> v1:172.21.15.98:6789/0 conn(0x7ff0d015a5b0 legacy=0x7ff0d00686c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:01.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.893+0000 7ff0d4c90700 1 --2- 172.21.15.98:0/1663188055 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff0bc003c20 0x7ff0bc004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:01.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.893+0000 7ff0d4c90700 1 --2- 172.21.15.98:0/1663188055 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff0d006a050 0x7ff0bc004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:01.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.893+0000 7ff0d4c90700 1 --2- 172.21.15.98:0/1663188055 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0bc004d50 0x7ff0bc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:01.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.893+0000 7ff0d4c90700 1 -- 172.21.15.98:0/1663188055 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d015d660 con 0x7ff0bc004d50 2023-10-06T05:07:01.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.893+0000 7ff0d4c90700 1 -- 172.21.15.98:0/1663188055 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0bc00bc40 con 0x7ff0bc003c20 2023-10-06T05:07:01.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.893+0000 7ff0d4c90700 1 -- 172.21.15.98:0/1663188055 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0bc00be00 con 0x7ff0d006a050 2023-10-06T05:07:01.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.893+0000 7ff0d6493700 1 --2- 172.21.15.98:0/1663188055 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0bc004d50 0x7ff0bc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:01.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.893+0000 7ff0d5c92700 1 --2- 172.21.15.98:0/1663188055 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff0bc003c20 0x7ff0bc004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:01.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.893+0000 7ff0d5491700 1 --2- 172.21.15.98:0/1663188055 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff0d006a050 0x7ff0bc004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:01.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.893+0000 7ff0d6493700 1 -- 172.21.15.98:0/1663188055 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff0d006a050 msgr2=0x7ff0bc004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:01.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.893+0000 7ff0d6493700 1 --2- 172.21.15.98:0/1663188055 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff0d006a050 0x7ff0bc004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:01.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.893+0000 7ff0d6493700 1 -- 172.21.15.98:0/1663188055 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff0bc003c20 msgr2=0x7ff0bc004010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:01.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.893+0000 7ff0d6493700 1 --2- 172.21.15.98:0/1663188055 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff0bc003c20 0x7ff0bc004010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:01.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.893+0000 7ff0d6493700 1 -- 172.21.15.98:0/1663188055 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0bc003570 con 0x7ff0bc004d50 2023-10-06T05:07:01.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.893+0000 7ff0d6493700 1 --2- 172.21.15.98:0/1663188055 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0bc004d50 0x7ff0bc0051c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff0c800a800 tx=0x7ff0c80020c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f19ee03424bffa9 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:01.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.894+0000 7ff0d4c90700 1 -- 172.21.15.98:0/1663188055 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0c80071a0 con 0x7ff0bc004d50 2023-10-06T05:07:01.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.894+0000 7ff0d4c90700 1 -- 172.21.15.98:0/1663188055 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0c8007300 con 0x7ff0bc004d50 2023-10-06T05:07:01.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.894+0000 7ff0d4c90700 1 -- 172.21.15.98:0/1663188055 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0c80076c0 con 0x7ff0bc004d50 2023-10-06T05:07:01.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.894+0000 7ff0d7ef6700 1 -- 172.21.15.98:0/1663188055 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0bc004d50 msgr2=0x7ff0bc0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:01.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.894+0000 7ff0d7ef6700 1 --2- 172.21.15.98:0/1663188055 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0bc004d50 0x7ff0bc0051c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff0c800a800 tx=0x7ff0c80020c0 comp rx=0 tx=0).stop 2023-10-06T05:07:01.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.894+0000 7ff0d6493700 1 -- 172.21.15.98:0/1663188055 reap_dead start 2023-10-06T05:07:01.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.894+0000 7ff0d7ef6700 1 -- 172.21.15.98:0/1663188055 shutdown_connections 2023-10-06T05:07:01.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.894+0000 7ff0d7ef6700 1 -- 172.21.15.98:0/1663188055 >> 172.21.15.98:0/1663188055 conn(0x7ff0d00ce2f0 msgr2=0x7ff0d00ce6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:01.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.894+0000 7ff0d7ef6700 1 -- 172.21.15.98:0/1663188055 shutdown_connections 2023-10-06T05:07:01.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.894+0000 7ff0d7ef6700 1 -- 172.21.15.98:0/1663188055 wait complete. 2023-10-06T05:07:01.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.895+0000 7ff0d7ef6700 1 Processor -- start 2023-10-06T05:07:01.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.895+0000 7ff0d7ef6700 1 -- start start 2023-10-06T05:07:01.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.896+0000 7ff0d7ef6700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff0bc004d50 0x7ff0d00b1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:01.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.896+0000 7ff0d7ef6700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff0d0155800 0x7ff0d015c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:01.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.896+0000 7ff0d7ef6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0d00b3450 0x7ff0d00b38c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:01.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.896+0000 7ff0d7ef6700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d00adb00 con 0x7ff0d00b3450 2023-10-06T05:07:01.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.896+0000 7ff0d7ef6700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d00b3e00 con 0x7ff0d0155800 2023-10-06T05:07:01.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.896+0000 7ff0d7ef6700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d00c33a0 con 0x7ff0bc004d50 2023-10-06T05:07:01.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.896+0000 7ff0d6493700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0d00b3450 0x7ff0d00b38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:01.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.896+0000 7ff0d6493700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0d00b3450 0x7ff0d00b38c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47740/0 (socket says 172.21.15.98:47740) 2023-10-06T05:07:01.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.896+0000 7ff0d6493700 1 -- 172.21.15.98:0/1268585455 learned_addr learned my addr 172.21.15.98:0/1268585455 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:01.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.896+0000 7ff0d5491700 1 --2- 172.21.15.98:0/1268585455 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff0d0155800 0x7ff0d015c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:01.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.896+0000 7ff0d5c92700 1 --2- 172.21.15.98:0/1268585455 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff0bc004d50 0x7ff0d00b1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:01.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.896+0000 7ff0d6493700 1 -- 172.21.15.98:0/1268585455 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff0bc004d50 msgr2=0x7ff0d00b1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:01.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.896+0000 7ff0d6493700 1 --2- 172.21.15.98:0/1268585455 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff0bc004d50 0x7ff0d00b1a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:01.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.896+0000 7ff0d6493700 1 -- 172.21.15.98:0/1268585455 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff0d0155800 msgr2=0x7ff0d015c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:01.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.896+0000 7ff0d6493700 1 --2- 172.21.15.98:0/1268585455 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff0d0155800 0x7ff0d015c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:01.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.896+0000 7ff0d6493700 1 -- 172.21.15.98:0/1268585455 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0d00b2090 con 0x7ff0d00b3450 2023-10-06T05:07:01.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.896+0000 7ff0d5491700 1 --2- 172.21.15.98:0/1268585455 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff0d0155800 0x7ff0d015c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:07:01.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.896+0000 7ff0d5c92700 1 --2- 172.21.15.98:0/1268585455 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff0bc004d50 0x7ff0d00b1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:07:01.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.897+0000 7ff0d6493700 1 --2- 172.21.15.98:0/1268585455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0d00b3450 0x7ff0d00b38c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff0c8011000 tx=0x7ff0c8016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:01.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.897+0000 7ff0c5ffb700 1 -- 172.21.15.98:0/1268585455 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0c8025070 con 0x7ff0d00b3450 2023-10-06T05:07:01.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.897+0000 7ff0c5ffb700 1 -- 172.21.15.98:0/1268585455 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0c800ae20 con 0x7ff0d00b3450 2023-10-06T05:07:01.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.897+0000 7ff0d7ef6700 1 -- 172.21.15.98:0/1268585455 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0d00b2370 con 0x7ff0d00b3450 2023-10-06T05:07:01.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.897+0000 7ff0c5ffb700 1 -- 172.21.15.98:0/1268585455 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0c80020c0 con 0x7ff0d00b3450 2023-10-06T05:07:01.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.897+0000 7ff0d7ef6700 1 -- 172.21.15.98:0/1268585455 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0d00c3640 con 0x7ff0d00b3450 2023-10-06T05:07:01.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.898+0000 7ff0d7ef6700 1 -- 172.21.15.98:0/1268585455 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0d00c3830 con 0x7ff0d00b3450 2023-10-06T05:07:01.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.898+0000 7ff0c5ffb700 1 -- 172.21.15.98:0/1268585455 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7ff0c8007470 con 0x7ff0d00b3450 2023-10-06T05:07:01.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.899+0000 7ff0c5ffb700 1 --2- 172.21.15.98:0/1268585455 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7ff0b4066f30 0x7ff0b40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:01.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.899+0000 7ff0c5ffb700 1 -- 172.21.15.98:0/1268585455 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6777+0+0 (secure 0 0 0) 0x7ff0c8086890 con 0x7ff0d00b3450 2023-10-06T05:07:01.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.899+0000 7ff0d5c92700 1 --2- 172.21.15.98:0/1268585455 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7ff0b4066f30 0x7ff0b40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:01.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.899+0000 7ff0d5c92700 1 --2- 172.21.15.98:0/1268585455 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7ff0b4066f30 0x7ff0b40693e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc0093f0 tx=0x7ff0cc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:01.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:01.903+0000 7ff0c5ffb700 1 -- 172.21.15.98:0/1268585455 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff0c8053630 con 0x7ff0d00b3450 2023-10-06T05:07:02.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:02.130+0000 7ff0d7ef6700 1 -- 172.21.15.98:0/1268585455 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) v1 -- 0x7ff0d0064a90 con 0x7ff0d00b3450 2023-10-06T05:07:02.691 INFO:tasks.ceph.mds.c.smithi098.stderr:2023-10-06T05:07:02.689+0000 7faba9a8f700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-06T05:07:02.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:02.759+0000 7ff0c5ffb700 1 -- 172.21.15.98:0/1268585455 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 5 to fsmap v6) v1 ==== 137+0+0 (secure 0 0 0) 0x7ff0c80180b0 con 0x7ff0d00b3450 2023-10-06T05:07:02.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:02.762+0000 7ff0d7ef6700 1 -- 172.21.15.98:0/1268585455 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7ff0b4066f30 msgr2=0x7ff0b40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:02.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:02.762+0000 7ff0d7ef6700 1 --2- 172.21.15.98:0/1268585455 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7ff0b4066f30 0x7ff0b40693e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc0093f0 tx=0x7ff0cc008040 comp rx=0 tx=0).stop 2023-10-06T05:07:02.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:02.762+0000 7ff0d7ef6700 1 -- 172.21.15.98:0/1268585455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0d00b3450 msgr2=0x7ff0d00b38c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:02.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:02.762+0000 7ff0d7ef6700 1 --2- 172.21.15.98:0/1268585455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0d00b3450 0x7ff0d00b38c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff0c8011000 tx=0x7ff0c8016000 comp rx=0 tx=0).stop 2023-10-06T05:07:02.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:02.762+0000 7ff0d7ef6700 1 -- 172.21.15.98:0/1268585455 shutdown_connections 2023-10-06T05:07:02.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:02.762+0000 7ff0d7ef6700 1 --2- 172.21.15.98:0/1268585455 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7ff0b4066f30 0x7ff0b40693e0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:02.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:02.762+0000 7ff0d7ef6700 1 --2- 172.21.15.98:0/1268585455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0d00b3450 0x7ff0d00b38c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:02.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:02.762+0000 7ff0d7ef6700 1 --2- 172.21.15.98:0/1268585455 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff0bc004d50 0x7ff0d00b1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:02.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:02.762+0000 7ff0d7ef6700 1 --2- 172.21.15.98:0/1268585455 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff0d0155800 0x7ff0d015c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:02.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:02.762+0000 7ff0d7ef6700 1 -- 172.21.15.98:0/1268585455 >> 172.21.15.98:0/1268585455 conn(0x7ff0d00ce2f0 msgr2=0x7ff0d0069840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:02.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:02.762+0000 7ff0d7ef6700 1 -- 172.21.15.98:0/1268585455 shutdown_connections 2023-10-06T05:07:02.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:02.762+0000 7ff0d7ef6700 1 -- 172.21.15.98:0/1268585455 wait complete. 2023-10-06T05:07:02.768 INFO:teuthology.orchestra.run.smithi098.stderr:added data pool 5 to fsmap 2023-10-06T05:07:02.776 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-06T05:07:02.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:02.888+0000 7fa1f1586700 1 Processor -- start 2023-10-06T05:07:02.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:02.888+0000 7fa1f1586700 1 -- start start 2023-10-06T05:07:02.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:02.889+0000 7fa1f1586700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa1ec0adb60 0x7fa1ec0ce190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:02.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:02.889+0000 7fa1f1586700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fa1ec0ce6d0 0x7fa1ec15aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:02.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:02.889+0000 7fa1f1586700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1ec157de0 con 0x7fa1ec0ca370 2023-10-06T05:07:02.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:02.889+0000 7fa1f1586700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1ec0cbb90 con 0x7fa1ec0adb60 2023-10-06T05:07:02.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:02.889+0000 7fa1f1586700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1ec0ceb10 con 0x7fa1ec0ce6d0 2023-10-06T05:07:02.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:02.889+0000 7fa1e27fc700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa1ec0adb60 0x7fa1ec0ce190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:02.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:02.889+0000 7fa1e27fc700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa1ec0adb60 0x7fa1ec0ce190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47744/0 (socket says 172.21.15.98:47744) 2023-10-06T05:07:02.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:02.889+0000 7fa1e27fc700 1 -- 172.21.15.98:0/1727639289 learned_addr learned my addr 172.21.15.98:0/1727639289 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:02.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:02.889+0000 7fa1eb7fe700 1 --2- 172.21.15.98:0/1727639289 >> v2:172.21.15.174:3300/0 conn(0x7fa1ec0ce6d0 0x7fa1ec15aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:02.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:02.890+0000 7fa1e27fc700 1 -- 172.21.15.98:0/1727639289 >> v2:172.21.15.174:3300/0 conn(0x7fa1ec0ce6d0 msgr2=0x7fa1ec15aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:02.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:02.890+0000 7fa1e27fc700 1 --2- 172.21.15.98:0/1727639289 >> v2:172.21.15.174:3300/0 conn(0x7fa1ec0ce6d0 0x7fa1ec15aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:02.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:02.890+0000 7fa1e27fc700 1 -- 172.21.15.98:0/1727639289 >> v1:172.21.15.174:6789/0 conn(0x7fa1ec0ca370 legacy=0x7fa1ec158270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:02.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:02.890+0000 7fa1e27fc700 1 -- 172.21.15.98:0/1727639289 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1ec0cb5d0 con 0x7fa1ec0adb60 2023-10-06T05:07:02.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:02.890+0000 7fa1e27fc700 1 --2- 172.21.15.98:0/1727639289 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1ec0adb60 0x7fa1ec0ce190 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec1554c0 tx=0x7fa1d4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13f64e31cf368695 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:02.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:02.890+0000 7fa1ea7fc700 1 -- 172.21.15.98:0/1727639289 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1d400b860 con 0x7fa1ec0adb60 2023-10-06T05:07:02.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:02.890+0000 7fa1ea7fc700 1 -- 172.21.15.98:0/1727639289 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1d400b9c0 con 0x7fa1ec0adb60 2023-10-06T05:07:02.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:02.890+0000 7fa1ea7fc700 1 -- 172.21.15.98:0/1727639289 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1d400bd60 con 0x7fa1ec0adb60 2023-10-06T05:07:02.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:02.890+0000 7fa1f1586700 1 -- 172.21.15.98:0/1727639289 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1ec0adb60 msgr2=0x7fa1ec0ce190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:02.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:02.890+0000 7fa1f1586700 1 --2- 172.21.15.98:0/1727639289 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1ec0adb60 0x7fa1ec0ce190 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec1554c0 tx=0x7fa1d4009250 comp rx=0 tx=0).stop 2023-10-06T05:07:02.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:02.890+0000 7fa1f1586700 1 -- 172.21.15.98:0/1727639289 shutdown_connections 2023-10-06T05:07:02.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:02.890+0000 7fa1f1586700 1 --2- 172.21.15.98:0/1727639289 >> v2:172.21.15.174:3300/0 conn(0x7fa1ec0ce6d0 0x7fa1ec15aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:02.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:02.890+0000 7fa1f1586700 1 --2- 172.21.15.98:0/1727639289 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1ec0adb60 0x7fa1ec0ce190 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:02.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:02.890+0000 7fa1f1586700 1 -- 172.21.15.98:0/1727639289 >> 172.21.15.98:0/1727639289 conn(0x7fa1ec0707d0 msgr2=0x7fa1ec070bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:02.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:02.891+0000 7fa1f1586700 1 -- 172.21.15.98:0/1727639289 shutdown_connections 2023-10-06T05:07:02.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:02.891+0000 7fa1f1586700 1 -- 172.21.15.98:0/1727639289 wait complete. 2023-10-06T05:07:02.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:02.892+0000 7fa1f1586700 1 Processor -- start 2023-10-06T05:07:02.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:02.892+0000 7fa1f1586700 1 -- start start 2023-10-06T05:07:02.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:02.892+0000 7fa1f1586700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1ec0adb60 0x7fa1ec0c26d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:02.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:02.892+0000 7fa1f1586700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa1ec0ca370 0x7fa1ec168ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:02.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:02.892+0000 7fa1f1586700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa1ec0ce6d0 0x7fa1ec16ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:02.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:02.892+0000 7fa1f1586700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ec16b140 con 0x7fa1ec0adb60 2023-10-06T05:07:02.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:02.892+0000 7fa1f1586700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ec168d50 con 0x7fa1ec0ca370 2023-10-06T05:07:02.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:02.892+0000 7fa1f1586700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ec1681d0 con 0x7fa1ec0ce6d0 2023-10-06T05:07:02.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:02.893+0000 7fa1eaffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1ec0adb60 0x7fa1ec0c26d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:02.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:02.893+0000 7fa1eaffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1ec0adb60 0x7fa1ec0c26d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47750/0 (socket says 172.21.15.98:47750) 2023-10-06T05:07:02.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:02.893+0000 7fa1eaffd700 1 -- 172.21.15.98:0/3868037304 learned_addr learned my addr 172.21.15.98:0/3868037304 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:02.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:02.893+0000 7fa1e27fc700 1 --2- 172.21.15.98:0/3868037304 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa1ec0ca370 0x7fa1ec168ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:02.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:02.893+0000 7fa1eb7fe700 1 --2- 172.21.15.98:0/3868037304 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa1ec0ce6d0 0x7fa1ec16ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:02.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:02.893+0000 7fa1eaffd700 1 -- 172.21.15.98:0/3868037304 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa1ec0ce6d0 msgr2=0x7fa1ec16ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:02.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:02.893+0000 7fa1eaffd700 1 --2- 172.21.15.98:0/3868037304 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa1ec0ce6d0 0x7fa1ec16ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:02.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:02.893+0000 7fa1eaffd700 1 -- 172.21.15.98:0/3868037304 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa1ec0ca370 msgr2=0x7fa1ec168ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:02.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:02.893+0000 7fa1eaffd700 1 --2- 172.21.15.98:0/3868037304 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa1ec0ca370 0x7fa1ec168ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:02.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:02.893+0000 7fa1eaffd700 1 -- 172.21.15.98:0/3868037304 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1d40094f0 con 0x7fa1ec0adb60 2023-10-06T05:07:02.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:02.893+0000 7fa1eb7fe700 1 --2- 172.21.15.98:0/3868037304 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa1ec0ce6d0 0x7fa1ec16ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:07:02.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:02.893+0000 7fa1eaffd700 1 --2- 172.21.15.98:0/3868037304 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1ec0adb60 0x7fa1ec0c26d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc00c4e0 tx=0x7fa1dc00c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:02.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:02.893+0000 7fa1e37fe700 1 -- 172.21.15.98:0/3868037304 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1dc011590 con 0x7fa1ec0adb60 2023-10-06T05:07:02.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:02.893+0000 7fa1e37fe700 1 -- 172.21.15.98:0/3868037304 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1dc011710 con 0x7fa1ec0adb60 2023-10-06T05:07:02.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:02.893+0000 7fa1f1586700 1 -- 172.21.15.98:0/3868037304 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1ec1684b0 con 0x7fa1ec0adb60 2023-10-06T05:07:02.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:02.893+0000 7fa1e37fe700 1 -- 172.21.15.98:0/3868037304 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1dc011ad0 con 0x7fa1ec0adb60 2023-10-06T05:07:02.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:02.893+0000 7fa1f1586700 1 -- 172.21.15.98:0/3868037304 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1ec168a70 con 0x7fa1ec0adb60 2023-10-06T05:07:02.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:02.894+0000 7fa1f1586700 1 -- 172.21.15.98:0/3868037304 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1ec0ad920 con 0x7fa1ec0adb60 2023-10-06T05:07:02.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:02.895+0000 7fa1e37fe700 1 -- 172.21.15.98:0/3868037304 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fa1dc01b3b0 con 0x7fa1ec0adb60 2023-10-06T05:07:02.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:02.895+0000 7fa1e37fe700 1 --2- 172.21.15.98:0/3868037304 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fa1cc066f30 0x7fa1cc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:02.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:02.895+0000 7fa1e37fe700 1 -- 172.21.15.98:0/3868037304 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fa1dc083e10 con 0x7fa1ec0adb60 2023-10-06T05:07:02.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:02.895+0000 7fa1e27fc700 1 --2- 172.21.15.98:0/3868037304 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fa1cc066f30 0x7fa1cc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:02.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:02.895+0000 7fa1e27fc700 1 --2- 172.21.15.98:0/3868037304 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fa1cc066f30 0x7fa1cc0693e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec1554c0 tx=0x7fa1d4017000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:02.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:02.899+0000 7fa1e37fe700 1 -- 172.21.15.98:0/3868037304 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa1dc050c30 con 0x7fa1ec0adb60 2023-10-06T05:07:03.127 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.125+0000 7fa1f1586700 1 -- 172.21.15.98:0/3868037304 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa1ec0b4240 con 0x7fa1ec0adb60 2023-10-06T05:07:03.127 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.125+0000 7fa1e37fe700 1 -- 172.21.15.98:0/3868037304 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4596 (secure 0 0 0) 0x7fa1dc017070 con 0x7fa1ec0adb60 2023-10-06T05:07:03.128 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:07:03.128 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4285,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6834/925250632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":925250632},{"type":"v1","addr":"172.21.15.174:6834","nonce":925250632}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4288,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/1629449810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6833","nonce":1629449810},{"type":"v1","addr":"172.21.15.174:6835","nonce":1629449810}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4299,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/3961327990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3961327990},{"type":"v1","addr":"172.21.15.98:6836","nonce":3961327990}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-06T05:06:57.086683+0000","modified":"2023-10-06T05:07:02.755894+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.98:6837/2683239094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":2683239094},{"type":"v1","addr":"172.21.15.98:6837","nonce":2683239094}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2023-10-06T05:07:03.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.127+0000 7fa1f1586700 1 -- 172.21.15.98:0/3868037304 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fa1cc066f30 msgr2=0x7fa1cc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:03.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.127+0000 7fa1f1586700 1 --2- 172.21.15.98:0/3868037304 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fa1cc066f30 0x7fa1cc0693e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec1554c0 tx=0x7fa1d4017000 comp rx=0 tx=0).stop 2023-10-06T05:07:03.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.127+0000 7fa1f1586700 1 -- 172.21.15.98:0/3868037304 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1ec0adb60 msgr2=0x7fa1ec0c26d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:03.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.127+0000 7fa1f1586700 1 --2- 172.21.15.98:0/3868037304 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1ec0adb60 0x7fa1ec0c26d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc00c4e0 tx=0x7fa1dc00c7f0 comp rx=0 tx=0).stop 2023-10-06T05:07:03.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.127+0000 7fa1f1586700 1 -- 172.21.15.98:0/3868037304 shutdown_connections 2023-10-06T05:07:03.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.127+0000 7fa1f1586700 1 --2- 172.21.15.98:0/3868037304 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa1ec0ce6d0 0x7fa1ec16ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:03.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.127+0000 7fa1f1586700 1 --2- 172.21.15.98:0/3868037304 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fa1cc066f30 0x7fa1cc0693e0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:03.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.127+0000 7fa1f1586700 1 --2- 172.21.15.98:0/3868037304 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1ec0adb60 0x7fa1ec0c26d0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:03.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.127+0000 7fa1f1586700 1 --2- 172.21.15.98:0/3868037304 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa1ec0ca370 0x7fa1ec168ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:03.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.127+0000 7fa1f1586700 1 -- 172.21.15.98:0/3868037304 >> 172.21.15.98:0/3868037304 conn(0x7fa1ec0707d0 msgr2=0x7fa1ec1593b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:03.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.127+0000 7fa1f1586700 1 -- 172.21.15.98:0/3868037304 shutdown_connections 2023-10-06T05:07:03.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.128+0000 7fa1f1586700 1 -- 172.21.15.98:0/3868037304 wait complete. 2023-10-06T05:07:03.131 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 6 2023-10-06T05:07:03.142 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T05:07:03.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.254+0000 7f8fa2fa4700 1 Processor -- start 2023-10-06T05:07:03.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.254+0000 7f8fa2fa4700 1 -- start start 2023-10-06T05:07:03.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.255+0000 7f8fa2fa4700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f8f9c159560 0x7f8f9c0aa530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:03.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.255+0000 7f8fa2fa4700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f9c0b0c60 con 0x7f8f9c0b0760 2023-10-06T05:07:03.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.255+0000 7f8fa2fa4700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f9c0b0f40 con 0x7f8f9c0b49a0 2023-10-06T05:07:03.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.255+0000 7f8fa2fa4700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f9c0b1220 con 0x7f8f9c159560 2023-10-06T05:07:03.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.255+0000 7f8fa1541700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f8f9c0b0760 0x7f8f9c0b1bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:57038/0 (socket says 172.21.15.98:57038) 2023-10-06T05:07:03.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.255+0000 7f8fa1541700 1 -- 172.21.15.98:0/1059343992 learned_addr learned my addr 172.21.15.98:0/1059343992 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:07:03.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.256+0000 7f8fa0d40700 1 --2- 172.21.15.98:0/1059343992 >> v2:172.21.15.174:3300/0 conn(0x7f8f9c159560 0x7f8f9c0aa530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:03.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.256+0000 7f8f937fe700 1 -- 172.21.15.98:0/1059343992 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 247837343 0 0) 0x7f8f9c0b0c60 con 0x7f8f9c0b0760 2023-10-06T05:07:03.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.256+0000 7f8f937fe700 1 -- 172.21.15.98:0/1059343992 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f84003570 con 0x7f8f9c0b0760 2023-10-06T05:07:03.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.256+0000 7f8fa0d40700 1 -- 172.21.15.98:0/1059343992 >> v1:172.21.15.174:6789/0 conn(0x7f8f9c0b49a0 legacy=0x7f8f9c0aaa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:03.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.256+0000 7f8f937fe700 1 -- 172.21.15.98:0/1059343992 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1996087818 0 0) 0x7f8f84003570 con 0x7f8f9c0b0760 2023-10-06T05:07:03.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.256+0000 7f8fa0d40700 1 -- 172.21.15.98:0/1059343992 >> v1:172.21.15.98:6789/0 conn(0x7f8f9c0b0760 legacy=0x7f8f9c0b1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:03.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.256+0000 7f8fa0d40700 1 -- 172.21.15.98:0/1059343992 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f9c0b1500 con 0x7f8f9c159560 2023-10-06T05:07:03.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.256+0000 7f8fa0d40700 1 --2- 172.21.15.98:0/1059343992 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8f9c159560 0x7f8f9c0aa530 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8f98009ee0 tx=0x7f8f98009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dcd683261b220fa5 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:03.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.256+0000 7f8f937fe700 1 -- 172.21.15.98:0/1059343992 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f9800e070 con 0x7f8f9c159560 2023-10-06T05:07:03.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.256+0000 7f8f937fe700 1 -- 172.21.15.98:0/1059343992 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f9800b920 con 0x7f8f9c159560 2023-10-06T05:07:03.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.256+0000 7f8f937fe700 1 -- 172.21.15.98:0/1059343992 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f94001470 con 0x7f8f9c159560 2023-10-06T05:07:03.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.257+0000 7f8fa2fa4700 1 -- 172.21.15.98:0/1059343992 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8f9c159560 msgr2=0x7f8f9c0aa530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:03.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.257+0000 7f8fa2fa4700 1 --2- 172.21.15.98:0/1059343992 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8f9c159560 0x7f8f9c0aa530 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8f98009ee0 tx=0x7f8f98009780 comp rx=0 tx=0).stop 2023-10-06T05:07:03.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.257+0000 7f8fa2fa4700 1 -- 172.21.15.98:0/1059343992 shutdown_connections 2023-10-06T05:07:03.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.257+0000 7f8fa2fa4700 1 --2- 172.21.15.98:0/1059343992 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8f9c159560 0x7f8f9c0aa530 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:03.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.257+0000 7f8fa2fa4700 1 -- 172.21.15.98:0/1059343992 >> 172.21.15.98:0/1059343992 conn(0x7f8f9c0997f0 msgr2=0x7f8f9c06f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:03.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.257+0000 7f8fa2fa4700 1 -- 172.21.15.98:0/1059343992 shutdown_connections 2023-10-06T05:07:03.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.257+0000 7f8fa2fa4700 1 -- 172.21.15.98:0/1059343992 wait complete. 2023-10-06T05:07:03.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.258+0000 7f8fa2fa4700 1 Processor -- start 2023-10-06T05:07:03.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.258+0000 7f8fa2fa4700 1 -- start start 2023-10-06T05:07:03.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.259+0000 7f8fa2fa4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f9c0b0760 0x7f8f9c158130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:03.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.259+0000 7f8fa2fa4700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8f9c0b49a0 0x7f8f9c0ad3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:03.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.259+0000 7f8fa2fa4700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8f9c159560 0x7f8f9c09d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:03.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.259+0000 7f8fa2fa4700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f9c0abc60 con 0x7f8f9c0b0760 2023-10-06T05:07:03.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.259+0000 7f8fa2fa4700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f9c158670 con 0x7f8f9c159560 2023-10-06T05:07:03.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.259+0000 7f8fa2fa4700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f9c09de90 con 0x7f8f9c0b49a0 2023-10-06T05:07:03.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.259+0000 7f8fa0d40700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f9c0b0760 0x7f8f9c158130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:03.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.259+0000 7f8fa0d40700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f9c0b0760 0x7f8f9c158130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47762/0 (socket says 172.21.15.98:47762) 2023-10-06T05:07:03.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.259+0000 7f8fa0d40700 1 -- 172.21.15.98:0/3462830462 learned_addr learned my addr 172.21.15.98:0/3462830462 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:03.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.259+0000 7f8fa1541700 1 --2- 172.21.15.98:0/3462830462 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8f9c159560 0x7f8f9c09d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:03.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.259+0000 7f8f93fff700 1 --2- 172.21.15.98:0/3462830462 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8f9c0b49a0 0x7f8f9c0ad3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:03.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.259+0000 7f8fa0d40700 1 -- 172.21.15.98:0/3462830462 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8f9c0b49a0 msgr2=0x7f8f9c0ad3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:03.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.259+0000 7f8fa0d40700 1 --2- 172.21.15.98:0/3462830462 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8f9c0b49a0 0x7f8f9c0ad3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:03.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.259+0000 7f8fa0d40700 1 -- 172.21.15.98:0/3462830462 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8f9c159560 msgr2=0x7f8f9c09d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:03.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.259+0000 7f8fa0d40700 1 --2- 172.21.15.98:0/3462830462 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8f9c159560 0x7f8f9c09d490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:03.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.259+0000 7f8fa0d40700 1 -- 172.21.15.98:0/3462830462 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f98009a20 con 0x7f8f9c0b0760 2023-10-06T05:07:03.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.259+0000 7f8fa0d40700 1 --2- 172.21.15.98:0/3462830462 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f9c0b0760 0x7f8f9c158130 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8f9c067c20 tx=0x7f8f98017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:03.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.259+0000 7f8f90ff9700 1 -- 172.21.15.98:0/3462830462 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f9801f410 con 0x7f8f9c0b0760 2023-10-06T05:07:03.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.259+0000 7f8f90ff9700 1 -- 172.21.15.98:0/3462830462 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f9801f570 con 0x7f8f9c0b0760 2023-10-06T05:07:03.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.259+0000 7f8fa2fa4700 1 -- 172.21.15.98:0/3462830462 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f9c0ab330 con 0x7f8f9c0b0760 2023-10-06T05:07:03.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.259+0000 7f8f90ff9700 1 -- 172.21.15.98:0/3462830462 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f9801f910 con 0x7f8f9c0b0760 2023-10-06T05:07:03.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.260+0000 7f8fa2fa4700 1 -- 172.21.15.98:0/3462830462 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f9c0ab8f0 con 0x7f8f9c0b0760 2023-10-06T05:07:03.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.260+0000 7f8fa2fa4700 1 -- 172.21.15.98:0/3462830462 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f9c064a90 con 0x7f8f9c0b0760 2023-10-06T05:07:03.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.261+0000 7f8f90ff9700 1 -- 172.21.15.98:0/3462830462 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f8f9801fc80 con 0x7f8f9c0b0760 2023-10-06T05:07:03.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.261+0000 7f8f90ff9700 1 --2- 172.21.15.98:0/3462830462 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f8f88066f30 0x7f8f880693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:03.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.261+0000 7f8f90ff9700 1 -- 172.21.15.98:0/3462830462 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f8f98086b80 con 0x7f8f9c0b0760 2023-10-06T05:07:03.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.261+0000 7f8f93fff700 1 --2- 172.21.15.98:0/3462830462 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f8f88066f30 0x7f8f880693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:03.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.261+0000 7f8f93fff700 1 --2- 172.21.15.98:0/3462830462 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f8f88066f30 0x7f8f880693e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f8f9c15cf20 tx=0x7f8f8c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:03.267 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.265+0000 7f8f90ff9700 1 -- 172.21.15.98:0/3462830462 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8f980539a0 con 0x7f8f9c0b0760 2023-10-06T05:07:03.453 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.451+0000 7f8fa2fa4700 1 -- 172.21.15.98:0/3462830462 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8f9c0b17e0 con 0x7f8f9c0b0760 2023-10-06T05:07:03.454 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.452+0000 7f8f90ff9700 1 -- 172.21.15.98:0/3462830462 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+18707 (secure 0 0 0) 0x7f8f98023090 con 0x7f8f9c0b0760 2023-10-06T05:07:03.455 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:07:03.455 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":29,"fsid":"f50c5df5-cad5-47a4-9bb1-8c791d4311ec","created":"2023-10-06T05:06:32.971175+0000","modified":"2023-10-06T05:07:02.131672+0000","last_up_change":"2023-10-06T05:06:44.016489+0000","last_in_change":"2023-10-06T05:06:35.544511+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-06T05:06:44.559671+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-06T05:06:51.432161+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-10-06T05:06:55.589057+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-10-06T05:06:56.041687+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-10-06T05:06:59.754099+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d9495d95-da75-4c3b-8927-e9bae6a8b093","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":90627},{"type":"v1","addr":"172.21.15.98:6801","nonce":90627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":90627},{"type":"v1","addr":"172.21.15.98:6803","nonce":90627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":90627},{"type":"v1","addr":"172.21.15.98:6807","nonce":90627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":90627},{"type":"v1","addr":"172.21.15.98:6805","nonce":90627}]},"public_addr":"172.21.15.98:6801/90627","cluster_addr":"172.21.15.98:6803/90627","heartbeat_back_addr":"172.21.15.98:6807/90627","heartbeat_front_addr":"172.21.15.98:6805/90627","state":["exists","up"]},{"osd":1,"uuid":"f25ba5be-d65e-4707-a134-154092deb754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":90628},{"type":"v1","addr":"172.21.15.98:6809","nonce":90628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":90628},{"type":"v1","addr":"172.21.15.98:6811","nonce":90628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":90628},{"type":"v1","addr":"172.21.15.98:6815","nonce":90628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":90628},{"type":"v1","addr":"172.21.15.98:6813","nonce":90628}]},"public_addr":"172.21.15.98:6809/90628","cluster_addr":"172.21.15.98:6811/90628","heartbeat_back_addr":"172.21.15.98:6815/90628","heartbeat_front_addr":"172.21.15.98:6813/90628","state":["exists","up"]},{"osd":2,"uuid":"5ab9e028-945c-43e2-8c95-3b0b4cf38e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":90662},{"type":"v1","addr":"172.21.15.98:6818","nonce":90662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":90662},{"type":"v1","addr":"172.21.15.98:6822","nonce":90662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":90662},{"type":"v1","addr":"172.21.15.98:6830","nonce":90662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":90662},{"type":"v1","addr":"172.21.15.98:6826","nonce":90662}]},"public_addr":"172.21.15.98:6818/90662","cluster_addr":"172.21.15.98:6822/90662","heartbeat_back_addr":"172.21.15.98:6830/90662","heartbeat_front_addr":"172.21.15.98:6826/90662","state":["exists","up"]},{"osd":3,"uuid":"3bb803ff-1738-46bf-be62-2267bf378822","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6817","nonce":90663},{"type":"v1","addr":"172.21.15.98:6819","nonce":90663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6821","nonce":90663},{"type":"v1","addr":"172.21.15.98:6823","nonce":90663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6829","nonce":90663},{"type":"v1","addr":"172.21.15.98:6831","nonce":90663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6825","nonce":90663},{"type":"v1","addr":"172.21.15.98:6827","nonce":90663}]},"public_addr":"172.21.15.98:6819/90663","cluster_addr":"172.21.15.98:6823/90663","heartbeat_back_addr":"172.21.15.98:6831/90663","heartbeat_front_addr":"172.21.15.98:6827/90663","state":["exists","up"]},{"osd":4,"uuid":"7d57a4c9-6462-43f6-b64a-7f9e9b4353d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":90847},{"type":"v1","addr":"172.21.15.174:6801","nonce":90847}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":90847},{"type":"v1","addr":"172.21.15.174:6803","nonce":90847}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":90847},{"type":"v1","addr":"172.21.15.174:6807","nonce":90847}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":90847},{"type":"v1","addr":"172.21.15.174:6805","nonce":90847}]},"public_addr":"172.21.15.174:6801/90847","cluster_addr":"172.21.15.174:6803/90847","heartbeat_back_addr":"172.21.15.174:6807/90847","heartbeat_front_addr":"172.21.15.174:6805/90847","state":["exists","up"]},{"osd":5,"uuid":"83287ea4-e44d-4af9-bc77-36d47c60d4a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":90848},{"type":"v1","addr":"172.21.15.174:6809","nonce":90848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":90848},{"type":"v1","addr":"172.21.15.174:6811","nonce":90848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":90848},{"type":"v1","addr":"172.21.15.174:6815","nonce":90848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":90848},{"type":"v1","addr":"172.21.15.174:6813","nonce":90848}]},"public_addr":"172.21.15.174:6809/90848","cluster_addr":"172.21.15.174:6811/90848","heartbeat_back_addr":"172.21.15.174:6815/90848","heartbeat_front_addr":"172.21.15.174:6813/90848","state":["exists","up"]},{"osd":6,"uuid":"496a6272-892e-43c4-a145-206623c22f99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":90877},{"type":"v1","addr":"172.21.15.174:6817","nonce":90877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":90877},{"type":"v1","addr":"172.21.15.174:6819","nonce":90877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":90877},{"type":"v1","addr":"172.21.15.174:6823","nonce":90877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":90877},{"type":"v1","addr":"172.21.15.174:6821","nonce":90877}]},"public_addr":"172.21.15.174:6817/90877","cluster_addr":"172.21.15.174:6819/90877","heartbeat_back_addr":"172.21.15.174:6823/90877","heartbeat_front_addr":"172.21.15.174:6821/90877","state":["exists","up"]},{"osd":7,"uuid":"49967e34-8b07-4bab-a13e-0c16d6a0f3a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":90878},{"type":"v1","addr":"172.21.15.174:6825","nonce":90878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":90878},{"type":"v1","addr":"172.21.15.174:6827","nonce":90878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":90878},{"type":"v1","addr":"172.21.15.174:6831","nonce":90878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":90878},{"type":"v1","addr":"172.21.15.174:6829","nonce":90878}]},"public_addr":"172.21.15.174:6825/90878","cluster_addr":"172.21.15.174:6827/90878","heartbeat_back_addr":"172.21.15.174:6831/90878","heartbeat_front_addr":"172.21.15.174:6829/90878","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.446479+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.514266+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.794108+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.810894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.731919+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.717756+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.737435+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.508682+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T05:07:03.457 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.454+0000 7f8fa2fa4700 1 -- 172.21.15.98:0/3462830462 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f8f88066f30 msgr2=0x7f8f880693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:03.457 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.454+0000 7f8fa2fa4700 1 --2- 172.21.15.98:0/3462830462 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f8f88066f30 0x7f8f880693e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f8f9c15cf20 tx=0x7f8f8c009040 comp rx=0 tx=0).stop 2023-10-06T05:07:03.457 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.454+0000 7f8fa2fa4700 1 -- 172.21.15.98:0/3462830462 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f9c0b0760 msgr2=0x7f8f9c158130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:03.457 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.454+0000 7f8fa2fa4700 1 --2- 172.21.15.98:0/3462830462 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f9c0b0760 0x7f8f9c158130 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8f9c067c20 tx=0x7f8f98017000 comp rx=0 tx=0).stop 2023-10-06T05:07:03.457 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.454+0000 7f8fa2fa4700 1 -- 172.21.15.98:0/3462830462 shutdown_connections 2023-10-06T05:07:03.457 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.454+0000 7f8fa2fa4700 1 --2- 172.21.15.98:0/3462830462 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8f9c159560 0x7f8f9c09d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:03.457 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.454+0000 7f8fa2fa4700 1 --2- 172.21.15.98:0/3462830462 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f8f88066f30 0x7f8f880693e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:03.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.454+0000 7f8fa2fa4700 1 --2- 172.21.15.98:0/3462830462 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f9c0b0760 0x7f8f9c158130 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:03.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.454+0000 7f8fa2fa4700 1 --2- 172.21.15.98:0/3462830462 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8f9c0b49a0 0x7f8f9c0ad3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:03.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.454+0000 7f8fa2fa4700 1 -- 172.21.15.98:0/3462830462 >> 172.21.15.98:0/3462830462 conn(0x7f8f9c0997f0 msgr2=0x7f8f9c158990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:03.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.455+0000 7f8fa2fa4700 1 -- 172.21.15.98:0/3462830462 shutdown_connections 2023-10-06T05:07:03.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.455+0000 7f8fa2fa4700 1 -- 172.21.15.98:0/3462830462 wait complete. 2023-10-06T05:07:03.469 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T05:07:03.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.581+0000 7f9d0168d700 1 Processor -- start 2023-10-06T05:07:03.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.581+0000 7f9d0168d700 1 -- start start 2023-10-06T05:07:03.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.582+0000 7f9d0168d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9cfc159560 0x7f9cfc0aa530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:03.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.582+0000 7f9d0168d700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f9cfc0b0760 0x7f9cfc0b1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:03.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.582+0000 7f9d0168d700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cfc0b0c60 con 0x7f9cfc0b49a0 2023-10-06T05:07:03.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.582+0000 7f9d0168d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cfc0b0f40 con 0x7f9cfc159560 2023-10-06T05:07:03.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.582+0000 7f9d0168d700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cfc0b1220 con 0x7f9cfc0b0760 2023-10-06T05:07:03.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.582+0000 7f9cfaffd700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9cfc159560 0x7f9cfc0aa530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:03.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.582+0000 7f9cfaffd700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9cfc159560 0x7f9cfc0aa530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47768/0 (socket says 172.21.15.98:47768) 2023-10-06T05:07:03.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.582+0000 7f9cfaffd700 1 -- 172.21.15.98:0/563997584 learned_addr learned my addr 172.21.15.98:0/563997584 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:03.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.582+0000 7f9cfb7fe700 1 --2- 172.21.15.98:0/563997584 >> v2:172.21.15.174:3300/0 conn(0x7f9cfc0b0760 0x7f9cfc0b1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:03.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.582+0000 7f9cfaffd700 1 -- 172.21.15.98:0/563997584 >> v2:172.21.15.174:3300/0 conn(0x7f9cfc0b0760 msgr2=0x7f9cfc0b1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:03.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.582+0000 7f9cfaffd700 1 --2- 172.21.15.98:0/563997584 >> v2:172.21.15.174:3300/0 conn(0x7f9cfc0b0760 0x7f9cfc0b1bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:03.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.582+0000 7f9cfaffd700 1 -- 172.21.15.98:0/563997584 >> v1:172.21.15.174:6789/0 conn(0x7f9cfc0b49a0 legacy=0x7f9cfc0aaa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:03.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.582+0000 7f9cfaffd700 1 -- 172.21.15.98:0/563997584 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cfc0b1500 con 0x7f9cfc159560 2023-10-06T05:07:03.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.582+0000 7f9cfaffd700 1 --2- 172.21.15.98:0/563997584 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9cfc159560 0x7f9cfc0aa530 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9cf4009990 tx=0x7f9cf4009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68ffc6f1bcbeca41 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:03.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.582+0000 7f9cf9ffb700 1 -- 172.21.15.98:0/563997584 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf4005c10 con 0x7f9cfc159560 2023-10-06T05:07:03.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.582+0000 7f9cf9ffb700 1 -- 172.21.15.98:0/563997584 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cf4005d70 con 0x7f9cfc159560 2023-10-06T05:07:03.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.582+0000 7f9cf9ffb700 1 -- 172.21.15.98:0/563997584 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf400b840 con 0x7f9cfc159560 2023-10-06T05:07:03.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.583+0000 7f9d0168d700 1 -- 172.21.15.98:0/563997584 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9cfc159560 msgr2=0x7f9cfc0aa530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:03.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.583+0000 7f9d0168d700 1 --2- 172.21.15.98:0/563997584 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9cfc159560 0x7f9cfc0aa530 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9cf4009990 tx=0x7f9cf4009220 comp rx=0 tx=0).stop 2023-10-06T05:07:03.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.583+0000 7f9d0168d700 1 -- 172.21.15.98:0/563997584 shutdown_connections 2023-10-06T05:07:03.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.583+0000 7f9d0168d700 1 --2- 172.21.15.98:0/563997584 >> v2:172.21.15.174:3300/0 conn(0x7f9cfc0b0760 0x7f9cfc0b1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:03.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.583+0000 7f9d0168d700 1 --2- 172.21.15.98:0/563997584 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9cfc159560 0x7f9cfc0aa530 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:03.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.583+0000 7f9d0168d700 1 -- 172.21.15.98:0/563997584 >> 172.21.15.98:0/563997584 conn(0x7f9cfc0997f0 msgr2=0x7f9cfc06f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:03.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.583+0000 7f9d0168d700 1 -- 172.21.15.98:0/563997584 shutdown_connections 2023-10-06T05:07:03.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.583+0000 7f9d0168d700 1 -- 172.21.15.98:0/563997584 wait complete. 2023-10-06T05:07:03.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.584+0000 7f9d0168d700 1 Processor -- start 2023-10-06T05:07:03.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.584+0000 7f9d0168d700 1 -- start start 2023-10-06T05:07:03.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.585+0000 7f9d0168d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9cfc0b0760 0x7f9cfc16a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:03.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.585+0000 7f9d0168d700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9cfc0b49a0 0x7f9cfc0aefb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:03.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.585+0000 7f9d0168d700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9cfc159560 0x7f9cfc09d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:03.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.585+0000 7f9d0168d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cfc0af4f0 con 0x7f9cfc0b0760 2023-10-06T05:07:03.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.585+0000 7f9d0168d700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cfc09de80 con 0x7f9cfc0b49a0 2023-10-06T05:07:03.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.585+0000 7f9d0168d700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cfc16b8a0 con 0x7f9cfc159560 2023-10-06T05:07:03.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.585+0000 7f9cfaffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9cfc0b0760 0x7f9cfc16a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:03.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.585+0000 7f9cfaffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9cfc0b0760 0x7f9cfc16a880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47782/0 (socket says 172.21.15.98:47782) 2023-10-06T05:07:03.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.585+0000 7f9cfaffd700 1 -- 172.21.15.98:0/1855171982 learned_addr learned my addr 172.21.15.98:0/1855171982 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:03.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.585+0000 7f9cfaffd700 1 -- 172.21.15.98:0/1855171982 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9cfc159560 msgr2=0x7f9cfc09d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:03.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.585+0000 7f9cfaffd700 1 --2- 172.21.15.98:0/1855171982 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9cfc159560 0x7f9cfc09d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:03.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.585+0000 7f9cfaffd700 1 -- 172.21.15.98:0/1855171982 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9cfc0b49a0 msgr2=0x7f9cfc0aefb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:03.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.585+0000 7f9cfaffd700 1 --2- 172.21.15.98:0/1855171982 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9cfc0b49a0 0x7f9cfc0aefb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:03.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.585+0000 7f9cfaffd700 1 -- 172.21.15.98:0/1855171982 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cf4009480 con 0x7f9cfc0b0760 2023-10-06T05:07:03.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.585+0000 7f9cfaffd700 1 --2- 172.21.15.98:0/1855171982 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9cfc0b0760 0x7f9cfc16a880 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc067c20 tx=0x7f9cf4016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:03.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.585+0000 7f9cf2ffd700 1 -- 172.21.15.98:0/1855171982 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf4011a20 con 0x7f9cfc0b0760 2023-10-06T05:07:03.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.586+0000 7f9cf2ffd700 1 -- 172.21.15.98:0/1855171982 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cf4011bc0 con 0x7f9cfc0b0760 2023-10-06T05:07:03.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.586+0000 7f9cf2ffd700 1 -- 172.21.15.98:0/1855171982 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf401b430 con 0x7f9cfc0b0760 2023-10-06T05:07:03.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.586+0000 7f9d0168d700 1 -- 172.21.15.98:0/1855171982 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cfc09dbb0 con 0x7f9cfc0b0760 2023-10-06T05:07:03.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.586+0000 7f9d0168d700 1 -- 172.21.15.98:0/1855171982 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cfc16b250 con 0x7f9cfc0b0760 2023-10-06T05:07:03.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.587+0000 7f9d0168d700 1 -- 172.21.15.98:0/1855171982 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cfc0cea20 con 0x7f9cfc0b0760 2023-10-06T05:07:03.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.587+0000 7f9cf2ffd700 1 -- 172.21.15.98:0/1855171982 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f9cf401b7e0 con 0x7f9cfc0b0760 2023-10-06T05:07:03.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.587+0000 7f9cf2ffd700 1 --2- 172.21.15.98:0/1855171982 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f9cdc066f30 0x7f9cdc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:03.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.587+0000 7f9cf2ffd700 1 -- 172.21.15.98:0/1855171982 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f9cf4086250 con 0x7f9cfc0b0760 2023-10-06T05:07:03.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.587+0000 7f9cfa7fc700 1 --2- 172.21.15.98:0/1855171982 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f9cdc066f30 0x7f9cdc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:03.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.588+0000 7f9cfa7fc700 1 --2- 172.21.15.98:0/1855171982 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f9cdc066f30 0x7f9cdc0693e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc0afad0 tx=0x7f9ce4007b40 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:03.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.591+0000 7f9cf2ffd700 1 -- 172.21.15.98:0/1855171982 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9cf4053070 con 0x7f9cfc0b0760 2023-10-06T05:07:03.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.781+0000 7f9d0168d700 1 -- 172.21.15.98:0/1855171982 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9cfc064a90 con 0x7f9cfc0b0760 2023-10-06T05:07:03.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.782+0000 7f9cf2ffd700 1 -- 172.21.15.98:0/1855171982 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+18707 (secure 0 0 0) 0x7f9cf4056690 con 0x7f9cfc0b0760 2023-10-06T05:07:03.784 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:07:03.785 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":29,"fsid":"f50c5df5-cad5-47a4-9bb1-8c791d4311ec","created":"2023-10-06T05:06:32.971175+0000","modified":"2023-10-06T05:07:02.131672+0000","last_up_change":"2023-10-06T05:06:44.016489+0000","last_in_change":"2023-10-06T05:06:35.544511+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-06T05:06:44.559671+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-06T05:06:51.432161+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-10-06T05:06:55.589057+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-10-06T05:06:56.041687+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-10-06T05:06:59.754099+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d9495d95-da75-4c3b-8927-e9bae6a8b093","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":90627},{"type":"v1","addr":"172.21.15.98:6801","nonce":90627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":90627},{"type":"v1","addr":"172.21.15.98:6803","nonce":90627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":90627},{"type":"v1","addr":"172.21.15.98:6807","nonce":90627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":90627},{"type":"v1","addr":"172.21.15.98:6805","nonce":90627}]},"public_addr":"172.21.15.98:6801/90627","cluster_addr":"172.21.15.98:6803/90627","heartbeat_back_addr":"172.21.15.98:6807/90627","heartbeat_front_addr":"172.21.15.98:6805/90627","state":["exists","up"]},{"osd":1,"uuid":"f25ba5be-d65e-4707-a134-154092deb754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":90628},{"type":"v1","addr":"172.21.15.98:6809","nonce":90628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":90628},{"type":"v1","addr":"172.21.15.98:6811","nonce":90628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":90628},{"type":"v1","addr":"172.21.15.98:6815","nonce":90628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":90628},{"type":"v1","addr":"172.21.15.98:6813","nonce":90628}]},"public_addr":"172.21.15.98:6809/90628","cluster_addr":"172.21.15.98:6811/90628","heartbeat_back_addr":"172.21.15.98:6815/90628","heartbeat_front_addr":"172.21.15.98:6813/90628","state":["exists","up"]},{"osd":2,"uuid":"5ab9e028-945c-43e2-8c95-3b0b4cf38e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":90662},{"type":"v1","addr":"172.21.15.98:6818","nonce":90662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":90662},{"type":"v1","addr":"172.21.15.98:6822","nonce":90662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":90662},{"type":"v1","addr":"172.21.15.98:6830","nonce":90662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":90662},{"type":"v1","addr":"172.21.15.98:6826","nonce":90662}]},"public_addr":"172.21.15.98:6818/90662","cluster_addr":"172.21.15.98:6822/90662","heartbeat_back_addr":"172.21.15.98:6830/90662","heartbeat_front_addr":"172.21.15.98:6826/90662","state":["exists","up"]},{"osd":3,"uuid":"3bb803ff-1738-46bf-be62-2267bf378822","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6817","nonce":90663},{"type":"v1","addr":"172.21.15.98:6819","nonce":90663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6821","nonce":90663},{"type":"v1","addr":"172.21.15.98:6823","nonce":90663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6829","nonce":90663},{"type":"v1","addr":"172.21.15.98:6831","nonce":90663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6825","nonce":90663},{"type":"v1","addr":"172.21.15.98:6827","nonce":90663}]},"public_addr":"172.21.15.98:6819/90663","cluster_addr":"172.21.15.98:6823/90663","heartbeat_back_addr":"172.21.15.98:6831/90663","heartbeat_front_addr":"172.21.15.98:6827/90663","state":["exists","up"]},{"osd":4,"uuid":"7d57a4c9-6462-43f6-b64a-7f9e9b4353d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":90847},{"type":"v1","addr":"172.21.15.174:6801","nonce":90847}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":90847},{"type":"v1","addr":"172.21.15.174:6803","nonce":90847}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":90847},{"type":"v1","addr":"172.21.15.174:6807","nonce":90847}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":90847},{"type":"v1","addr":"172.21.15.174:6805","nonce":90847}]},"public_addr":"172.21.15.174:6801/90847","cluster_addr":"172.21.15.174:6803/90847","heartbeat_back_addr":"172.21.15.174:6807/90847","heartbeat_front_addr":"172.21.15.174:6805/90847","state":["exists","up"]},{"osd":5,"uuid":"83287ea4-e44d-4af9-bc77-36d47c60d4a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":90848},{"type":"v1","addr":"172.21.15.174:6809","nonce":90848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":90848},{"type":"v1","addr":"172.21.15.174:6811","nonce":90848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":90848},{"type":"v1","addr":"172.21.15.174:6815","nonce":90848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":90848},{"type":"v1","addr":"172.21.15.174:6813","nonce":90848}]},"public_addr":"172.21.15.174:6809/90848","cluster_addr":"172.21.15.174:6811/90848","heartbeat_back_addr":"172.21.15.174:6815/90848","heartbeat_front_addr":"172.21.15.174:6813/90848","state":["exists","up"]},{"osd":6,"uuid":"496a6272-892e-43c4-a145-206623c22f99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":90877},{"type":"v1","addr":"172.21.15.174:6817","nonce":90877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":90877},{"type":"v1","addr":"172.21.15.174:6819","nonce":90877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":90877},{"type":"v1","addr":"172.21.15.174:6823","nonce":90877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":90877},{"type":"v1","addr":"172.21.15.174:6821","nonce":90877}]},"public_addr":"172.21.15.174:6817/90877","cluster_addr":"172.21.15.174:6819/90877","heartbeat_back_addr":"172.21.15.174:6823/90877","heartbeat_front_addr":"172.21.15.174:6821/90877","state":["exists","up"]},{"osd":7,"uuid":"49967e34-8b07-4bab-a13e-0c16d6a0f3a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":90878},{"type":"v1","addr":"172.21.15.174:6825","nonce":90878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":90878},{"type":"v1","addr":"172.21.15.174:6827","nonce":90878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":90878},{"type":"v1","addr":"172.21.15.174:6831","nonce":90878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":90878},{"type":"v1","addr":"172.21.15.174:6829","nonce":90878}]},"public_addr":"172.21.15.174:6825/90878","cluster_addr":"172.21.15.174:6827/90878","heartbeat_back_addr":"172.21.15.174:6831/90878","heartbeat_front_addr":"172.21.15.174:6829/90878","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.446479+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.514266+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.794108+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.810894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.731919+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.717756+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.737435+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.508682+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T05:07:03.787 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.784+0000 7f9d0168d700 1 -- 172.21.15.98:0/1855171982 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f9cdc066f30 msgr2=0x7f9cdc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:03.787 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.784+0000 7f9d0168d700 1 --2- 172.21.15.98:0/1855171982 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f9cdc066f30 0x7f9cdc0693e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc0afad0 tx=0x7f9ce4007b40 comp rx=0 tx=0).stop 2023-10-06T05:07:03.787 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.784+0000 7f9d0168d700 1 -- 172.21.15.98:0/1855171982 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9cfc0b0760 msgr2=0x7f9cfc16a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:03.787 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.784+0000 7f9d0168d700 1 --2- 172.21.15.98:0/1855171982 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9cfc0b0760 0x7f9cfc16a880 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc067c20 tx=0x7f9cf4016000 comp rx=0 tx=0).stop 2023-10-06T05:07:03.787 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.784+0000 7f9d0168d700 1 -- 172.21.15.98:0/1855171982 shutdown_connections 2023-10-06T05:07:03.787 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.784+0000 7f9d0168d700 1 --2- 172.21.15.98:0/1855171982 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9cfc159560 0x7f9cfc09d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:03.787 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.784+0000 7f9d0168d700 1 --2- 172.21.15.98:0/1855171982 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f9cdc066f30 0x7f9cdc0693e0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:03.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.784+0000 7f9d0168d700 1 --2- 172.21.15.98:0/1855171982 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9cfc0b0760 0x7f9cfc16a880 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:03.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.784+0000 7f9d0168d700 1 --2- 172.21.15.98:0/1855171982 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9cfc0b49a0 0x7f9cfc0aefb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:03.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.784+0000 7f9d0168d700 1 -- 172.21.15.98:0/1855171982 >> 172.21.15.98:0/1855171982 conn(0x7f9cfc0997f0 msgr2=0x7f9cfc0b3230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:03.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.784+0000 7f9d0168d700 1 -- 172.21.15.98:0/1855171982 shutdown_connections 2023-10-06T05:07:03.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.784+0000 7f9d0168d700 1 -- 172.21.15.98:0/1855171982 wait complete. 2023-10-06T05:07:03.810 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-06T05:07:03.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.921+0000 7f85d494e700 1 Processor -- start 2023-10-06T05:07:03.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.922+0000 7f85d494e700 1 -- start start 2023-10-06T05:07:03.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.922+0000 7f85d494e700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f85d0156280 0x7f85d015a060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:03.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.922+0000 7f85d494e700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f85d0068cd0 0x7f85d00abb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:03.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.922+0000 7f85d494e700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85d00c9700 con 0x7f85d0156650 2023-10-06T05:07:03.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.922+0000 7f85d494e700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85d015b000 con 0x7f85d0068cd0 2023-10-06T05:07:03.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.922+0000 7f85d494e700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85d00ad810 con 0x7f85d0156280 2023-10-06T05:07:03.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.922+0000 7f85ced9d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f85d0068cd0 0x7f85d00abb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:03.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.923+0000 7f85ced9d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f85d0068cd0 0x7f85d00abb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47786/0 (socket says 172.21.15.98:47786) 2023-10-06T05:07:03.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.923+0000 7f85ced9d700 1 -- 172.21.15.98:0/190488317 learned_addr learned my addr 172.21.15.98:0/190488317 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:03.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.923+0000 7f85ce59c700 1 --2- 172.21.15.98:0/190488317 >> v2:172.21.15.174:3300/0 conn(0x7f85d0156280 0x7f85d015a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:03.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.923+0000 7f85ced9d700 1 -- 172.21.15.98:0/190488317 >> v2:172.21.15.174:3300/0 conn(0x7f85d0156280 msgr2=0x7f85d015a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:03.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.923+0000 7f85ced9d700 1 --2- 172.21.15.98:0/190488317 >> v2:172.21.15.174:3300/0 conn(0x7f85d0156280 0x7f85d015a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:03.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.923+0000 7f85ced9d700 1 -- 172.21.15.98:0/190488317 >> v1:172.21.15.174:6789/0 conn(0x7f85d0156650 legacy=0x7f85d00685c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:03.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.923+0000 7f85ced9d700 1 -- 172.21.15.98:0/190488317 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85d015a760 con 0x7f85d0068cd0 2023-10-06T05:07:03.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.923+0000 7f85ced9d700 1 --2- 172.21.15.98:0/190488317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85d0068cd0 0x7f85d00abb10 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f85d0070b70 tx=0x7f85c800ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccf1f57c8e930e33 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:03.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.923+0000 7f85cdd9b700 1 -- 172.21.15.98:0/190488317 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c8010070 con 0x7f85d0068cd0 2023-10-06T05:07:03.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.923+0000 7f85cdd9b700 1 -- 172.21.15.98:0/190488317 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85c800de50 con 0x7f85d0068cd0 2023-10-06T05:07:03.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.923+0000 7f85cdd9b700 1 -- 172.21.15.98:0/190488317 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c800b320 con 0x7f85d0068cd0 2023-10-06T05:07:03.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.924+0000 7f85d494e700 1 -- 172.21.15.98:0/190488317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85d0068cd0 msgr2=0x7f85d00abb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:03.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.924+0000 7f85d494e700 1 --2- 172.21.15.98:0/190488317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85d0068cd0 0x7f85d00abb10 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f85d0070b70 tx=0x7f85c800ba50 comp rx=0 tx=0).stop 2023-10-06T05:07:03.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.924+0000 7f85d494e700 1 -- 172.21.15.98:0/190488317 shutdown_connections 2023-10-06T05:07:03.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.924+0000 7f85d494e700 1 --2- 172.21.15.98:0/190488317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85d0068cd0 0x7f85d00abb10 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:03.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.924+0000 7f85d494e700 1 --2- 172.21.15.98:0/190488317 >> v2:172.21.15.174:3300/0 conn(0x7f85d0156280 0x7f85d015a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:03.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.924+0000 7f85d494e700 1 -- 172.21.15.98:0/190488317 >> 172.21.15.98:0/190488317 conn(0x7f85d006de40 msgr2=0x7f85d006e240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:03.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.924+0000 7f85d494e700 1 -- 172.21.15.98:0/190488317 shutdown_connections 2023-10-06T05:07:03.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.924+0000 7f85d494e700 1 -- 172.21.15.98:0/190488317 wait complete. 2023-10-06T05:07:03.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.925+0000 7f85d494e700 1 Processor -- start 2023-10-06T05:07:03.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.925+0000 7f85d494e700 1 -- start start 2023-10-06T05:07:03.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.925+0000 7f85d494e700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f85d0068cd0 0x7f85d014c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:03.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.925+0000 7f85d494e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85d0156280 0x7f85d0151040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:03.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.925+0000 7f85d494e700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f85d0156650 0x7f85d0154b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:03.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.925+0000 7f85d494e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85d0156c10 con 0x7f85d0156280 2023-10-06T05:07:03.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.925+0000 7f85d494e700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85d01695b0 con 0x7f85d0156650 2023-10-06T05:07:03.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.925+0000 7f85d494e700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85d0169890 con 0x7f85d0068cd0 2023-10-06T05:07:03.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.925+0000 7f85c7fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85d0156280 0x7f85d0151040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:03.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.926+0000 7f85c7fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85d0156280 0x7f85d0151040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47788/0 (socket says 172.21.15.98:47788) 2023-10-06T05:07:03.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.926+0000 7f85c7fff700 1 -- 172.21.15.98:0/1352837362 learned_addr learned my addr 172.21.15.98:0/1352837362 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:03.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.926+0000 7f85c7fff700 1 -- 172.21.15.98:0/1352837362 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f85d0068cd0 msgr2=0x7f85d014c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:03.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.926+0000 7f85c7fff700 1 --2- 172.21.15.98:0/1352837362 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f85d0068cd0 0x7f85d014c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:03.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.926+0000 7f85c7fff700 1 -- 172.21.15.98:0/1352837362 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f85d0156650 msgr2=0x7f85d0154b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:03.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.926+0000 7f85c7fff700 1 --2- 172.21.15.98:0/1352837362 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f85d0156650 0x7f85d0154b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:03.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.926+0000 7f85c7fff700 1 -- 172.21.15.98:0/1352837362 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85c800bcf0 con 0x7f85d0156280 2023-10-06T05:07:03.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.926+0000 7f85c7fff700 1 --2- 172.21.15.98:0/1352837362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85d0156280 0x7f85d0151040 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f85b800b5e0 tx=0x7f85b8009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:03.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.926+0000 7f85c67fc700 1 -- 172.21.15.98:0/1352837362 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b800f510 con 0x7f85d0156280 2023-10-06T05:07:03.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.926+0000 7f85c67fc700 1 -- 172.21.15.98:0/1352837362 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85b800f670 con 0x7f85d0156280 2023-10-06T05:07:03.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.926+0000 7f85c67fc700 1 -- 172.21.15.98:0/1352837362 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b800fa30 con 0x7f85d0156280 2023-10-06T05:07:03.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.926+0000 7f85d494e700 1 -- 172.21.15.98:0/1352837362 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85d0169b70 con 0x7f85d0156280 2023-10-06T05:07:03.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.926+0000 7f85d494e700 1 -- 172.21.15.98:0/1352837362 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85d014cd80 con 0x7f85d0156280 2023-10-06T05:07:03.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.927+0000 7f85d494e700 1 -- 172.21.15.98:0/1352837362 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85d0064a90 con 0x7f85d0156280 2023-10-06T05:07:03.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.927+0000 7f85c67fc700 1 -- 172.21.15.98:0/1352837362 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f85b801a400 con 0x7f85d0156280 2023-10-06T05:07:03.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.928+0000 7f85c67fc700 1 --2- 172.21.15.98:0/1352837362 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f85b0066f30 0x7f85b00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:03.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.928+0000 7f85ce59c700 1 --2- 172.21.15.98:0/1352837362 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f85b0066f30 0x7f85b00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:03.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.928+0000 7f85c67fc700 1 -- 172.21.15.98:0/1352837362 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f85b8084550 con 0x7f85d0156280 2023-10-06T05:07:03.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.928+0000 7f85ce59c700 1 --2- 172.21.15.98:0/1352837362 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f85b0066f30 0x7f85b00693e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f85c0009a40 tx=0x7f85c0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:03.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:03.932+0000 7f85c67fc700 1 -- 172.21.15.98:0/1352837362 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f85b80513f0 con 0x7f85d0156280 2023-10-06T05:07:04.160 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.158+0000 7f85d494e700 1 -- 172.21.15.98:0/1352837362 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f85d0156d50 con 0x7f85d0156280 2023-10-06T05:07:04.160 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.158+0000 7f85c67fc700 1 -- 172.21.15.98:0/1352837362 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4596 (secure 0 0 0) 0x7f85b8054a10 con 0x7f85d0156280 2023-10-06T05:07:04.161 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:07:04.161 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4285,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6834/925250632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":925250632},{"type":"v1","addr":"172.21.15.174:6834","nonce":925250632}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4288,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/1629449810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6833","nonce":1629449810},{"type":"v1","addr":"172.21.15.174:6835","nonce":1629449810}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4299,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/3961327990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3961327990},{"type":"v1","addr":"172.21.15.98:6836","nonce":3961327990}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-06T05:06:57.086683+0000","modified":"2023-10-06T05:07:02.755894+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.98:6837/2683239094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":2683239094},{"type":"v1","addr":"172.21.15.98:6837","nonce":2683239094}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2023-10-06T05:07:04.163 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.161+0000 7f85d494e700 1 -- 172.21.15.98:0/1352837362 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f85b0066f30 msgr2=0x7f85b00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:04.163 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.161+0000 7f85d494e700 1 --2- 172.21.15.98:0/1352837362 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f85b0066f30 0x7f85b00693e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f85c0009a40 tx=0x7f85c0008040 comp rx=0 tx=0).stop 2023-10-06T05:07:04.163 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.161+0000 7f85d494e700 1 -- 172.21.15.98:0/1352837362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85d0156280 msgr2=0x7f85d0151040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:04.163 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.161+0000 7f85d494e700 1 --2- 172.21.15.98:0/1352837362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85d0156280 0x7f85d0151040 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f85b800b5e0 tx=0x7f85b8009120 comp rx=0 tx=0).stop 2023-10-06T05:07:04.163 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.161+0000 7f85d494e700 1 -- 172.21.15.98:0/1352837362 shutdown_connections 2023-10-06T05:07:04.163 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.161+0000 7f85d494e700 1 --2- 172.21.15.98:0/1352837362 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f85d0156650 0x7f85d0154b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:04.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.161+0000 7f85d494e700 1 --2- 172.21.15.98:0/1352837362 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f85d0068cd0 0x7f85d014c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:04.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.161+0000 7f85d494e700 1 --2- 172.21.15.98:0/1352837362 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f85b0066f30 0x7f85b00693e0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:04.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.161+0000 7f85d494e700 1 --2- 172.21.15.98:0/1352837362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85d0156280 0x7f85d0151040 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:04.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.161+0000 7f85d494e700 1 -- 172.21.15.98:0/1352837362 >> 172.21.15.98:0/1352837362 conn(0x7f85d006de40 msgr2=0x7f85d014e9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:04.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.161+0000 7f85d494e700 1 -- 172.21.15.98:0/1352837362 shutdown_connections 2023-10-06T05:07:04.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.161+0000 7f85d494e700 1 -- 172.21.15.98:0/1352837362 wait complete. 2023-10-06T05:07:04.164 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 6 2023-10-06T05:07:04.175 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T05:07:04.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.286+0000 7f0006e61700 1 Processor -- start 2023-10-06T05:07:04.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.286+0000 7f0006e61700 1 -- start start 2023-10-06T05:07:04.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.287+0000 7f0006e61700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0000157040 0x7f00000cde60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:04.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.287+0000 7f0006e61700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f0000157410 0x7f000006d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:04.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.287+0000 7f0006e61700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f000015b1d0 con 0x7f000015b660 2023-10-06T05:07:04.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.287+0000 7f0006e61700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f000006dae0 con 0x7f0000157410 2023-10-06T05:07:04.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.287+0000 7f0006e61700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00000ce4b0 con 0x7f0000157040 2023-10-06T05:07:04.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.288+0000 7f0005e5f700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0000157040 0x7f00000cde60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:04.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.288+0000 7f000565e700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f0000157410 0x7f000006d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:04.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.288+0000 7f0005e5f700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0000157040 0x7f00000cde60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.98:41346/0 (socket says 172.21.15.98:41346) 2023-10-06T05:07:04.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.288+0000 7f000565e700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f0000157410 0x7f000006d5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.98:57882/0 (socket says 172.21.15.98:57882) 2023-10-06T05:07:04.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.288+0000 7f0005e5f700 1 -- 172.21.15.98:0/689048886 learned_addr learned my addr 172.21.15.98:0/689048886 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:04.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.288+0000 7f0006660700 1 --1- 172.21.15.98:0/689048886 >> v1:172.21.15.174:6789/0 conn(0x7f000015b660 0x7f00000c9f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.98:60296/0 (socket says 172.21.15.98:60296) 2023-10-06T05:07:04.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.288+0000 7f0005e5f700 1 -- 172.21.15.98:0/689048886 >> v2:172.21.15.174:3300/0 conn(0x7f0000157410 msgr2=0x7f000006d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:04.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.288+0000 7f0005e5f700 1 --2- 172.21.15.98:0/689048886 >> v2:172.21.15.174:3300/0 conn(0x7f0000157410 0x7f000006d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:04.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.288+0000 7f0005e5f700 1 -- 172.21.15.98:0/689048886 >> v1:172.21.15.174:6789/0 conn(0x7f000015b660 legacy=0x7f00000c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:04.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.288+0000 7f0005e5f700 1 -- 172.21.15.98:0/689048886 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00000ce790 con 0x7f0000157040 2023-10-06T05:07:04.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.288+0000 7f000565e700 1 --2- 172.21.15.98:0/689048886 >> v2:172.21.15.174:3300/0 conn(0x7f0000157410 0x7f000006d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T05:07:04.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.288+0000 7f0005e5f700 1 --2- 172.21.15.98:0/689048886 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0000157040 0x7f00000cde60 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7efffc0099b0 tx=0x7efffc009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=29ce7b10bea9b273 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:04.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.288+0000 7f0004e5d700 1 -- 172.21.15.98:0/689048886 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efffc009ea0 con 0x7f0000157040 2023-10-06T05:07:04.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.288+0000 7f0004e5d700 1 -- 172.21.15.98:0/689048886 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efffc00b860 con 0x7f0000157040 2023-10-06T05:07:04.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.288+0000 7f0004e5d700 1 -- 172.21.15.98:0/689048886 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efffc00bc00 con 0x7f0000157040 2023-10-06T05:07:04.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.289+0000 7f0006e61700 1 -- 172.21.15.98:0/689048886 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0000157040 msgr2=0x7f00000cde60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:04.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.289+0000 7f0006e61700 1 --2- 172.21.15.98:0/689048886 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0000157040 0x7f00000cde60 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7efffc0099b0 tx=0x7efffc009250 comp rx=0 tx=0).stop 2023-10-06T05:07:04.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.289+0000 7f0006e61700 1 -- 172.21.15.98:0/689048886 shutdown_connections 2023-10-06T05:07:04.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.289+0000 7f0006e61700 1 --2- 172.21.15.98:0/689048886 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0000157040 0x7f00000cde60 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:04.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.289+0000 7f0006e61700 1 --2- 172.21.15.98:0/689048886 >> v2:172.21.15.174:3300/0 conn(0x7f0000157410 0x7f000006d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:04.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.289+0000 7f0006e61700 1 -- 172.21.15.98:0/689048886 >> 172.21.15.98:0/689048886 conn(0x7f000006b500 msgr2=0x7f000006b900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:04.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.289+0000 7f0006e61700 1 -- 172.21.15.98:0/689048886 shutdown_connections 2023-10-06T05:07:04.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.289+0000 7f0006e61700 1 -- 172.21.15.98:0/689048886 wait complete. 2023-10-06T05:07:04.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.290+0000 7f0006e61700 1 Processor -- start 2023-10-06T05:07:04.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.290+0000 7f0006e61700 1 -- start start 2023-10-06T05:07:04.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.291+0000 7f0006e61700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0000157040 0x7f0000165860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:04.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.291+0000 7f0006e61700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0000157410 0x7f00000aecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:04.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.291+0000 7f0006e61700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f000015b660 0x7f000014de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:04.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.291+0000 7f0006e61700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0000165eb0 con 0x7f000015b660 2023-10-06T05:07:04.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.291+0000 7f0006e61700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0000166190 con 0x7f0000157040 2023-10-06T05:07:04.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.291+0000 7f0006e61700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0000166470 con 0x7f0000157410 2023-10-06T05:07:04.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.291+0000 7f0006660700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f000015b660 0x7f000014de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:04.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.291+0000 7f0006660700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f000015b660 0x7f000014de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47802/0 (socket says 172.21.15.98:47802) 2023-10-06T05:07:04.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.291+0000 7f0006660700 1 -- 172.21.15.98:0/1484532967 learned_addr learned my addr 172.21.15.98:0/1484532967 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:04.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.291+0000 7f000565e700 1 --2- 172.21.15.98:0/1484532967 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0000157410 0x7f00000aecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:04.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.291+0000 7f0005e5f700 1 --2- 172.21.15.98:0/1484532967 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0000157040 0x7f0000165860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:04.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.291+0000 7f0006660700 1 -- 172.21.15.98:0/1484532967 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0000157410 msgr2=0x7f00000aecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:04.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.291+0000 7f0006660700 1 --2- 172.21.15.98:0/1484532967 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0000157410 0x7f00000aecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:04.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.291+0000 7f0006660700 1 -- 172.21.15.98:0/1484532967 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0000157040 msgr2=0x7f0000165860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:04.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.291+0000 7f0006660700 1 --2- 172.21.15.98:0/1484532967 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0000157040 0x7f0000165860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:04.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.291+0000 7f0006660700 1 -- 172.21.15.98:0/1484532967 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efff000d8d0 con 0x7f000015b660 2023-10-06T05:07:04.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.291+0000 7f000565e700 1 --2- 172.21.15.98:0/1484532967 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0000157410 0x7f00000aecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:07:04.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.291+0000 7f0006660700 1 --2- 172.21.15.98:0/1484532967 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f000015b660 0x7f000014de90 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7efff800cd00 tx=0x7efff8001b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:04.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.292+0000 7efff5ffb700 1 -- 172.21.15.98:0/1484532967 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efff800ce60 con 0x7f000015b660 2023-10-06T05:07:04.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.292+0000 7efff5ffb700 1 -- 172.21.15.98:0/1484532967 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efff8005df0 con 0x7f000015b660 2023-10-06T05:07:04.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.292+0000 7efff5ffb700 1 -- 172.21.15.98:0/1484532967 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efff800abb0 con 0x7f000015b660 2023-10-06T05:07:04.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.292+0000 7f0006e61700 1 -- 172.21.15.98:0/1484532967 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efffc0094f0 con 0x7f000015b660 2023-10-06T05:07:04.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.292+0000 7f0006e61700 1 -- 172.21.15.98:0/1484532967 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0000168ee0 con 0x7f000015b660 2023-10-06T05:07:04.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.293+0000 7f0006e61700 1 -- 172.21.15.98:0/1484532967 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0000169100 con 0x7f000015b660 2023-10-06T05:07:04.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.293+0000 7efff5ffb700 1 -- 172.21.15.98:0/1484532967 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7efff80070f0 con 0x7f000015b660 2023-10-06T05:07:04.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.294+0000 7efff5ffb700 1 --2- 172.21.15.98:0/1484532967 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7effe4066f30 0x7effe40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:04.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.294+0000 7f0005e5f700 1 --2- 172.21.15.98:0/1484532967 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7effe4066f30 0x7effe40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:04.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.294+0000 7efff5ffb700 1 -- 172.21.15.98:0/1484532967 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7efff8084a00 con 0x7f000015b660 2023-10-06T05:07:04.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.294+0000 7f0005e5f700 1 --2- 172.21.15.98:0/1484532967 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7effe4066f30 0x7effe40693e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7efffc0099b0 tx=0x7efffc017000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:04.300 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.298+0000 7efff5ffb700 1 -- 172.21.15.98:0/1484532967 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7efff80518a0 con 0x7f000015b660 2023-10-06T05:07:04.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.480+0000 7f0006e61700 1 -- 172.21.15.98:0/1484532967 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f000005f230 con 0x7f000015b660 2023-10-06T05:07:04.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.481+0000 7efff5ffb700 1 -- 172.21.15.98:0/1484532967 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+18707 (secure 0 0 0) 0x7efff8054ec0 con 0x7f000015b660 2023-10-06T05:07:04.484 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:07:04.484 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":29,"fsid":"f50c5df5-cad5-47a4-9bb1-8c791d4311ec","created":"2023-10-06T05:06:32.971175+0000","modified":"2023-10-06T05:07:02.131672+0000","last_up_change":"2023-10-06T05:06:44.016489+0000","last_in_change":"2023-10-06T05:06:35.544511+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-06T05:06:44.559671+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-06T05:06:51.432161+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-10-06T05:06:55.589057+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-10-06T05:06:56.041687+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-10-06T05:06:59.754099+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d9495d95-da75-4c3b-8927-e9bae6a8b093","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":90627},{"type":"v1","addr":"172.21.15.98:6801","nonce":90627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":90627},{"type":"v1","addr":"172.21.15.98:6803","nonce":90627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":90627},{"type":"v1","addr":"172.21.15.98:6807","nonce":90627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":90627},{"type":"v1","addr":"172.21.15.98:6805","nonce":90627}]},"public_addr":"172.21.15.98:6801/90627","cluster_addr":"172.21.15.98:6803/90627","heartbeat_back_addr":"172.21.15.98:6807/90627","heartbeat_front_addr":"172.21.15.98:6805/90627","state":["exists","up"]},{"osd":1,"uuid":"f25ba5be-d65e-4707-a134-154092deb754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":90628},{"type":"v1","addr":"172.21.15.98:6809","nonce":90628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":90628},{"type":"v1","addr":"172.21.15.98:6811","nonce":90628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":90628},{"type":"v1","addr":"172.21.15.98:6815","nonce":90628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":90628},{"type":"v1","addr":"172.21.15.98:6813","nonce":90628}]},"public_addr":"172.21.15.98:6809/90628","cluster_addr":"172.21.15.98:6811/90628","heartbeat_back_addr":"172.21.15.98:6815/90628","heartbeat_front_addr":"172.21.15.98:6813/90628","state":["exists","up"]},{"osd":2,"uuid":"5ab9e028-945c-43e2-8c95-3b0b4cf38e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":90662},{"type":"v1","addr":"172.21.15.98:6818","nonce":90662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":90662},{"type":"v1","addr":"172.21.15.98:6822","nonce":90662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":90662},{"type":"v1","addr":"172.21.15.98:6830","nonce":90662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":90662},{"type":"v1","addr":"172.21.15.98:6826","nonce":90662}]},"public_addr":"172.21.15.98:6818/90662","cluster_addr":"172.21.15.98:6822/90662","heartbeat_back_addr":"172.21.15.98:6830/90662","heartbeat_front_addr":"172.21.15.98:6826/90662","state":["exists","up"]},{"osd":3,"uuid":"3bb803ff-1738-46bf-be62-2267bf378822","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6817","nonce":90663},{"type":"v1","addr":"172.21.15.98:6819","nonce":90663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6821","nonce":90663},{"type":"v1","addr":"172.21.15.98:6823","nonce":90663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6829","nonce":90663},{"type":"v1","addr":"172.21.15.98:6831","nonce":90663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6825","nonce":90663},{"type":"v1","addr":"172.21.15.98:6827","nonce":90663}]},"public_addr":"172.21.15.98:6819/90663","cluster_addr":"172.21.15.98:6823/90663","heartbeat_back_addr":"172.21.15.98:6831/90663","heartbeat_front_addr":"172.21.15.98:6827/90663","state":["exists","up"]},{"osd":4,"uuid":"7d57a4c9-6462-43f6-b64a-7f9e9b4353d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":90847},{"type":"v1","addr":"172.21.15.174:6801","nonce":90847}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":90847},{"type":"v1","addr":"172.21.15.174:6803","nonce":90847}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":90847},{"type":"v1","addr":"172.21.15.174:6807","nonce":90847}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":90847},{"type":"v1","addr":"172.21.15.174:6805","nonce":90847}]},"public_addr":"172.21.15.174:6801/90847","cluster_addr":"172.21.15.174:6803/90847","heartbeat_back_addr":"172.21.15.174:6807/90847","heartbeat_front_addr":"172.21.15.174:6805/90847","state":["exists","up"]},{"osd":5,"uuid":"83287ea4-e44d-4af9-bc77-36d47c60d4a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":90848},{"type":"v1","addr":"172.21.15.174:6809","nonce":90848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":90848},{"type":"v1","addr":"172.21.15.174:6811","nonce":90848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":90848},{"type":"v1","addr":"172.21.15.174:6815","nonce":90848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":90848},{"type":"v1","addr":"172.21.15.174:6813","nonce":90848}]},"public_addr":"172.21.15.174:6809/90848","cluster_addr":"172.21.15.174:6811/90848","heartbeat_back_addr":"172.21.15.174:6815/90848","heartbeat_front_addr":"172.21.15.174:6813/90848","state":["exists","up"]},{"osd":6,"uuid":"496a6272-892e-43c4-a145-206623c22f99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":90877},{"type":"v1","addr":"172.21.15.174:6817","nonce":90877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":90877},{"type":"v1","addr":"172.21.15.174:6819","nonce":90877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":90877},{"type":"v1","addr":"172.21.15.174:6823","nonce":90877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":90877},{"type":"v1","addr":"172.21.15.174:6821","nonce":90877}]},"public_addr":"172.21.15.174:6817/90877","cluster_addr":"172.21.15.174:6819/90877","heartbeat_back_addr":"172.21.15.174:6823/90877","heartbeat_front_addr":"172.21.15.174:6821/90877","state":["exists","up"]},{"osd":7,"uuid":"49967e34-8b07-4bab-a13e-0c16d6a0f3a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":90878},{"type":"v1","addr":"172.21.15.174:6825","nonce":90878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":90878},{"type":"v1","addr":"172.21.15.174:6827","nonce":90878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":90878},{"type":"v1","addr":"172.21.15.174:6831","nonce":90878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":90878},{"type":"v1","addr":"172.21.15.174:6829","nonce":90878}]},"public_addr":"172.21.15.174:6825/90878","cluster_addr":"172.21.15.174:6827/90878","heartbeat_back_addr":"172.21.15.174:6831/90878","heartbeat_front_addr":"172.21.15.174:6829/90878","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.446479+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.514266+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.794108+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.810894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.731919+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.717756+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.737435+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.508682+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T05:07:04.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.483+0000 7f0006e61700 1 -- 172.21.15.98:0/1484532967 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7effe4066f30 msgr2=0x7effe40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:04.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.483+0000 7f0006e61700 1 --2- 172.21.15.98:0/1484532967 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7effe4066f30 0x7effe40693e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7efffc0099b0 tx=0x7efffc017000 comp rx=0 tx=0).stop 2023-10-06T05:07:04.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.483+0000 7f0006e61700 1 -- 172.21.15.98:0/1484532967 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f000015b660 msgr2=0x7f000014de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:04.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.483+0000 7f0006e61700 1 --2- 172.21.15.98:0/1484532967 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f000015b660 0x7f000014de90 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7efff800cd00 tx=0x7efff8001b10 comp rx=0 tx=0).stop 2023-10-06T05:07:04.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.483+0000 7f0006e61700 1 -- 172.21.15.98:0/1484532967 shutdown_connections 2023-10-06T05:07:04.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.483+0000 7f0006e61700 1 --2- 172.21.15.98:0/1484532967 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7effe4066f30 0x7effe40693e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:04.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.483+0000 7f0006e61700 1 --2- 172.21.15.98:0/1484532967 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f000015b660 0x7f000014de90 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:04.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.483+0000 7f0006e61700 1 --2- 172.21.15.98:0/1484532967 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0000157040 0x7f0000165860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:04.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.483+0000 7f0006e61700 1 --2- 172.21.15.98:0/1484532967 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0000157410 0x7f00000aecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:04.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.483+0000 7f0006e61700 1 -- 172.21.15.98:0/1484532967 >> 172.21.15.98:0/1484532967 conn(0x7f000006b500 msgr2=0x7f00000686c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:04.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.483+0000 7f0006e61700 1 -- 172.21.15.98:0/1484532967 shutdown_connections 2023-10-06T05:07:04.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:04.483+0000 7f0006e61700 1 -- 172.21.15.98:0/1484532967 wait complete. 2023-10-06T05:07:04.902 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 6, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-06T05:06:57.086683+0000', 'modified': '2023-10-06T05:07:02.755894+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4302}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4302': {'gid': 4302, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.98:6837/2683239094', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 2683239094}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 2683239094}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [4, 5], 'metadata_pool': 3, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2023-10-06T05:07:04.902 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-06T05:07:04.903 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-10-06T05:07:05.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.016+0000 7f96bc396700 1 Processor -- start 2023-10-06T05:07:05.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.016+0000 7f96bc396700 1 -- start start 2023-10-06T05:07:05.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.017+0000 7f96bc396700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f96b4159560 0x7f96b40aa530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:05.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.017+0000 7f96bc396700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f96b40b49a0 0x7f96b40aaa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:05.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.017+0000 7f96bc396700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96b40b0c60 con 0x7f96b40b0760 2023-10-06T05:07:05.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.017+0000 7f96bc396700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96b40b0f40 con 0x7f96b4159560 2023-10-06T05:07:05.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.017+0000 7f96bc396700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96b40b1220 con 0x7f96b40b49a0 2023-10-06T05:07:05.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.017+0000 7f96ba132700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f96b4159560 0x7f96b40aa530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:05.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.017+0000 7f96ba132700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f96b4159560 0x7f96b40aa530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47814/0 (socket says 172.21.15.98:47814) 2023-10-06T05:07:05.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.017+0000 7f96ba132700 1 -- 172.21.15.98:0/3963463734 learned_addr learned my addr 172.21.15.98:0/3963463734 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:05.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.018+0000 7f96b9931700 1 --2- 172.21.15.98:0/3963463734 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f96b40b49a0 0x7f96b40aaa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:05.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.018+0000 7f96ba132700 1 -- 172.21.15.98:0/3963463734 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f96b40b49a0 msgr2=0x7f96b40aaa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:05.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.018+0000 7f96ba132700 1 --2- 172.21.15.98:0/3963463734 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f96b40b49a0 0x7f96b40aaa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:05.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.018+0000 7f96ba132700 1 -- 172.21.15.98:0/3963463734 >> v1:172.21.15.174:6789/0 conn(0x7f96b40b0760 legacy=0x7f96b40b3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:05.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.018+0000 7f96ba132700 1 -- 172.21.15.98:0/3963463734 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96b40b1500 con 0x7f96b4159560 2023-10-06T05:07:05.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.018+0000 7f96ba132700 1 --2- 172.21.15.98:0/3963463734 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96b4159560 0x7f96b40aa530 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f96b0009f70 tx=0x7f96b0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ba91946a6d2188e server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:05.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.018+0000 7f96b9130700 1 -- 172.21.15.98:0/3963463734 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96b000e070 con 0x7f96b4159560 2023-10-06T05:07:05.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.018+0000 7f96b9130700 1 -- 172.21.15.98:0/3963463734 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96b000bad0 con 0x7f96b4159560 2023-10-06T05:07:05.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.018+0000 7f96b9130700 1 -- 172.21.15.98:0/3963463734 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96b00092e0 con 0x7f96b4159560 2023-10-06T05:07:05.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.018+0000 7f96bc396700 1 -- 172.21.15.98:0/3963463734 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96b4159560 msgr2=0x7f96b40aa530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:05.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.018+0000 7f96bc396700 1 --2- 172.21.15.98:0/3963463734 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96b4159560 0x7f96b40aa530 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f96b0009f70 tx=0x7f96b0009840 comp rx=0 tx=0).stop 2023-10-06T05:07:05.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.019+0000 7f96bc396700 1 -- 172.21.15.98:0/3963463734 shutdown_connections 2023-10-06T05:07:05.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.019+0000 7f96bc396700 1 --2- 172.21.15.98:0/3963463734 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96b4159560 0x7f96b40aa530 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:05.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.019+0000 7f96bc396700 1 --2- 172.21.15.98:0/3963463734 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f96b40b49a0 0x7f96b40aaa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:05.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.019+0000 7f96bc396700 1 -- 172.21.15.98:0/3963463734 >> 172.21.15.98:0/3963463734 conn(0x7f96b40997f0 msgr2=0x7f96b406f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:05.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.019+0000 7f96bc396700 1 -- 172.21.15.98:0/3963463734 shutdown_connections 2023-10-06T05:07:05.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.019+0000 7f96bc396700 1 -- 172.21.15.98:0/3963463734 wait complete. 2023-10-06T05:07:05.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.020+0000 7f96bc396700 1 Processor -- start 2023-10-06T05:07:05.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.020+0000 7f96bc396700 1 -- start start 2023-10-06T05:07:05.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.020+0000 7f96bc396700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f96b40b0760 0x7f96b409d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:05.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.020+0000 7f96bc396700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96b40b49a0 0x7f96b4069250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:05.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.020+0000 7f96bc396700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f96b4159560 0x7f96b409ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:05.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.020+0000 7f96bc396700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96b409aa10 con 0x7f96b40b49a0 2023-10-06T05:07:05.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.020+0000 7f96bc396700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96b409acb0 con 0x7f96b4159560 2023-10-06T05:07:05.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.020+0000 7f96bc396700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96b409af90 con 0x7f96b40b0760 2023-10-06T05:07:05.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.020+0000 7f96b9931700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96b40b49a0 0x7f96b4069250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:05.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.020+0000 7f96b9931700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96b40b49a0 0x7f96b4069250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47826/0 (socket says 172.21.15.98:47826) 2023-10-06T05:07:05.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.020+0000 7f96b9931700 1 -- 172.21.15.98:0/3634738265 learned_addr learned my addr 172.21.15.98:0/3634738265 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:05.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.021+0000 7f96ba132700 1 --2- 172.21.15.98:0/3634738265 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f96b40b0760 0x7f96b409d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:05.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.021+0000 7f96ba933700 1 --2- 172.21.15.98:0/3634738265 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f96b4159560 0x7f96b409ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:05.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.021+0000 7f96b9931700 1 -- 172.21.15.98:0/3634738265 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f96b40b0760 msgr2=0x7f96b409d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:05.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.021+0000 7f96b9931700 1 --2- 172.21.15.98:0/3634738265 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f96b40b0760 0x7f96b409d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:05.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.021+0000 7f96b9931700 1 -- 172.21.15.98:0/3634738265 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f96b4159560 msgr2=0x7f96b409ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:05.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.021+0000 7f96b9931700 1 --2- 172.21.15.98:0/3634738265 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f96b4159560 0x7f96b409ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:05.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.021+0000 7f96b9931700 1 -- 172.21.15.98:0/3634738265 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96b0009ae0 con 0x7f96b40b49a0 2023-10-06T05:07:05.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.021+0000 7f96b9931700 1 --2- 172.21.15.98:0/3634738265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96b40b49a0 0x7f96b4069250 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f96a400d810 tx=0x7f96a4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:05.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.021+0000 7f96aa7fc700 1 -- 172.21.15.98:0/3634738265 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96a40125f0 con 0x7f96b40b49a0 2023-10-06T05:07:05.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.021+0000 7f96aa7fc700 1 -- 172.21.15.98:0/3634738265 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96a4012750 con 0x7f96b40b49a0 2023-10-06T05:07:05.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.021+0000 7f96aa7fc700 1 -- 172.21.15.98:0/3634738265 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96a4012b10 con 0x7f96b40b49a0 2023-10-06T05:07:05.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.021+0000 7f96bc396700 1 -- 172.21.15.98:0/3634738265 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96b409b270 con 0x7f96b40b49a0 2023-10-06T05:07:05.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.021+0000 7f96bc396700 1 -- 172.21.15.98:0/3634738265 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96b409dca0 con 0x7f96b40b49a0 2023-10-06T05:07:05.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.023+0000 7f96aa7fc700 1 -- 172.21.15.98:0/3634738265 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f96a401e460 con 0x7f96b40b49a0 2023-10-06T05:07:05.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.023+0000 7f96aa7fc700 1 --2- 172.21.15.98:0/3634738265 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f9698066f30 0x7f96980693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:05.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.023+0000 7f96aa7fc700 1 -- 172.21.15.98:0/3634738265 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f96a4084f30 con 0x7f96b40b49a0 2023-10-06T05:07:05.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.023+0000 7f96ba132700 1 --2- 172.21.15.98:0/3634738265 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f9698066f30 0x7f96980693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:05.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.024+0000 7f96ba132700 1 --2- 172.21.15.98:0/3634738265 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f9698066f30 0x7f96980693e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f96b0009880 tx=0x7f96b0018000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:05.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.025+0000 7f96c568eb80 1 Processor -- start 2023-10-06T05:07:05.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.025+0000 7f96c568eb80 1 -- start start 2023-10-06T05:07:05.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.026+0000 7f96c568eb80 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55dad43cfa90 0x55dad43cfe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:05.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.026+0000 7f96c568eb80 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dad438b9a0 0x55dad4388580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:05.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.026+0000 7f96c568eb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dad4389bd0 0x55dad4388ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:05.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.026+0000 7f96c568eb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dad4308c80 con 0x55dad4389bd0 2023-10-06T05:07:05.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.026+0000 7f96c568eb80 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dad42dfa70 con 0x55dad438b9a0 2023-10-06T05:07:05.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.026+0000 7f96c568eb80 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x55dad42dd270 con 0x55dad43cfa90 2023-10-06T05:07:05.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.026+0000 7f96ba933700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dad4389bd0 0x55dad4388ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:05.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.026+0000 7f96ba933700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dad4389bd0 0x55dad4388ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47840/0 (socket says 172.21.15.98:47840) 2023-10-06T05:07:05.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.026+0000 7f96ba933700 1 -- 172.21.15.98:0/1184394413 learned_addr learned my addr 172.21.15.98:0/1184394413 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:05.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.026+0000 7f96b9931700 1 --2- 172.21.15.98:0/1184394413 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dad438b9a0 0x55dad4388580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:05.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.026+0000 7f96ba132700 1 --2- 172.21.15.98:0/1184394413 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55dad43cfa90 0x55dad43cfe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:05.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.026+0000 7f96ba933700 1 -- 172.21.15.98:0/1184394413 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55dad43cfa90 msgr2=0x55dad43cfe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:05.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.026+0000 7f96ba933700 1 --2- 172.21.15.98:0/1184394413 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55dad43cfa90 0x55dad43cfe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:05.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.026+0000 7f96ba933700 1 -- 172.21.15.98:0/1184394413 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dad438b9a0 msgr2=0x55dad4388580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:05.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.026+0000 7f96ba933700 1 --2- 172.21.15.98:0/1184394413 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dad438b9a0 0x55dad4388580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:05.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.026+0000 7f96ba933700 1 -- 172.21.15.98:0/1184394413 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dad4173270 con 0x55dad4389bd0 2023-10-06T05:07:05.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.026+0000 7f96ba933700 1 --2- 172.21.15.98:0/1184394413 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dad4389bd0 0x55dad4388ac0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f96ac003840 tx=0x7f96ac007e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c0ab37e94f23e21 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:05.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.026+0000 7f96977fe700 1 -- 172.21.15.98:0/1184394413 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96ac003430 con 0x55dad4389bd0 2023-10-06T05:07:05.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.026+0000 7f96977fe700 1 -- 172.21.15.98:0/1184394413 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96ac002770 con 0x55dad4389bd0 2023-10-06T05:07:05.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.026+0000 7f96977fe700 1 -- 172.21.15.98:0/1184394413 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96ac002af0 con 0x55dad4389bd0 2023-10-06T05:07:05.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.027+0000 7f96c568eb80 1 -- 172.21.15.98:0/1184394413 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dad4389bd0 msgr2=0x55dad4388ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:05.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.027+0000 7f96c568eb80 1 --2- 172.21.15.98:0/1184394413 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dad4389bd0 0x55dad4388ac0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f96ac003840 tx=0x7f96ac007e60 comp rx=0 tx=0).stop 2023-10-06T05:07:05.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.027+0000 7f96c568eb80 1 -- 172.21.15.98:0/1184394413 shutdown_connections 2023-10-06T05:07:05.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.027+0000 7f96c568eb80 1 --2- 172.21.15.98:0/1184394413 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dad4389bd0 0x55dad4388ac0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:05.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.027+0000 7f96c568eb80 1 --2- 172.21.15.98:0/1184394413 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55dad43cfa90 0x55dad43cfe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:05.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.027+0000 7f96c568eb80 1 --2- 172.21.15.98:0/1184394413 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dad438b9a0 0x55dad4388580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:05.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.027+0000 7f96c568eb80 1 -- 172.21.15.98:0/1184394413 >> 172.21.15.98:0/1184394413 conn(0x55dad438a070 msgr2=0x55dad438a440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:05.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.027+0000 7f96c568eb80 1 -- 172.21.15.98:0/1184394413 shutdown_connections 2023-10-06T05:07:05.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.027+0000 7f96c568eb80 1 -- 172.21.15.98:0/1184394413 wait complete. 2023-10-06T05:07:05.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.028+0000 7f96c568eb80 1 -- start start 2023-10-06T05:07:05.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.028+0000 7f96c568eb80 1 Processor -- start 2023-10-06T05:07:05.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.029+0000 7f96c568eb80 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dad4389bd0 0x55dad43b1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:05.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.029+0000 7f96c568eb80 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55dad438b9a0 0x55dad43caa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:05.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.029+0000 7f96c568eb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dad43cfa90 0x55dad43caf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:05.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.029+0000 7f96c568eb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dad42d9060 con 0x55dad43cfa90 2023-10-06T05:07:05.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.029+0000 7f96c568eb80 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dad4284890 con 0x55dad4389bd0 2023-10-06T05:07:05.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.029+0000 7f96c568eb80 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x55dad429fea0 con 0x55dad438b9a0 2023-10-06T05:07:05.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.029+0000 7f96c568eb80 10 client.0 _ll_register_callbacks cb 0x55dad4175f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-06T05:07:05.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.029+0000 7f96ba933700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dad43cfa90 0x55dad43caf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:05.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.029+0000 7f96ba933700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dad43cfa90 0x55dad43caf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47844/0 (socket says 172.21.15.98:47844) 2023-10-06T05:07:05.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.029+0000 7f96ba933700 1 -- 172.21.15.98:0/1089268822 learned_addr learned my addr 172.21.15.98:0/1089268822 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:05.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.029+0000 7f96b9931700 1 --2- 172.21.15.98:0/1089268822 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55dad438b9a0 0x55dad43caa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:05.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.029+0000 7f96ba132700 1 --2- 172.21.15.98:0/1089268822 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dad4389bd0 0x55dad43b1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:05.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.029+0000 7f96ba933700 1 -- 172.21.15.98:0/1089268822 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55dad438b9a0 msgr2=0x55dad43caa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:05.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.029+0000 7f96ba933700 1 --2- 172.21.15.98:0/1089268822 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55dad438b9a0 0x55dad43caa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:05.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.029+0000 7f96ba933700 1 -- 172.21.15.98:0/1089268822 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dad4389bd0 msgr2=0x55dad43b1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:05.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.029+0000 7f96ba933700 1 --2- 172.21.15.98:0/1089268822 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dad4389bd0 0x55dad43b1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:05.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.029+0000 7f96ba933700 1 -- 172.21.15.98:0/1089268822 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96b40b0c60 con 0x55dad43cfa90 2023-10-06T05:07:05.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.029+0000 7f96ba933700 1 --2- 172.21.15.98:0/1089268822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dad43cfa90 0x55dad43caf70 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f96ac003f10 tx=0x7f96ac018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f257a646e770303e server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:05.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.029+0000 7f9694ff9700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2023-10-06T05:07:05.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.029+0000 7f9694ff9700 1 -- 172.21.15.98:0/1089268822 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96ac01c630 con 0x55dad43cfa90 2023-10-06T05:07:05.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.029+0000 7f96c568eb80 1 -- 172.21.15.98:0/1089268822 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55dad437d690 con 0x55dad43cfa90 2023-10-06T05:07:05.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.029+0000 7f9694ff9700 1 -- 172.21.15.98:0/1089268822 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f96ac01c790 con 0x55dad43cfa90 2023-10-06T05:07:05.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.029+0000 7f9694ff9700 1 -- 172.21.15.98:0/1089268822 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=6) v2 ==== 24+0+0 (secure 0 0 0) 0x7f96ac01cba0 con 0x55dad43cfa90 2023-10-06T05:07:05.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.029+0000 7f96c568eb80 10 client.4458 fetch_fsmap learned FSMap version 6 2023-10-06T05:07:05.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.029+0000 7f96c568eb80 1 -- 172.21.15.98:0/1089268822 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=6,monmap=2+}) v3 -- 0x55dad4115fe0 con 0x55dad43cfa90 2023-10-06T05:07:05.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.030+0000 7f9694ff9700 1 -- 172.21.15.98:0/1089268822 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55dad437d690 con 0x55dad43cfa90 2023-10-06T05:07:05.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.030+0000 7f9694ff9700 1 -- 172.21.15.98:0/1089268822 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 6) v1 ==== 3108+0+0 (secure 0 0 0) 0x7f96ac012070 con 0x55dad43cfa90 2023-10-06T05:07:05.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.030+0000 7f96c568eb80 10 client.4458 fetch_fsmap finished waiting for FSMap version 6 2023-10-06T05:07:05.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.030+0000 7f96c568eb80 10 client.4458 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-06T05:07:05.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.030+0000 7f96c568eb80 20 client.4458 populate_metadata read hostname 'smithi098' 2023-10-06T05:07:05.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.030+0000 7f96c568eb80 1 --2- 172.21.15.98:0/1089268822 >> [v2:172.21.15.98:6835/2683239094,v1:172.21.15.98:6837/2683239094] conn(0x55dad43aee80 0x55dad43af250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:05.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.030+0000 7f96c568eb80 4 client.4458 mds_command: new command op to 4302 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-06T05:07:05.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.030+0000 7f96c568eb80 1 -- 172.21.15.98:0/1089268822 --> [v2:172.21.15.98:6835/2683239094,v1:172.21.15.98:6837/2683239094] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55dad435a5c0 con 0x55dad43aee80 2023-10-06T05:07:05.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.030+0000 7f96ba132700 1 --2- 172.21.15.98:0/1089268822 >> [v2:172.21.15.98:6835/2683239094,v1:172.21.15.98:6837/2683239094] conn(0x55dad43aee80 0x55dad43af250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:05.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.030+0000 7f9694ff9700 10 client.4458 ms_handle_connect on v2:172.21.15.98:6835/2683239094 2023-10-06T05:07:05.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.030+0000 7f96ba132700 1 --2- 172.21.15.98:0/1089268822 >> [v2:172.21.15.98:6835/2683239094,v1:172.21.15.98:6837/2683239094] conn(0x55dad43aee80 0x55dad43af250 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aa5a30d0e50fcb58 server_cookie=aa76cf3d1266daad in_seq=0 out_seq=0 2023-10-06T05:07:05.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.031+0000 7f9694ff9700 1 -- 172.21.15.98:0/1089268822 <== mds.0 v2:172.21.15.98:6835/2683239094 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+16954 (crc 0 0 0) 0x55dad435a5c0 con 0x55dad43aee80 2023-10-06T05:07:05.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.031+0000 7f9694ff9700 10 client.4458 handle_command_reply: tid=0 2023-10-06T05:07:05.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.031+0000 7f96c568eb80 1 client.4458 shutdown 2023-10-06T05:07:05.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.031+0000 7f96c568eb80 1 -- 172.21.15.98:0/1089268822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dad43cfa90 msgr2=0x55dad43caf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:05.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.031+0000 7f96c568eb80 1 --2- 172.21.15.98:0/1089268822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dad43cfa90 0x55dad43caf70 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f96ac003f10 tx=0x7f96ac018000 comp rx=0 tx=0).stop 2023-10-06T05:07:05.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.031+0000 7f96c568eb80 1 -- 172.21.15.98:0/1089268822 shutdown_connections 2023-10-06T05:07:05.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.031+0000 7f96c568eb80 1 --2- 172.21.15.98:0/1089268822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dad43cfa90 0x55dad43caf70 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:05.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.031+0000 7f96c568eb80 1 --2- 172.21.15.98:0/1089268822 >> [v2:172.21.15.98:6835/2683239094,v1:172.21.15.98:6837/2683239094] conn(0x55dad43aee80 0x55dad43af250 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:05.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.031+0000 7f96c568eb80 1 --2- 172.21.15.98:0/1089268822 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dad4389bd0 0x55dad43b1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:05.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.031+0000 7f9694ff9700 0 client.4458 ms_handle_reset on v2:172.21.15.98:6835/2683239094 2023-10-06T05:07:05.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.031+0000 7f96c568eb80 1 --2- 172.21.15.98:0/1089268822 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55dad438b9a0 0x55dad43caa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:05.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.031+0000 7f96c568eb80 1 -- 172.21.15.98:0/1089268822 >> 172.21.15.98:0/1089268822 conn(0x55dad438a070 msgr2=0x55dad43cefe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:05.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.031+0000 7f96c568eb80 1 -- 172.21.15.98:0/1089268822 shutdown_connections 2023-10-06T05:07:05.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.031+0000 7f96c568eb80 1 -- 172.21.15.98:0/1089268822 wait complete. 2023-10-06T05:07:05.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.031+0000 7f96c568eb80 20 client.4458 trim_cache size 0 max 16384 2023-10-06T05:07:05.050 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.048+0000 7f96c568eb80 1 Processor -- start 2023-10-06T05:07:05.050 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.048+0000 7f96c568eb80 1 -- start start 2023-10-06T05:07:05.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.049+0000 7f96c568eb80 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dad4389bd0 0x55dad43de070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:05.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.049+0000 7f96c568eb80 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55dad438b9a0 0x55dad43af250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:05.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.049+0000 7f96c568eb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dad43cfa90 0x55dad43ce280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:05.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.049+0000 7f96c568eb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dad415be30 con 0x55dad43cfa90 2023-10-06T05:07:05.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.049+0000 7f96c568eb80 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dad43cfed0 con 0x55dad4389bd0 2023-10-06T05:07:05.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.049+0000 7f96c568eb80 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x55dad43ccb20 con 0x55dad438b9a0 2023-10-06T05:07:05.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.049+0000 7f96ba933700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dad43cfa90 0x55dad43ce280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:05.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.049+0000 7f96ba933700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dad43cfa90 0x55dad43ce280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47848/0 (socket says 172.21.15.98:47848) 2023-10-06T05:07:05.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.049+0000 7f96ba933700 1 -- 172.21.15.98:0/3539613714 learned_addr learned my addr 172.21.15.98:0/3539613714 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:05.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.049+0000 7f96ba132700 1 --2- 172.21.15.98:0/3539613714 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dad4389bd0 0x55dad43de070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:05.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.049+0000 7f96b9931700 1 --2- 172.21.15.98:0/3539613714 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55dad438b9a0 0x55dad43af250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:05.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.049+0000 7f96ba933700 1 -- 172.21.15.98:0/3539613714 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55dad438b9a0 msgr2=0x55dad43af250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:05.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.049+0000 7f96ba933700 1 --2- 172.21.15.98:0/3539613714 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55dad438b9a0 0x55dad43af250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:05.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.049+0000 7f96ba933700 1 -- 172.21.15.98:0/3539613714 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dad4389bd0 msgr2=0x55dad43de070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:05.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.049+0000 7f96ba933700 1 --2- 172.21.15.98:0/3539613714 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dad4389bd0 0x55dad43de070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:05.054 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.049+0000 7f96ba933700 1 -- 172.21.15.98:0/3539613714 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dad42a7780 con 0x55dad43cfa90 2023-10-06T05:07:05.054 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.049+0000 7f96ba933700 1 --2- 172.21.15.98:0/3539613714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dad43cfa90 0x55dad43ce280 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f96ac0011d0 tx=0x7f96ac021040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0f237739866e494 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:05.054 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.049+0000 7f9697fff700 1 -- 172.21.15.98:0/3539613714 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96ac003430 con 0x55dad43cfa90 2023-10-06T05:07:05.054 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.049+0000 7f9697fff700 1 -- 172.21.15.98:0/3539613714 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96ac002560 con 0x55dad43cfa90 2023-10-06T05:07:05.054 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.049+0000 7f9697fff700 1 -- 172.21.15.98:0/3539613714 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96ac01c630 con 0x55dad43cfa90 2023-10-06T05:07:05.054 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.050+0000 7f96c568eb80 1 -- 172.21.15.98:0/3539613714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dad43cfa90 msgr2=0x55dad43ce280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:05.054 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.050+0000 7f96c568eb80 1 --2- 172.21.15.98:0/3539613714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dad43cfa90 0x55dad43ce280 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f96ac0011d0 tx=0x7f96ac021040 comp rx=0 tx=0).stop 2023-10-06T05:07:05.054 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.050+0000 7f96c568eb80 1 -- 172.21.15.98:0/3539613714 shutdown_connections 2023-10-06T05:07:05.054 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.050+0000 7f96c568eb80 1 --2- 172.21.15.98:0/3539613714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dad43cfa90 0x55dad43ce280 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:05.054 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.050+0000 7f96c568eb80 1 --2- 172.21.15.98:0/3539613714 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dad4389bd0 0x55dad43de070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:05.054 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.050+0000 7f96c568eb80 1 --2- 172.21.15.98:0/3539613714 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55dad438b9a0 0x55dad43af250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:05.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.050+0000 7f96c568eb80 1 -- 172.21.15.98:0/3539613714 >> 172.21.15.98:0/3539613714 conn(0x55dad438a070 msgr2=0x55dad43cefe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:05.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.050+0000 7f96c568eb80 1 -- 172.21.15.98:0/3539613714 shutdown_connections 2023-10-06T05:07:05.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.050+0000 7f96c568eb80 1 -- 172.21.15.98:0/3539613714 wait complete. 2023-10-06T05:07:05.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.050+0000 7f96c568eb80 1 -- start start 2023-10-06T05:07:05.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.050+0000 7f96c568eb80 1 Processor -- start 2023-10-06T05:07:05.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.051+0000 7f96c568eb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dad4389bd0 0x55dad43c9530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:05.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.051+0000 7f96c568eb80 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55dad438b9a0 0x55dad43b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:05.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.051+0000 7f96c568eb80 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dad43cfa90 0x55dad43afdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:05.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.051+0000 7f96c568eb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dad43a8dd0 con 0x55dad4389bd0 2023-10-06T05:07:05.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.051+0000 7f96c568eb80 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dad43c9a70 con 0x55dad43cfa90 2023-10-06T05:07:05.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.051+0000 7f96c568eb80 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x55dad4383c10 con 0x55dad438b9a0 2023-10-06T05:07:05.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.051+0000 7f96c568eb80 10 client.0 _ll_register_callbacks cb 0x55dad4175f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-06T05:07:05.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.051+0000 7f96ba132700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dad4389bd0 0x55dad43c9530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:05.061 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.051+0000 7f96ba132700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dad4389bd0 0x55dad43c9530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47852/0 (socket says 172.21.15.98:47852) 2023-10-06T05:07:05.061 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.051+0000 7f96ba132700 1 -- 172.21.15.98:0/1649451313 learned_addr learned my addr 172.21.15.98:0/1649451313 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:05.061 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.051+0000 7f96b9931700 1 --2- 172.21.15.98:0/1649451313 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55dad438b9a0 0x55dad43b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:05.061 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.051+0000 7f96ba933700 1 --2- 172.21.15.98:0/1649451313 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dad43cfa90 0x55dad43afdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:05.061 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.051+0000 7f96ba132700 1 -- 172.21.15.98:0/1649451313 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55dad438b9a0 msgr2=0x55dad43b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:05.061 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.052+0000 7f96ba132700 1 --2- 172.21.15.98:0/1649451313 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55dad438b9a0 0x55dad43b4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:05.061 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.052+0000 7f96ba132700 1 -- 172.21.15.98:0/1649451313 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dad43cfa90 msgr2=0x55dad43afdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:05.061 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.052+0000 7f96ba132700 1 --2- 172.21.15.98:0/1649451313 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dad43cfa90 0x55dad43afdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:05.061 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.052+0000 7f96ba132700 1 -- 172.21.15.98:0/1649451313 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55dad4115fe0 con 0x55dad4389bd0 2023-10-06T05:07:05.061 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.052+0000 7f96b9931700 1 --2- 172.21.15.98:0/1649451313 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55dad438b9a0 0x55dad43b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:07:05.061 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.052+0000 7f96ba132700 1 --2- 172.21.15.98:0/1649451313 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dad4389bd0 0x55dad43c9530 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f96b0012000 tx=0x7f96b0016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a56627b915e6956 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:05.061 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.052+0000 7f9694ff9700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2023-10-06T05:07:05.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.052+0000 7f9694ff9700 1 -- 172.21.15.98:0/1649451313 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96b00344a0 con 0x55dad4389bd0 2023-10-06T05:07:05.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.052+0000 7f9694ff9700 1 -- 172.21.15.98:0/1649451313 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f96b0034600 con 0x55dad4389bd0 2023-10-06T05:07:05.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.052+0000 7f96c568eb80 1 -- 172.21.15.98:0/1649451313 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f96ac002770 con 0x55dad4389bd0 2023-10-06T05:07:05.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.052+0000 7f9694ff9700 1 -- 172.21.15.98:0/1649451313 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=6) v2 ==== 24+0+0 (secure 0 0 0) 0x7f96b00349d0 con 0x55dad4389bd0 2023-10-06T05:07:05.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.052+0000 7f96c568eb80 10 client.4464 fetch_fsmap learned FSMap version 6 2023-10-06T05:07:05.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.052+0000 7f96c568eb80 1 -- 172.21.15.98:0/1649451313 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=6,monmap=2+}) v3 -- 0x55dad428e470 con 0x55dad4389bd0 2023-10-06T05:07:05.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.052+0000 7f9694ff9700 1 -- 172.21.15.98:0/1649451313 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96ac002770 con 0x55dad4389bd0 2023-10-06T05:07:05.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.052+0000 7f9694ff9700 1 -- 172.21.15.98:0/1649451313 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 6) v1 ==== 3108+0+0 (secure 0 0 0) 0x7f96b0034b30 con 0x55dad4389bd0 2023-10-06T05:07:05.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.052+0000 7f96c568eb80 10 client.4464 fetch_fsmap finished waiting for FSMap version 6 2023-10-06T05:07:05.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.052+0000 7f96c568eb80 10 client.4464 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-06T05:07:05.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.052+0000 7f96c568eb80 20 client.4464 populate_metadata read hostname 'smithi098' 2023-10-06T05:07:05.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.052+0000 7f96c568eb80 1 --2- 172.21.15.98:0/1649451313 >> [v2:172.21.15.98:6835/2683239094,v1:172.21.15.98:6837/2683239094] conn(0x55dad43ca960 0x55dad43cad30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:05.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.052+0000 7f96c568eb80 4 client.4464 mds_command: new command op to 4302 tid=0[{"prefix": "status"}] 2023-10-06T05:07:05.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.052+0000 7f96c568eb80 1 -- 172.21.15.98:0/1649451313 --> [v2:172.21.15.98:6835/2683239094,v1:172.21.15.98:6837/2683239094] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55dad427be00 con 0x55dad43ca960 2023-10-06T05:07:05.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.052+0000 7f96b9931700 1 --2- 172.21.15.98:0/1649451313 >> [v2:172.21.15.98:6835/2683239094,v1:172.21.15.98:6837/2683239094] conn(0x55dad43ca960 0x55dad43cad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:05.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.053+0000 7f9694ff9700 10 client.4464 ms_handle_connect on v2:172.21.15.98:6835/2683239094 2023-10-06T05:07:05.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.053+0000 7f96b9931700 1 --2- 172.21.15.98:0/1649451313 >> [v2:172.21.15.98:6835/2683239094,v1:172.21.15.98:6837/2683239094] conn(0x55dad43ca960 0x55dad43cad30 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ff6b4aedbc826cd server_cookie=1a3a1196cbc9b734 in_seq=0 out_seq=0 2023-10-06T05:07:05.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.053+0000 7f9694ff9700 1 -- 172.21.15.98:0/1649451313 <== mds.0 v2:172.21.15.98:6835/2683239094 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x55dad427be00 con 0x55dad43ca960 2023-10-06T05:07:05.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.053+0000 7f9694ff9700 10 client.4464 handle_command_reply: tid=0 2023-10-06T05:07:05.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.053+0000 7f96c568eb80 1 client.4464 shutdown 2023-10-06T05:07:05.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.053+0000 7f96c568eb80 1 -- 172.21.15.98:0/1649451313 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dad4389bd0 msgr2=0x55dad43c9530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:05.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.053+0000 7f96c568eb80 1 --2- 172.21.15.98:0/1649451313 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dad4389bd0 0x55dad43c9530 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f96b0012000 tx=0x7f96b0016000 comp rx=0 tx=0).stop 2023-10-06T05:07:05.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.053+0000 7f96c568eb80 1 -- 172.21.15.98:0/1649451313 shutdown_connections 2023-10-06T05:07:05.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.053+0000 7f96c568eb80 1 --2- 172.21.15.98:0/1649451313 >> [v2:172.21.15.98:6835/2683239094,v1:172.21.15.98:6837/2683239094] conn(0x55dad43ca960 0x55dad43cad30 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:05.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.053+0000 7f96c568eb80 1 --2- 172.21.15.98:0/1649451313 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55dad43cfa90 0x55dad43afdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:05.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.053+0000 7f96c568eb80 1 --2- 172.21.15.98:0/1649451313 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55dad4389bd0 0x55dad43c9530 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:05.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.053+0000 7f9694ff9700 0 client.4464 ms_handle_reset on v2:172.21.15.98:6835/2683239094 2023-10-06T05:07:05.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.053+0000 7f96c568eb80 1 --2- 172.21.15.98:0/1649451313 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55dad438b9a0 0x55dad43b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:05.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.053+0000 7f96c568eb80 1 -- 172.21.15.98:0/1649451313 >> 172.21.15.98:0/1649451313 conn(0x55dad438a070 msgr2=0x55dad43b0c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:05.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.053+0000 7f96c568eb80 1 -- 172.21.15.98:0/1649451313 shutdown_connections 2023-10-06T05:07:05.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.053+0000 7f96c568eb80 1 -- 172.21.15.98:0/1649451313 wait complete. 2023-10-06T05:07:05.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.054+0000 7f96c568eb80 20 client.4464 trim_cache size 0 max 16384 2023-10-06T05:07:05.064 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-10-06T05:07:05.065 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "f50c5df5-cad5-47a4-9bb1-8c791d4311ec", 2023-10-06T05:07:05.065 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 0, 2023-10-06T05:07:05.065 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 4302, 2023-10-06T05:07:05.065 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2023-10-06T05:07:05.065 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2023-10-06T05:07:05.065 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2023-10-06T05:07:05.065 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 7.3562810660000002, 2023-10-06T05:07:05.065 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 6, 2023-10-06T05:07:05.065 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 29, 2023-10-06T05:07:05.065 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 29, 2023-10-06T05:07:05.065 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 12.372128817 2023-10-06T05:07:05.065 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-10-06T05:07:05.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.054+0000 7f96bc396700 1 -- 172.21.15.98:0/3634738265 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f9698066f30 msgr2=0x7f96980693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:05.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.054+0000 7f96bc396700 1 --2- 172.21.15.98:0/3634738265 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f9698066f30 0x7f96980693e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f96b0009880 tx=0x7f96b0018000 comp rx=0 tx=0).stop 2023-10-06T05:07:05.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.054+0000 7f96bc396700 1 -- 172.21.15.98:0/3634738265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96b40b49a0 msgr2=0x7f96b4069250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:05.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.054+0000 7f96bc396700 1 --2- 172.21.15.98:0/3634738265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96b40b49a0 0x7f96b4069250 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f96a400d810 tx=0x7f96a4008040 comp rx=0 tx=0).stop 2023-10-06T05:07:05.067 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.054+0000 7f96bc396700 1 -- 172.21.15.98:0/3634738265 shutdown_connections 2023-10-06T05:07:05.067 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.054+0000 7f96bc396700 1 --2- 172.21.15.98:0/3634738265 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f96b4159560 0x7f96b409ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:05.067 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.054+0000 7f96bc396700 1 --2- 172.21.15.98:0/3634738265 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f96b40b0760 0x7f96b409d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:05.067 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.054+0000 7f96bc396700 1 --2- 172.21.15.98:0/3634738265 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f9698066f30 0x7f96980693e0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:05.067 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.054+0000 7f96bc396700 1 --2- 172.21.15.98:0/3634738265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96b40b49a0 0x7f96b4069250 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:05.067 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.054+0000 7f96bc396700 1 -- 172.21.15.98:0/3634738265 >> 172.21.15.98:0/3634738265 conn(0x7f96b40997f0 msgr2=0x7f96b40ccb50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:05.067 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.054+0000 7f96bc396700 1 -- 172.21.15.98:0/3634738265 shutdown_connections 2023-10-06T05:07:05.067 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.054+0000 7f96bc396700 1 -- 172.21.15.98:0/3634738265 wait complete. 2023-10-06T05:07:05.068 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-06T05:07:05.182 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.180+0000 7f4661344700 1 Processor -- start 2023-10-06T05:07:05.182 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.180+0000 7f4661344700 1 -- start start 2023-10-06T05:07:05.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.181+0000 7f4661344700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f465c0cb5f0 0x7f465c0691a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:05.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.181+0000 7f4661344700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f465c068d10 con 0x7f465c0696e0 2023-10-06T05:07:05.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.181+0000 7f4661344700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f465c15a670 con 0x7f465c15a1e0 2023-10-06T05:07:05.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.181+0000 7f4661344700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f465c154020 con 0x7f465c0cb5f0 2023-10-06T05:07:05.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.181+0000 7f465affd700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f465c0cb5f0 0x7f465c0691a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:05.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.181+0000 7f465a7fc700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f465c0696e0 0x7f465c159ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:57040/0 (socket says 172.21.15.98:57040) 2023-10-06T05:07:05.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.181+0000 7f465a7fc700 1 -- 172.21.15.98:0/780475256 learned_addr learned my addr 172.21.15.98:0/780475256 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:07:05.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.181+0000 7f465affd700 1 -- 172.21.15.98:0/780475256 >> v1:172.21.15.174:6789/0 conn(0x7f465c15a1e0 legacy=0x7f465c0ca670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:05.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.181+0000 7f465affd700 1 -- 172.21.15.98:0/780475256 >> v1:172.21.15.98:6789/0 conn(0x7f465c0696e0 legacy=0x7f465c159ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:05.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.181+0000 7f465affd700 1 -- 172.21.15.98:0/780475256 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f465c15a950 con 0x7f465c0cb5f0 2023-10-06T05:07:05.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.181+0000 7f4659ffb700 1 -- 172.21.15.98:0/780475256 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 194931663 0 0) 0x7f465c068d10 con 0x7f465c0696e0 2023-10-06T05:07:05.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.181+0000 7f465affd700 1 --2- 172.21.15.98:0/780475256 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f465c0cb5f0 0x7f465c0691a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4654009e60 tx=0x7f4654009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=804ed781ef13df16 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:05.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.181+0000 7f4659ffb700 1 -- 172.21.15.98:0/780475256 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f465400e070 con 0x7f465c0cb5f0 2023-10-06T05:07:05.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.181+0000 7f4659ffb700 1 -- 172.21.15.98:0/780475256 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f465400bc30 con 0x7f465c0cb5f0 2023-10-06T05:07:05.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.181+0000 7f4659ffb700 1 -- 172.21.15.98:0/780475256 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4654012430 con 0x7f465c0cb5f0 2023-10-06T05:07:05.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.182+0000 7f4661344700 1 -- 172.21.15.98:0/780475256 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f465c0cb5f0 msgr2=0x7f465c0691a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:05.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.182+0000 7f4661344700 1 --2- 172.21.15.98:0/780475256 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f465c0cb5f0 0x7f465c0691a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4654009e60 tx=0x7f4654009780 comp rx=0 tx=0).stop 2023-10-06T05:07:05.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.182+0000 7f4661344700 1 -- 172.21.15.98:0/780475256 shutdown_connections 2023-10-06T05:07:05.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.182+0000 7f4661344700 1 --2- 172.21.15.98:0/780475256 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f465c0cb5f0 0x7f465c0691a0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:05.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.182+0000 7f4661344700 1 -- 172.21.15.98:0/780475256 >> 172.21.15.98:0/780475256 conn(0x7f465c0707d0 msgr2=0x7f465c070bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:05.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.182+0000 7f4661344700 1 -- 172.21.15.98:0/780475256 shutdown_connections 2023-10-06T05:07:05.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.182+0000 7f4661344700 1 -- 172.21.15.98:0/780475256 wait complete. 2023-10-06T05:07:05.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.183+0000 7f4661344700 1 Processor -- start 2023-10-06T05:07:05.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.183+0000 7f4661344700 1 -- start start 2023-10-06T05:07:05.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.184+0000 7f4661344700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f465c0696e0 0x7f465c152a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:05.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.184+0000 7f4661344700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f465c0cb5f0 0x7f465c0b1400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:05.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.184+0000 7f4661344700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f465c15a1e0 0x7f465c0b1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:05.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.184+0000 7f4661344700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f465c0c2d00 con 0x7f465c0696e0 2023-10-06T05:07:05.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.184+0000 7f4661344700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f465c0c7860 con 0x7f465c15a1e0 2023-10-06T05:07:05.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.184+0000 7f4661344700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f465c0c7b40 con 0x7f465c0cb5f0 2023-10-06T05:07:05.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.184+0000 7f465affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f465c0696e0 0x7f465c152a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:05.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.184+0000 7f465affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f465c0696e0 0x7f465c152a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47872/0 (socket says 172.21.15.98:47872) 2023-10-06T05:07:05.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.184+0000 7f465affd700 1 -- 172.21.15.98:0/971224932 learned_addr learned my addr 172.21.15.98:0/971224932 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:05.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.184+0000 7f465b7fe700 1 --2- 172.21.15.98:0/971224932 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f465c15a1e0 0x7f465c0b1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:05.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.184+0000 7f465affd700 1 -- 172.21.15.98:0/971224932 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f465c0cb5f0 msgr2=0x7f465c0b1400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:05.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.184+0000 7f465a7fc700 1 --2- 172.21.15.98:0/971224932 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f465c0cb5f0 0x7f465c0b1400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:05.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.184+0000 7f465affd700 1 --2- 172.21.15.98:0/971224932 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f465c0cb5f0 0x7f465c0b1400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:05.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.184+0000 7f465affd700 1 -- 172.21.15.98:0/971224932 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f465c15a1e0 msgr2=0x7f465c0b1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:05.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.184+0000 7f465affd700 1 --2- 172.21.15.98:0/971224932 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f465c15a1e0 0x7f465c0b1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:05.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.184+0000 7f465affd700 1 -- 172.21.15.98:0/971224932 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4654009a20 con 0x7f465c0696e0 2023-10-06T05:07:05.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.184+0000 7f465b7fe700 1 --2- 172.21.15.98:0/971224932 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f465c15a1e0 0x7f465c0b1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:07:05.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.184+0000 7f465a7fc700 1 --2- 172.21.15.98:0/971224932 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f465c0cb5f0 0x7f465c0b1400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:07:05.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.184+0000 7f465affd700 1 --2- 172.21.15.98:0/971224932 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f465c0696e0 0x7f465c152a50 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f46540099f0 tx=0x7f4654019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:05.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.184+0000 7f4652ffd700 1 -- 172.21.15.98:0/971224932 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f465401b630 con 0x7f465c0696e0 2023-10-06T05:07:05.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.185+0000 7f4652ffd700 1 -- 172.21.15.98:0/971224932 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f465401b7d0 con 0x7f465c0696e0 2023-10-06T05:07:05.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.185+0000 7f4652ffd700 1 -- 172.21.15.98:0/971224932 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f465401bb90 con 0x7f465c0696e0 2023-10-06T05:07:05.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.185+0000 7f4661344700 1 -- 172.21.15.98:0/971224932 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f465c0c7e20 con 0x7f465c0696e0 2023-10-06T05:07:05.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.185+0000 7f4661344700 1 -- 172.21.15.98:0/971224932 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f465c152f90 con 0x7f465c0696e0 2023-10-06T05:07:05.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.186+0000 7f4661344700 1 -- 172.21.15.98:0/971224932 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f465c064a90 con 0x7f465c0696e0 2023-10-06T05:07:05.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.186+0000 7f4652ffd700 1 -- 172.21.15.98:0/971224932 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f465401c400 con 0x7f465c0696e0 2023-10-06T05:07:05.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.186+0000 7f4652ffd700 1 --2- 172.21.15.98:0/971224932 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f463c066fb0 0x7f463c069460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:05.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.186+0000 7f4652ffd700 1 -- 172.21.15.98:0/971224932 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f4654086790 con 0x7f465c0696e0 2023-10-06T05:07:05.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.186+0000 7f465a7fc700 1 --2- 172.21.15.98:0/971224932 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f463c066fb0 0x7f463c069460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:05.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.187+0000 7f465a7fc700 1 --2- 172.21.15.98:0/971224932 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f463c066fb0 0x7f463c069460 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f465c0cd050 tx=0x7f4644009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:05.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.191+0000 7f4652ffd700 1 -- 172.21.15.98:0/971224932 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f46540535c0 con 0x7f465c0696e0 2023-10-06T05:07:05.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.416+0000 7f4661344700 1 -- 172.21.15.98:0/971224932 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f465c04bb40 con 0x7f465c0696e0 2023-10-06T05:07:05.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.416+0000 7f4652ffd700 1 -- 172.21.15.98:0/971224932 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v6) v1 ==== 53+0+88 (secure 0 0 0) 0x7f4654056be0 con 0x7f465c0696e0 2023-10-06T05:07:05.418 INFO:teuthology.orchestra.run.smithi098.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-10-06T05:07:05.420 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.418+0000 7f4661344700 1 -- 172.21.15.98:0/971224932 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f463c066fb0 msgr2=0x7f463c069460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:05.420 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.418+0000 7f4661344700 1 --2- 172.21.15.98:0/971224932 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f463c066fb0 0x7f463c069460 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f465c0cd050 tx=0x7f4644009040 comp rx=0 tx=0).stop 2023-10-06T05:07:05.420 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.418+0000 7f4661344700 1 -- 172.21.15.98:0/971224932 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f465c0696e0 msgr2=0x7f465c152a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:05.420 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.418+0000 7f4661344700 1 --2- 172.21.15.98:0/971224932 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f465c0696e0 0x7f465c152a50 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f46540099f0 tx=0x7f4654019000 comp rx=0 tx=0).stop 2023-10-06T05:07:05.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.418+0000 7f4661344700 1 -- 172.21.15.98:0/971224932 shutdown_connections 2023-10-06T05:07:05.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.418+0000 7f4661344700 1 --2- 172.21.15.98:0/971224932 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f465c15a1e0 0x7f465c0b1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:05.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.418+0000 7f4661344700 1 --2- 172.21.15.98:0/971224932 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f463c066fb0 0x7f463c069460 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:05.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.418+0000 7f4661344700 1 --2- 172.21.15.98:0/971224932 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f465c0696e0 0x7f465c152a50 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:05.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.418+0000 7f4661344700 1 --2- 172.21.15.98:0/971224932 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f465c0cb5f0 0x7f465c0b1400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:05.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.418+0000 7f4661344700 1 -- 172.21.15.98:0/971224932 >> 172.21.15.98:0/971224932 conn(0x7f465c0707d0 msgr2=0x7f465c0cbe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:05.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.418+0000 7f4661344700 1 -- 172.21.15.98:0/971224932 shutdown_connections 2023-10-06T05:07:05.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:05.418+0000 7f4661344700 1 -- 172.21.15.98:0/971224932 wait complete. 2023-10-06T05:07:05.432 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-06T05:07:05.433 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-06T05:07:05.433 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi098.front.sepia.ceph.com 2023-10-06T05:07:05.433 INFO:tasks.cephfs.mount:self.client.name = client.admin 2023-10-06T05:07:05.433 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2023-10-06T05:07:05.433 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-06T05:07:05.433 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-06T05:07:05.433 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-06T05:07:05.433 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2023-10-06T05:07:05.434 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:07:05.434 DEBUG:teuthology.orchestra.run.smithi098:> ip addr 2023-10-06T05:07:05.449 INFO:teuthology.orchestra.run.smithi098.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-06T05:07:05.449 INFO:teuthology.orchestra.run.smithi098.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-06T05:07:05.449 INFO:teuthology.orchestra.run.smithi098.stdout: inet 127.0.0.1/8 scope host lo 2023-10-06T05:07:05.450 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-10-06T05:07:05.450 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 ::1/128 scope host 2023-10-06T05:07:05.450 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-10-06T05:07:05.450 INFO:teuthology.orchestra.run.smithi098.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-06T05:07:05.450 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:d8:d9:b8 brd ff:ff:ff:ff:ff:ff 2023-10-06T05:07:05.450 INFO:teuthology.orchestra.run.smithi098.stdout: altname enp7s0f0 2023-10-06T05:07:05.450 INFO:teuthology.orchestra.run.smithi098.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-06T05:07:05.450 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:d8:d9:b9 brd ff:ff:ff:ff:ff:ff 2023-10-06T05:07:05.450 INFO:teuthology.orchestra.run.smithi098.stdout: altname enp7s0f1 2023-10-06T05:07:05.450 INFO:teuthology.orchestra.run.smithi098.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-06T05:07:05.451 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:8f:d2:6a brd ff:ff:ff:ff:ff:ff 2023-10-06T05:07:05.451 INFO:teuthology.orchestra.run.smithi098.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-06T05:07:05.451 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:8f:d2:6b brd ff:ff:ff:ff:ff:ff 2023-10-06T05:07:05.451 INFO:teuthology.orchestra.run.smithi098.stdout: inet 172.21.15.98/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-10-06T05:07:05.451 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft 42500sec preferred_lft 42500sec 2023-10-06T05:07:05.451 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::ec4:7aff:fe8f:d26b/64 scope link 2023-10-06T05:07:05.451 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-10-06T05:07:05.452 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-10-06T05:07:05.452 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:07:05.452 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec bash -c ' 2023-10-06T05:07:05.452 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2023-10-06T05:07:05.452 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link add name ceph-brx type bridge 2023-10-06T05:07:05.452 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip addr flush dev ceph-brx 2023-10-06T05:07:05.452 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set ceph-brx up 2023-10-06T05:07:05.452 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-10-06T05:07:05.453 DEBUG:teuthology.orchestra.run.smithi098:> ') 2023-10-06T05:07:05.616 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:07:05.616 DEBUG:teuthology.orchestra.run.smithi098:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-06T05:07:05.658 INFO:teuthology.orchestra.run.smithi098.stdout:1 2023-10-06T05:07:05.661 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:07:05.662 DEBUG:teuthology.orchestra.run.smithi098:> route 2023-10-06T05:07:05.739 INFO:teuthology.orchestra.run.smithi098.stdout:Kernel IP routing table 2023-10-06T05:07:05.739 INFO:teuthology.orchestra.run.smithi098.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-06T05:07:05.739 INFO:teuthology.orchestra.run.smithi098.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-10-06T05:07:05.739 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-10-06T05:07:05.739 INFO:teuthology.orchestra.run.smithi098.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-06T05:07:05.740 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:07:05.740 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec bash -c ' 2023-10-06T05:07:05.740 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2023-10-06T05:07:05.740 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-10-06T05:07:05.740 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-10-06T05:07:05.740 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-10-06T05:07:05.740 DEBUG:teuthology.orchestra.run.smithi098:> ') 2023-10-06T05:07:05.997 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:07:05.997 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list 2023-10-06T05:07:06.012 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:07:06.013 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list-id 2023-10-06T05:07:06.067 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:07:06.068 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec bash -c ' 2023-10-06T05:07:06.068 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2023-10-06T05:07:06.068 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-10-06T05:07:06.068 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2023-10-06T05:07:06.068 DEBUG:teuthology.orchestra.run.smithi098:> ') 2023-10-06T05:07:06.156 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2023-10-06T05:07:06.157 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:07:06.157 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec bash -c ' 2023-10-06T05:07:06.157 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2023-10-06T05:07:06.157 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2023-10-06T05:07:06.157 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-10-06T05:07:06.157 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2023-10-06T05:07:06.157 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2023-10-06T05:07:06.157 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2023-10-06T05:07:06.157 DEBUG:teuthology.orchestra.run.smithi098:> ') 2023-10-06T05:07:06.358 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:07:06.359 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec bash -c ' 2023-10-06T05:07:06.359 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2023-10-06T05:07:06.359 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set brx.0 up 2023-10-06T05:07:06.359 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set dev brx.0 master ceph-brx 2023-10-06T05:07:06.359 DEBUG:teuthology.orchestra.run.smithi098:> ') 2023-10-06T05:07:06.451 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2023-10-06T05:07:06.451 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-06T05:07:06.451 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2023-10-06T05:07:06.465 INFO:teuthology.orchestra.run.smithi098.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2023-10-06T05:07:06.466 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-06T05:07:06.466 DEBUG:teuthology.orchestra.run.smithi098:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2023-10-06T05:07:06.520 DEBUG:teuthology.orchestra.run.smithi098:> sudo modprobe fuse 2023-10-06T05:07:06.584 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-10-06T05:07:06.585 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-10-06T05:07:06.648 INFO:teuthology.orchestra.run.smithi098.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-10-06T05:07:06.649 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-10-06T05:07:06.650 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T05:07:06.650 DEBUG:teuthology.orchestra.run.smithi098:> ls /sys/fs/fuse/connections 2023-10-06T05:07:06.705 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-10-06T05:07:06.705 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2023-10-06T05:07:06.749 DEBUG:teuthology.orchestra.run.smithi098:> sudo modprobe fuse 2023-10-06T05:07:06.773 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-10-06T05:07:06.773 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-10-06T05:07:06.837 INFO:teuthology.orchestra.run.smithi098.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-10-06T05:07:06.838 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-10-06T05:07:06.838 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T05:07:06.838 DEBUG:teuthology.orchestra.run.smithi098:> ls /sys/fs/fuse/connections 2023-10-06T05:07:06.842 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi098.stderr:2023-10-06T05:07:06.840+0000 7f954422c540 -1 init, newargv = 0x563d7a46a6b0 newargc=15 2023-10-06T05:07:06.842 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi098.stderr:ceph-fuse[98074]: starting ceph client 2023-10-06T05:07:06.848 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi098.stderr:ceph-fuse[98074]: starting fuse 2023-10-06T05:07:06.857 INFO:teuthology.orchestra.run.smithi098.stdout:46 2023-10-06T05:07:06.857 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [46] 2023-10-06T05:07:06.858 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-10-06T05:07:06.858 DEBUG:teuthology.orchestra.run.smithi098:> import glob 2023-10-06T05:07:06.858 DEBUG:teuthology.orchestra.run.smithi098:> import re 2023-10-06T05:07:06.858 DEBUG:teuthology.orchestra.run.smithi098:> import os 2023-10-06T05:07:06.858 DEBUG:teuthology.orchestra.run.smithi098:> import subprocess 2023-10-06T05:07:06.858 DEBUG:teuthology.orchestra.run.smithi098:> 2023-10-06T05:07:06.858 DEBUG:teuthology.orchestra.run.smithi098:> def _find_admin_socket(client_name): 2023-10-06T05:07:06.858 DEBUG:teuthology.orchestra.run.smithi098:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2023-10-06T05:07:06.858 DEBUG:teuthology.orchestra.run.smithi098:> files = glob.glob(asok_path) 2023-10-06T05:07:06.858 DEBUG:teuthology.orchestra.run.smithi098:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2023-10-06T05:07:06.858 DEBUG:teuthology.orchestra.run.smithi098:> 2023-10-06T05:07:06.858 DEBUG:teuthology.orchestra.run.smithi098:> # Given a non-glob path, it better be there 2023-10-06T05:07:06.859 DEBUG:teuthology.orchestra.run.smithi098:> if "*" not in asok_path: 2023-10-06T05:07:06.859 DEBUG:teuthology.orchestra.run.smithi098:> assert(len(files) == 1) 2023-10-06T05:07:06.859 DEBUG:teuthology.orchestra.run.smithi098:> return files[0] 2023-10-06T05:07:06.859 DEBUG:teuthology.orchestra.run.smithi098:> 2023-10-06T05:07:06.859 DEBUG:teuthology.orchestra.run.smithi098:> for f in files: 2023-10-06T05:07:06.859 DEBUG:teuthology.orchestra.run.smithi098:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-10-06T05:07:06.859 DEBUG:teuthology.orchestra.run.smithi098:> if os.path.exists("/proc/{0}".format(pid)): 2023-10-06T05:07:06.859 DEBUG:teuthology.orchestra.run.smithi098:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-10-06T05:07:06.859 DEBUG:teuthology.orchestra.run.smithi098:> contents = proc_f.read() 2023-10-06T05:07:06.859 DEBUG:teuthology.orchestra.run.smithi098:> if mountpoint in contents: 2023-10-06T05:07:06.859 DEBUG:teuthology.orchestra.run.smithi098:> return f 2023-10-06T05:07:06.859 DEBUG:teuthology.orchestra.run.smithi098:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-10-06T05:07:06.859 DEBUG:teuthology.orchestra.run.smithi098:> 2023-10-06T05:07:06.860 DEBUG:teuthology.orchestra.run.smithi098:> print(_find_admin_socket("client.admin")) 2023-10-06T05:07:06.860 DEBUG:teuthology.orchestra.run.smithi098:> ' 2023-10-06T05:07:06.978 INFO:teuthology.orchestra.run.smithi098.stdout:/var/run/ceph/ceph-client.admin.98074.asok 2023-10-06T05:07:07.161 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.98074.asok 2023-10-06T05:07:07.161 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T05:07:07.161 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.98074.asok status 2023-10-06T05:07:07.311 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-10-06T05:07:07.311 INFO:teuthology.orchestra.run.smithi098.stdout: "metadata": { 2023-10-06T05:07:07.311 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_sha1": "68e53a4a7d4ae06d4fc1a0a7cfef840d1d942564", 2023-10-06T05:07:07.312 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_version": "ceph version 17.2.6-1274-g68e53a4a (68e53a4a7d4ae06d4fc1a0a7cfef840d1d942564) quincy (stable)", 2023-10-06T05:07:07.312 INFO:teuthology.orchestra.run.smithi098.stdout: "entity_id": "admin", 2023-10-06T05:07:07.312 INFO:teuthology.orchestra.run.smithi098.stdout: "hostname": "smithi098", 2023-10-06T05:07:07.312 INFO:teuthology.orchestra.run.smithi098.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2023-10-06T05:07:07.312 INFO:teuthology.orchestra.run.smithi098.stdout: "pid": "98074", 2023-10-06T05:07:07.312 INFO:teuthology.orchestra.run.smithi098.stdout: "root": "/" 2023-10-06T05:07:07.312 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-10-06T05:07:07.312 INFO:teuthology.orchestra.run.smithi098.stdout: "dentry_count": 0, 2023-10-06T05:07:07.313 INFO:teuthology.orchestra.run.smithi098.stdout: "dentry_pinned_count": 0, 2023-10-06T05:07:07.313 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 4479, 2023-10-06T05:07:07.313 INFO:teuthology.orchestra.run.smithi098.stdout: "inst": { 2023-10-06T05:07:07.313 INFO:teuthology.orchestra.run.smithi098.stdout: "name": { 2023-10-06T05:07:07.313 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "client", 2023-10-06T05:07:07.313 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 4479 2023-10-06T05:07:07.313 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-10-06T05:07:07.313 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": { 2023-10-06T05:07:07.313 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-10-06T05:07:07.314 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "192.168.0.1:0", 2023-10-06T05:07:07.314 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 2496927800 2023-10-06T05:07:07.314 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-10-06T05:07:07.314 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-10-06T05:07:07.314 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": { 2023-10-06T05:07:07.314 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2023-10-06T05:07:07.314 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "192.168.0.1:0", 2023-10-06T05:07:07.314 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 2496927800 2023-10-06T05:07:07.314 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-10-06T05:07:07.315 INFO:teuthology.orchestra.run.smithi098.stdout: "inst_str": "client.4479 192.168.0.1:0/2496927800", 2023-10-06T05:07:07.315 INFO:teuthology.orchestra.run.smithi098.stdout: "addr_str": "192.168.0.1:0/2496927800", 2023-10-06T05:07:07.315 INFO:teuthology.orchestra.run.smithi098.stdout: "inode_count": 1, 2023-10-06T05:07:07.315 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_epoch": 6, 2023-10-06T05:07:07.315 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_epoch": 29, 2023-10-06T05:07:07.315 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_epoch_barrier": 0, 2023-10-06T05:07:07.315 INFO:teuthology.orchestra.run.smithi098.stdout: "blocklisted": false, 2023-10-06T05:07:07.315 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs" 2023-10-06T05:07:07.315 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-10-06T05:07:07.322 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T05:07:07.322 DEBUG:teuthology.orchestra.run.smithi098:> stat --file-system '--printf=%T 2023-10-06T05:07:07.322 DEBUG:teuthology.orchestra.run.smithi098:> ' -- /home/ubuntu/cephtest/mnt.admin 2023-10-06T05:07:07.337 INFO:teuthology.orchestra.run.smithi098.stdout:fuseblk 2023-10-06T05:07:07.337 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2023-10-06T05:07:07.338 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T05:07:07.338 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2023-10-06T05:07:07.403 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T05:07:07.403 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.admin && exec bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2023-10-06T05:07:07.504 INFO:teuthology.orchestra.run.smithi098.stdout:# file: . 2023-10-06T05:07:07.504 INFO:teuthology.orchestra.run.smithi098.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2023-10-06T05:07:07.504 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:07:07.504 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi098.front.sepia.ceph.com... 2023-10-06T05:07:07.505 INFO:teuthology.orchestra.run:Running command with timeout 1800 2023-10-06T05:07:07.505 DEBUG:teuthology.orchestra.run.smithi098:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2023-10-06T05:07:07.544 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi098.stderr:ceph-fuse[98074]: fuse finished with error 0 and tester_r 0 2023-10-06T05:07:07.545 INFO:teuthology.orchestra.run:waiting for 900 2023-10-06T05:07:07.634 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi098.front.sepia.ceph.com 2023-10-06T05:07:07.634 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:07:07.634 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2023-10-06T05:07:07.649 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T05:07:07.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:07.803+0000 7fc32b59e700 1 Processor -- start 2023-10-06T05:07:07.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:07.803+0000 7fc32b59e700 1 -- start start 2023-10-06T05:07:07.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:07.804+0000 7fc32b59e700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc32c0cbc30 0x7fc32c0cc080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:07.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:07.804+0000 7fc32b59e700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fc32c0b2100 0x7fc32c0b2520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:07.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:07.804+0000 7fc32b59e700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc32c0cb7a0 con 0x7fc32c159440 2023-10-06T05:07:07.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:07.804+0000 7fc32b59e700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc32c0b0860 con 0x7fc32c0cbc30 2023-10-06T05:07:07.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:07.804+0000 7fc32b59e700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc32c0cb150 con 0x7fc32c0b2100 2023-10-06T05:07:07.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:07.804+0000 7fc329d9b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc32c0cbc30 0x7fc32c0cc080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:07.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:07.804+0000 7fc329d9b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc32c0cbc30 0x7fc32c0cc080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47884/0 (socket says 172.21.15.98:47884) 2023-10-06T05:07:07.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:07.804+0000 7fc329d9b700 1 -- 172.21.15.98:0/617295810 learned_addr learned my addr 172.21.15.98:0/617295810 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:07.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:07.804+0000 7fc32ad9d700 1 --2- 172.21.15.98:0/617295810 >> v2:172.21.15.174:3300/0 conn(0x7fc32c0b2100 0x7fc32c0b2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:07.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:07.804+0000 7fc329d9b700 1 -- 172.21.15.98:0/617295810 >> v2:172.21.15.174:3300/0 conn(0x7fc32c0b2100 msgr2=0x7fc32c0b2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:07.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:07.804+0000 7fc329d9b700 1 --2- 172.21.15.98:0/617295810 >> v2:172.21.15.174:3300/0 conn(0x7fc32c0b2100 0x7fc32c0b2520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:07.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:07.804+0000 7fc329d9b700 1 -- 172.21.15.98:0/617295810 >> v1:172.21.15.174:6789/0 conn(0x7fc32c159440 legacy=0x7fc32c0b3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:07.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:07.804+0000 7fc329d9b700 1 -- 172.21.15.98:0/617295810 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc32c0cb430 con 0x7fc32c0cbc30 2023-10-06T05:07:07.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:07.804+0000 7fc32ad9d700 1 --2- 172.21.15.98:0/617295810 >> v2:172.21.15.174:3300/0 conn(0x7fc32c0b2100 0x7fc32c0b2520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:07:07.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:07.805+0000 7fc329d9b700 1 --2- 172.21.15.98:0/617295810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc32c0cbc30 0x7fc32c0cc080 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc32c0ca7f0 tx=0x7fc320009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f461451a6a9a080 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:07.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:07.805+0000 7fc32959a700 1 -- 172.21.15.98:0/617295810 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc32000b820 con 0x7fc32c0cbc30 2023-10-06T05:07:07.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:07.805+0000 7fc32959a700 1 -- 172.21.15.98:0/617295810 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc32000b980 con 0x7fc32c0cbc30 2023-10-06T05:07:07.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:07.805+0000 7fc32959a700 1 -- 172.21.15.98:0/617295810 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc32000bd20 con 0x7fc32c0cbc30 2023-10-06T05:07:07.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:07.805+0000 7fc32b59e700 1 -- 172.21.15.98:0/617295810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc32c0cbc30 msgr2=0x7fc32c0cc080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:07.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:07.805+0000 7fc32b59e700 1 --2- 172.21.15.98:0/617295810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc32c0cbc30 0x7fc32c0cc080 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc32c0ca7f0 tx=0x7fc320009250 comp rx=0 tx=0).stop 2023-10-06T05:07:07.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:07.805+0000 7fc32b59e700 1 -- 172.21.15.98:0/617295810 shutdown_connections 2023-10-06T05:07:07.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:07.805+0000 7fc32b59e700 1 --2- 172.21.15.98:0/617295810 >> v2:172.21.15.174:3300/0 conn(0x7fc32c0b2100 0x7fc32c0b2520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:07.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:07.805+0000 7fc32b59e700 1 --2- 172.21.15.98:0/617295810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc32c0cbc30 0x7fc32c0cc080 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:07.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:07.805+0000 7fc32b59e700 1 -- 172.21.15.98:0/617295810 >> 172.21.15.98:0/617295810 conn(0x7fc32c06ece0 msgr2=0x7fc32c06f0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:07.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:07.805+0000 7fc32b59e700 1 -- 172.21.15.98:0/617295810 shutdown_connections 2023-10-06T05:07:07.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:07.805+0000 7fc32b59e700 1 -- 172.21.15.98:0/617295810 wait complete. 2023-10-06T05:07:07.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:07.806+0000 7fc32b59e700 1 Processor -- start 2023-10-06T05:07:07.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:07.806+0000 7fc32b59e700 1 -- start start 2023-10-06T05:07:07.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:07.807+0000 7fc32b59e700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc32c0b2100 0x7fc32c153f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:07.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:07.807+0000 7fc32b59e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc32c0cbc30 0x7fc32c0c4460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:07.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:07.807+0000 7fc32b59e700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc32c159440 0x7fc32c169400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:07.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:07.807+0000 7fc32b59e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc32c16a400 con 0x7fc32c0cbc30 2023-10-06T05:07:07.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:07.807+0000 7fc32b59e700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc32c0c3fd0 con 0x7fc32c0b2100 2023-10-06T05:07:07.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:07.807+0000 7fc32b59e700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc32c169e40 con 0x7fc32c159440 2023-10-06T05:07:07.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:07.807+0000 7fc329d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc32c0cbc30 0x7fc32c0c4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:07.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:07.807+0000 7fc329d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc32c0cbc30 0x7fc32c0c4460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47894/0 (socket says 172.21.15.98:47894) 2023-10-06T05:07:07.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:07.807+0000 7fc329d9b700 1 -- 172.21.15.98:0/1822442232 learned_addr learned my addr 172.21.15.98:0/1822442232 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:07.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:07.807+0000 7fc32a59c700 1 --2- 172.21.15.98:0/1822442232 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc32c0b2100 0x7fc32c153f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:07.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:07.807+0000 7fc32ad9d700 1 --2- 172.21.15.98:0/1822442232 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc32c159440 0x7fc32c169400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:07.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:07.808+0000 7fc329d9b700 1 -- 172.21.15.98:0/1822442232 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc32c159440 msgr2=0x7fc32c169400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:07.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:07.808+0000 7fc329d9b700 1 --2- 172.21.15.98:0/1822442232 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc32c159440 0x7fc32c169400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:07.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:07.808+0000 7fc329d9b700 1 -- 172.21.15.98:0/1822442232 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc32c0b2100 msgr2=0x7fc32c153f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:07.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:07.808+0000 7fc329d9b700 1 --2- 172.21.15.98:0/1822442232 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc32c0b2100 0x7fc32c153f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:07.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:07.808+0000 7fc329d9b700 1 -- 172.21.15.98:0/1822442232 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3200094f0 con 0x7fc32c0cbc30 2023-10-06T05:07:07.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:07.808+0000 7fc329d9b700 1 --2- 172.21.15.98:0/1822442232 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc32c0cbc30 0x7fc32c0c4460 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fc3200030c0 tx=0x7fc320016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:07.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:07.808+0000 7fc31a7fc700 1 -- 172.21.15.98:0/1822442232 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc320011d00 con 0x7fc32c0cbc30 2023-10-06T05:07:07.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:07.808+0000 7fc31a7fc700 1 -- 172.21.15.98:0/1822442232 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc320020070 con 0x7fc32c0cbc30 2023-10-06T05:07:07.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:07.808+0000 7fc32b59e700 1 -- 172.21.15.98:0/1822442232 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc32c16a120 con 0x7fc32c0cbc30 2023-10-06T05:07:07.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:07.808+0000 7fc31a7fc700 1 -- 172.21.15.98:0/1822442232 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc32001b6a0 con 0x7fc32c0cbc30 2023-10-06T05:07:07.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:07.808+0000 7fc32b59e700 1 -- 172.21.15.98:0/1822442232 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc32c0c49a0 con 0x7fc32c0cbc30 2023-10-06T05:07:07.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:07.809+0000 7fc32b59e700 1 -- 172.21.15.98:0/1822442232 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc32c0c3ac0 con 0x7fc32c0cbc30 2023-10-06T05:07:07.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:07.809+0000 7fc31a7fc700 1 -- 172.21.15.98:0/1822442232 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fc320022030 con 0x7fc32c0cbc30 2023-10-06T05:07:07.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:07.809+0000 7fc31a7fc700 1 --2- 172.21.15.98:0/1822442232 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fc30c066f30 0x7fc30c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:07.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:07.810+0000 7fc31a7fc700 1 -- 172.21.15.98:0/1822442232 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fc320086790 con 0x7fc32c0cbc30 2023-10-06T05:07:07.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:07.810+0000 7fc32a59c700 1 --2- 172.21.15.98:0/1822442232 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fc30c066f30 0x7fc30c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:07.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:07.810+0000 7fc32a59c700 1 --2- 172.21.15.98:0/1822442232 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fc30c066f30 0x7fc30c0693e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc31c007560 tx=0x7fc31c009760 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:07.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:07.814+0000 7fc31a7fc700 1 -- 172.21.15.98:0/1822442232 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc32c0c3ac0 con 0x7fc32c0cbc30 2023-10-06T05:07:08.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:07.998+0000 7fc32b59e700 1 -- 172.21.15.98:0/1822442232 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc32c0c3ac0 con 0x7fc32c0cbc30 2023-10-06T05:07:08.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:07.999+0000 7fc31a7fc700 1 -- 172.21.15.98:0/1822442232 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+18707 (secure 0 0 0) 0x7fc32c0c3ac0 con 0x7fc32c0cbc30 2023-10-06T05:07:08.001 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:07:08.001 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":29,"fsid":"f50c5df5-cad5-47a4-9bb1-8c791d4311ec","created":"2023-10-06T05:06:32.971175+0000","modified":"2023-10-06T05:07:02.131672+0000","last_up_change":"2023-10-06T05:06:44.016489+0000","last_in_change":"2023-10-06T05:06:35.544511+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-06T05:06:44.559671+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-06T05:06:51.432161+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-10-06T05:06:55.589057+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-10-06T05:06:56.041687+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-10-06T05:06:59.754099+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d9495d95-da75-4c3b-8927-e9bae6a8b093","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":90627},{"type":"v1","addr":"172.21.15.98:6801","nonce":90627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":90627},{"type":"v1","addr":"172.21.15.98:6803","nonce":90627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":90627},{"type":"v1","addr":"172.21.15.98:6807","nonce":90627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":90627},{"type":"v1","addr":"172.21.15.98:6805","nonce":90627}]},"public_addr":"172.21.15.98:6801/90627","cluster_addr":"172.21.15.98:6803/90627","heartbeat_back_addr":"172.21.15.98:6807/90627","heartbeat_front_addr":"172.21.15.98:6805/90627","state":["exists","up"]},{"osd":1,"uuid":"f25ba5be-d65e-4707-a134-154092deb754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":90628},{"type":"v1","addr":"172.21.15.98:6809","nonce":90628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":90628},{"type":"v1","addr":"172.21.15.98:6811","nonce":90628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":90628},{"type":"v1","addr":"172.21.15.98:6815","nonce":90628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":90628},{"type":"v1","addr":"172.21.15.98:6813","nonce":90628}]},"public_addr":"172.21.15.98:6809/90628","cluster_addr":"172.21.15.98:6811/90628","heartbeat_back_addr":"172.21.15.98:6815/90628","heartbeat_front_addr":"172.21.15.98:6813/90628","state":["exists","up"]},{"osd":2,"uuid":"5ab9e028-945c-43e2-8c95-3b0b4cf38e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":90662},{"type":"v1","addr":"172.21.15.98:6818","nonce":90662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":90662},{"type":"v1","addr":"172.21.15.98:6822","nonce":90662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":90662},{"type":"v1","addr":"172.21.15.98:6830","nonce":90662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":90662},{"type":"v1","addr":"172.21.15.98:6826","nonce":90662}]},"public_addr":"172.21.15.98:6818/90662","cluster_addr":"172.21.15.98:6822/90662","heartbeat_back_addr":"172.21.15.98:6830/90662","heartbeat_front_addr":"172.21.15.98:6826/90662","state":["exists","up"]},{"osd":3,"uuid":"3bb803ff-1738-46bf-be62-2267bf378822","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6817","nonce":90663},{"type":"v1","addr":"172.21.15.98:6819","nonce":90663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6821","nonce":90663},{"type":"v1","addr":"172.21.15.98:6823","nonce":90663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6829","nonce":90663},{"type":"v1","addr":"172.21.15.98:6831","nonce":90663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6825","nonce":90663},{"type":"v1","addr":"172.21.15.98:6827","nonce":90663}]},"public_addr":"172.21.15.98:6819/90663","cluster_addr":"172.21.15.98:6823/90663","heartbeat_back_addr":"172.21.15.98:6831/90663","heartbeat_front_addr":"172.21.15.98:6827/90663","state":["exists","up"]},{"osd":4,"uuid":"7d57a4c9-6462-43f6-b64a-7f9e9b4353d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":90847},{"type":"v1","addr":"172.21.15.174:6801","nonce":90847}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":90847},{"type":"v1","addr":"172.21.15.174:6803","nonce":90847}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":90847},{"type":"v1","addr":"172.21.15.174:6807","nonce":90847}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":90847},{"type":"v1","addr":"172.21.15.174:6805","nonce":90847}]},"public_addr":"172.21.15.174:6801/90847","cluster_addr":"172.21.15.174:6803/90847","heartbeat_back_addr":"172.21.15.174:6807/90847","heartbeat_front_addr":"172.21.15.174:6805/90847","state":["exists","up"]},{"osd":5,"uuid":"83287ea4-e44d-4af9-bc77-36d47c60d4a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":90848},{"type":"v1","addr":"172.21.15.174:6809","nonce":90848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":90848},{"type":"v1","addr":"172.21.15.174:6811","nonce":90848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":90848},{"type":"v1","addr":"172.21.15.174:6815","nonce":90848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":90848},{"type":"v1","addr":"172.21.15.174:6813","nonce":90848}]},"public_addr":"172.21.15.174:6809/90848","cluster_addr":"172.21.15.174:6811/90848","heartbeat_back_addr":"172.21.15.174:6815/90848","heartbeat_front_addr":"172.21.15.174:6813/90848","state":["exists","up"]},{"osd":6,"uuid":"496a6272-892e-43c4-a145-206623c22f99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":90877},{"type":"v1","addr":"172.21.15.174:6817","nonce":90877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":90877},{"type":"v1","addr":"172.21.15.174:6819","nonce":90877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":90877},{"type":"v1","addr":"172.21.15.174:6823","nonce":90877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":90877},{"type":"v1","addr":"172.21.15.174:6821","nonce":90877}]},"public_addr":"172.21.15.174:6817/90877","cluster_addr":"172.21.15.174:6819/90877","heartbeat_back_addr":"172.21.15.174:6823/90877","heartbeat_front_addr":"172.21.15.174:6821/90877","state":["exists","up"]},{"osd":7,"uuid":"49967e34-8b07-4bab-a13e-0c16d6a0f3a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":90878},{"type":"v1","addr":"172.21.15.174:6825","nonce":90878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":90878},{"type":"v1","addr":"172.21.15.174:6827","nonce":90878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":90878},{"type":"v1","addr":"172.21.15.174:6831","nonce":90878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":90878},{"type":"v1","addr":"172.21.15.174:6829","nonce":90878}]},"public_addr":"172.21.15.174:6825/90878","cluster_addr":"172.21.15.174:6827/90878","heartbeat_back_addr":"172.21.15.174:6831/90878","heartbeat_front_addr":"172.21.15.174:6829/90878","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.446479+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.514266+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.794108+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.810894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.731919+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.717756+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.737435+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.508682+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T05:07:08.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.001+0000 7fc32b59e700 1 -- 172.21.15.98:0/1822442232 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fc30c066f30 msgr2=0x7fc30c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:08.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.001+0000 7fc32b59e700 1 --2- 172.21.15.98:0/1822442232 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fc30c066f30 0x7fc30c0693e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc31c007560 tx=0x7fc31c009760 comp rx=0 tx=0).stop 2023-10-06T05:07:08.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.001+0000 7fc32b59e700 1 -- 172.21.15.98:0/1822442232 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc32c0cbc30 msgr2=0x7fc32c0c4460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:08.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.001+0000 7fc32b59e700 1 --2- 172.21.15.98:0/1822442232 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc32c0cbc30 0x7fc32c0c4460 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fc3200030c0 tx=0x7fc320016000 comp rx=0 tx=0).stop 2023-10-06T05:07:08.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.001+0000 7fc32b59e700 1 -- 172.21.15.98:0/1822442232 shutdown_connections 2023-10-06T05:07:08.005 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.001+0000 7fc32b59e700 1 --2- 172.21.15.98:0/1822442232 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc32c159440 0x7fc32c169400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:08.005 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.001+0000 7fc32b59e700 1 --2- 172.21.15.98:0/1822442232 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc32c0b2100 0x7fc32c153f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:08.005 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.001+0000 7fc32b59e700 1 --2- 172.21.15.98:0/1822442232 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fc30c066f30 0x7fc30c0693e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:08.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.001+0000 7fc32b59e700 1 --2- 172.21.15.98:0/1822442232 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc32c0cbc30 0x7fc32c0c4460 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:08.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.001+0000 7fc32b59e700 1 -- 172.21.15.98:0/1822442232 >> 172.21.15.98:0/1822442232 conn(0x7fc32c06ece0 msgr2=0x7fc32c0b4a60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:08.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.001+0000 7fc32b59e700 1 -- 172.21.15.98:0/1822442232 shutdown_connections 2023-10-06T05:07:08.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.001+0000 7fc32b59e700 1 -- 172.21.15.98:0/1822442232 wait complete. 2023-10-06T05:07:08.015 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T05:07:08.128 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.126+0000 7fca549c9700 1 Processor -- start 2023-10-06T05:07:08.128 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.126+0000 7fca549c9700 1 -- start start 2023-10-06T05:07:08.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.127+0000 7fca549c9700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fca500b49a0 0x7fca500aac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:08.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.127+0000 7fca549c9700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca500b0c60 con 0x7fca500b0760 2023-10-06T05:07:08.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.127+0000 7fca549c9700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca500b0f40 con 0x7fca50159560 2023-10-06T05:07:08.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.127+0000 7fca549c9700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca500b1220 con 0x7fca500b49a0 2023-10-06T05:07:08.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.127+0000 7fca4ed9d700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fca500b0760 0x7fca500b1bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:57044/0 (socket says 172.21.15.98:57044) 2023-10-06T05:07:08.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.127+0000 7fca4ed9d700 1 -- 172.21.15.98:0/2886103623 learned_addr learned my addr 172.21.15.98:0/2886103623 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:07:08.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.127+0000 7fca4dd9b700 1 --2- 172.21.15.98:0/2886103623 >> v2:172.21.15.174:3300/0 conn(0x7fca500b49a0 0x7fca500aac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:08.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.128+0000 7fca4d59a700 1 -- 172.21.15.98:0/2886103623 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 490679449 0 0) 0x7fca500b0c60 con 0x7fca500b0760 2023-10-06T05:07:08.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.128+0000 7fca4d59a700 1 -- 172.21.15.98:0/2886103623 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca3c003570 con 0x7fca500b0760 2023-10-06T05:07:08.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.128+0000 7fca4d59a700 1 -- 172.21.15.98:0/2886103623 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 297504741 0 0) 0x7fca500b0f40 con 0x7fca50159560 2023-10-06T05:07:08.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.128+0000 7fca4d59a700 1 -- 172.21.15.98:0/2886103623 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca500b0c60 con 0x7fca50159560 2023-10-06T05:07:08.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.128+0000 7fca4dd9b700 1 -- 172.21.15.98:0/2886103623 >> v1:172.21.15.174:6789/0 conn(0x7fca50159560 legacy=0x7fca500aa530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:08.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.128+0000 7fca4dd9b700 1 -- 172.21.15.98:0/2886103623 >> v1:172.21.15.98:6789/0 conn(0x7fca500b0760 legacy=0x7fca500b1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:08.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.128+0000 7fca4dd9b700 1 -- 172.21.15.98:0/2886103623 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca500b1500 con 0x7fca500b49a0 2023-10-06T05:07:08.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.128+0000 7fca4dd9b700 1 --2- 172.21.15.98:0/2886103623 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fca500b49a0 0x7fca500aac40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fca38009910 tx=0x7fca380091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9c6e41a0237685e6 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:08.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.128+0000 7fca4d59a700 1 -- 172.21.15.98:0/2886103623 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca38011630 con 0x7fca500b49a0 2023-10-06T05:07:08.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.128+0000 7fca4d59a700 1 -- 172.21.15.98:0/2886103623 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca38011790 con 0x7fca500b49a0 2023-10-06T05:07:08.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.128+0000 7fca4d59a700 1 -- 172.21.15.98:0/2886103623 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca38011b30 con 0x7fca500b49a0 2023-10-06T05:07:08.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.128+0000 7fca549c9700 1 -- 172.21.15.98:0/2886103623 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fca500b49a0 msgr2=0x7fca500aac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:08.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.128+0000 7fca549c9700 1 --2- 172.21.15.98:0/2886103623 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fca500b49a0 0x7fca500aac40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fca38009910 tx=0x7fca380091a0 comp rx=0 tx=0).stop 2023-10-06T05:07:08.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.129+0000 7fca549c9700 1 -- 172.21.15.98:0/2886103623 shutdown_connections 2023-10-06T05:07:08.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.129+0000 7fca549c9700 1 --2- 172.21.15.98:0/2886103623 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fca500b49a0 0x7fca500aac40 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:08.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.129+0000 7fca549c9700 1 -- 172.21.15.98:0/2886103623 >> 172.21.15.98:0/2886103623 conn(0x7fca500997f0 msgr2=0x7fca5006f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:08.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.129+0000 7fca549c9700 1 -- 172.21.15.98:0/2886103623 shutdown_connections 2023-10-06T05:07:08.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.129+0000 7fca549c9700 1 -- 172.21.15.98:0/2886103623 wait complete. 2023-10-06T05:07:08.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.130+0000 7fca549c9700 1 Processor -- start 2023-10-06T05:07:08.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.130+0000 7fca549c9700 1 -- start start 2023-10-06T05:07:08.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.130+0000 7fca549c9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca500b0760 0x7fca500ae900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:08.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.130+0000 7fca549c9700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fca500b49a0 0x7fca50159930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:08.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.130+0000 7fca549c9700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fca50159560 0x7fca50159e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:08.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.130+0000 7fca549c9700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca5015b280 con 0x7fca500b0760 2023-10-06T05:07:08.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.130+0000 7fca549c9700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca5015acc0 con 0x7fca50159560 2023-10-06T05:07:08.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.130+0000 7fca549c9700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca5015afa0 con 0x7fca500b49a0 2023-10-06T05:07:08.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.131+0000 7fca4e59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca500b0760 0x7fca500ae900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:08.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.131+0000 7fca4e59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca500b0760 0x7fca500ae900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47896/0 (socket says 172.21.15.98:47896) 2023-10-06T05:07:08.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.131+0000 7fca4e59c700 1 -- 172.21.15.98:0/4129430731 learned_addr learned my addr 172.21.15.98:0/4129430731 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:08.138 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.131+0000 7fca4ed9d700 1 --2- 172.21.15.98:0/4129430731 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fca50159560 0x7fca50159e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:08.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.131+0000 7fca4dd9b700 1 --2- 172.21.15.98:0/4129430731 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fca500b49a0 0x7fca50159930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:08.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.131+0000 7fca4e59c700 1 -- 172.21.15.98:0/4129430731 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fca500b49a0 msgr2=0x7fca50159930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:08.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.131+0000 7fca4e59c700 1 --2- 172.21.15.98:0/4129430731 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fca500b49a0 0x7fca50159930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:08.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.131+0000 7fca4e59c700 1 -- 172.21.15.98:0/4129430731 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fca50159560 msgr2=0x7fca50159e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:08.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.131+0000 7fca4e59c700 1 --2- 172.21.15.98:0/4129430731 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fca50159560 0x7fca50159e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:08.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.131+0000 7fca4e59c700 1 -- 172.21.15.98:0/4129430731 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca38009420 con 0x7fca500b0760 2023-10-06T05:07:08.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.131+0000 7fca4ed9d700 1 --2- 172.21.15.98:0/4129430731 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fca50159560 0x7fca50159e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:07:08.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.131+0000 7fca4dd9b700 1 --2- 172.21.15.98:0/4129430731 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fca500b49a0 0x7fca50159930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:07:08.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.131+0000 7fca4e59c700 1 --2- 172.21.15.98:0/4129430731 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca500b0760 0x7fca500ae900 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fca4000cd30 tx=0x7fca40002b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:08.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.131+0000 7fca467fc700 1 -- 172.21.15.98:0/4129430731 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca4000ab70 con 0x7fca500b0760 2023-10-06T05:07:08.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.131+0000 7fca467fc700 1 -- 172.21.15.98:0/4129430731 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca4000acd0 con 0x7fca500b0760 2023-10-06T05:07:08.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.131+0000 7fca549c9700 1 -- 172.21.15.98:0/4129430731 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca500ab3c0 con 0x7fca500b0760 2023-10-06T05:07:08.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.131+0000 7fca467fc700 1 -- 172.21.15.98:0/4129430731 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca40012650 con 0x7fca500b0760 2023-10-06T05:07:08.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.131+0000 7fca549c9700 1 -- 172.21.15.98:0/4129430731 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca50156ff0 con 0x7fca500b0760 2023-10-06T05:07:08.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.132+0000 7fca549c9700 1 -- 172.21.15.98:0/4129430731 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca500ab8e0 con 0x7fca500b0760 2023-10-06T05:07:08.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.132+0000 7fca467fc700 1 -- 172.21.15.98:0/4129430731 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fca40012a00 con 0x7fca500b0760 2023-10-06T05:07:08.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.133+0000 7fca467fc700 1 --2- 172.21.15.98:0/4129430731 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fca30066f30 0x7fca300693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:08.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.133+0000 7fca467fc700 1 -- 172.21.15.98:0/4129430731 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fca40085130 con 0x7fca500b0760 2023-10-06T05:07:08.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.133+0000 7fca4dd9b700 1 --2- 172.21.15.98:0/4129430731 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fca30066f30 0x7fca300693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:08.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.133+0000 7fca4dd9b700 1 --2- 172.21.15.98:0/4129430731 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fca30066f30 0x7fca300693e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fca38009910 tx=0x7fca38017000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:08.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.137+0000 7fca467fc700 1 -- 172.21.15.98:0/4129430731 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fca40051f50 con 0x7fca500b0760 2023-10-06T05:07:08.324 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.322+0000 7fca549c9700 1 -- 172.21.15.98:0/4129430731 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fca50064a90 con 0x7fca500b0760 2023-10-06T05:07:08.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.323+0000 7fca467fc700 1 -- 172.21.15.98:0/4129430731 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+18707 (secure 0 0 0) 0x7fca40012cb0 con 0x7fca500b0760 2023-10-06T05:07:08.326 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:07:08.326 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":29,"fsid":"f50c5df5-cad5-47a4-9bb1-8c791d4311ec","created":"2023-10-06T05:06:32.971175+0000","modified":"2023-10-06T05:07:02.131672+0000","last_up_change":"2023-10-06T05:06:44.016489+0000","last_in_change":"2023-10-06T05:06:35.544511+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-06T05:06:44.559671+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-06T05:06:51.432161+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-10-06T05:06:55.589057+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-10-06T05:06:56.041687+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-10-06T05:06:59.754099+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d9495d95-da75-4c3b-8927-e9bae6a8b093","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":90627},{"type":"v1","addr":"172.21.15.98:6801","nonce":90627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":90627},{"type":"v1","addr":"172.21.15.98:6803","nonce":90627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":90627},{"type":"v1","addr":"172.21.15.98:6807","nonce":90627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":90627},{"type":"v1","addr":"172.21.15.98:6805","nonce":90627}]},"public_addr":"172.21.15.98:6801/90627","cluster_addr":"172.21.15.98:6803/90627","heartbeat_back_addr":"172.21.15.98:6807/90627","heartbeat_front_addr":"172.21.15.98:6805/90627","state":["exists","up"]},{"osd":1,"uuid":"f25ba5be-d65e-4707-a134-154092deb754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":90628},{"type":"v1","addr":"172.21.15.98:6809","nonce":90628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":90628},{"type":"v1","addr":"172.21.15.98:6811","nonce":90628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":90628},{"type":"v1","addr":"172.21.15.98:6815","nonce":90628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":90628},{"type":"v1","addr":"172.21.15.98:6813","nonce":90628}]},"public_addr":"172.21.15.98:6809/90628","cluster_addr":"172.21.15.98:6811/90628","heartbeat_back_addr":"172.21.15.98:6815/90628","heartbeat_front_addr":"172.21.15.98:6813/90628","state":["exists","up"]},{"osd":2,"uuid":"5ab9e028-945c-43e2-8c95-3b0b4cf38e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":90662},{"type":"v1","addr":"172.21.15.98:6818","nonce":90662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":90662},{"type":"v1","addr":"172.21.15.98:6822","nonce":90662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":90662},{"type":"v1","addr":"172.21.15.98:6830","nonce":90662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":90662},{"type":"v1","addr":"172.21.15.98:6826","nonce":90662}]},"public_addr":"172.21.15.98:6818/90662","cluster_addr":"172.21.15.98:6822/90662","heartbeat_back_addr":"172.21.15.98:6830/90662","heartbeat_front_addr":"172.21.15.98:6826/90662","state":["exists","up"]},{"osd":3,"uuid":"3bb803ff-1738-46bf-be62-2267bf378822","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6817","nonce":90663},{"type":"v1","addr":"172.21.15.98:6819","nonce":90663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6821","nonce":90663},{"type":"v1","addr":"172.21.15.98:6823","nonce":90663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6829","nonce":90663},{"type":"v1","addr":"172.21.15.98:6831","nonce":90663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6825","nonce":90663},{"type":"v1","addr":"172.21.15.98:6827","nonce":90663}]},"public_addr":"172.21.15.98:6819/90663","cluster_addr":"172.21.15.98:6823/90663","heartbeat_back_addr":"172.21.15.98:6831/90663","heartbeat_front_addr":"172.21.15.98:6827/90663","state":["exists","up"]},{"osd":4,"uuid":"7d57a4c9-6462-43f6-b64a-7f9e9b4353d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":90847},{"type":"v1","addr":"172.21.15.174:6801","nonce":90847}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":90847},{"type":"v1","addr":"172.21.15.174:6803","nonce":90847}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":90847},{"type":"v1","addr":"172.21.15.174:6807","nonce":90847}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":90847},{"type":"v1","addr":"172.21.15.174:6805","nonce":90847}]},"public_addr":"172.21.15.174:6801/90847","cluster_addr":"172.21.15.174:6803/90847","heartbeat_back_addr":"172.21.15.174:6807/90847","heartbeat_front_addr":"172.21.15.174:6805/90847","state":["exists","up"]},{"osd":5,"uuid":"83287ea4-e44d-4af9-bc77-36d47c60d4a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":90848},{"type":"v1","addr":"172.21.15.174:6809","nonce":90848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":90848},{"type":"v1","addr":"172.21.15.174:6811","nonce":90848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":90848},{"type":"v1","addr":"172.21.15.174:6815","nonce":90848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":90848},{"type":"v1","addr":"172.21.15.174:6813","nonce":90848}]},"public_addr":"172.21.15.174:6809/90848","cluster_addr":"172.21.15.174:6811/90848","heartbeat_back_addr":"172.21.15.174:6815/90848","heartbeat_front_addr":"172.21.15.174:6813/90848","state":["exists","up"]},{"osd":6,"uuid":"496a6272-892e-43c4-a145-206623c22f99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":90877},{"type":"v1","addr":"172.21.15.174:6817","nonce":90877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":90877},{"type":"v1","addr":"172.21.15.174:6819","nonce":90877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":90877},{"type":"v1","addr":"172.21.15.174:6823","nonce":90877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":90877},{"type":"v1","addr":"172.21.15.174:6821","nonce":90877}]},"public_addr":"172.21.15.174:6817/90877","cluster_addr":"172.21.15.174:6819/90877","heartbeat_back_addr":"172.21.15.174:6823/90877","heartbeat_front_addr":"172.21.15.174:6821/90877","state":["exists","up"]},{"osd":7,"uuid":"49967e34-8b07-4bab-a13e-0c16d6a0f3a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":90878},{"type":"v1","addr":"172.21.15.174:6825","nonce":90878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":90878},{"type":"v1","addr":"172.21.15.174:6827","nonce":90878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":90878},{"type":"v1","addr":"172.21.15.174:6831","nonce":90878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":90878},{"type":"v1","addr":"172.21.15.174:6829","nonce":90878}]},"public_addr":"172.21.15.174:6825/90878","cluster_addr":"172.21.15.174:6827/90878","heartbeat_back_addr":"172.21.15.174:6831/90878","heartbeat_front_addr":"172.21.15.174:6829/90878","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.446479+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.514266+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.794108+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.810894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.731919+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.717756+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.737435+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.508682+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T05:07:08.328 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.325+0000 7fca549c9700 1 -- 172.21.15.98:0/4129430731 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fca30066f30 msgr2=0x7fca300693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:08.328 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.325+0000 7fca549c9700 1 --2- 172.21.15.98:0/4129430731 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fca30066f30 0x7fca300693e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fca38009910 tx=0x7fca38017000 comp rx=0 tx=0).stop 2023-10-06T05:07:08.329 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.325+0000 7fca549c9700 1 -- 172.21.15.98:0/4129430731 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca500b0760 msgr2=0x7fca500ae900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:08.329 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.325+0000 7fca549c9700 1 --2- 172.21.15.98:0/4129430731 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca500b0760 0x7fca500ae900 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fca4000cd30 tx=0x7fca40002b90 comp rx=0 tx=0).stop 2023-10-06T05:07:08.329 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.325+0000 7fca549c9700 1 -- 172.21.15.98:0/4129430731 shutdown_connections 2023-10-06T05:07:08.329 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.325+0000 7fca549c9700 1 --2- 172.21.15.98:0/4129430731 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fca50159560 0x7fca50159e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:08.329 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.325+0000 7fca549c9700 1 --2- 172.21.15.98:0/4129430731 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fca30066f30 0x7fca300693e0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:08.329 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.325+0000 7fca549c9700 1 --2- 172.21.15.98:0/4129430731 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca500b0760 0x7fca500ae900 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:08.330 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.325+0000 7fca549c9700 1 --2- 172.21.15.98:0/4129430731 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fca500b49a0 0x7fca50159930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:08.330 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.325+0000 7fca549c9700 1 -- 172.21.15.98:0/4129430731 >> 172.21.15.98:0/4129430731 conn(0x7fca500997f0 msgr2=0x7fca50158a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:08.331 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.325+0000 7fca549c9700 1 -- 172.21.15.98:0/4129430731 shutdown_connections 2023-10-06T05:07:08.331 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.325+0000 7fca549c9700 1 -- 172.21.15.98:0/4129430731 wait complete. 2023-10-06T05:07:08.341 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-06T05:07:08.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.456+0000 7f12be017700 1 Processor -- start 2023-10-06T05:07:08.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.456+0000 7f12be017700 1 -- start start 2023-10-06T05:07:08.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.457+0000 7f12be017700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f12b80c7ac0 0x7f12b80c7e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:08.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.457+0000 7f12be017700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f12b806e340 0x7f12b815a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:08.462 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.457+0000 7f12be017700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12b806e790 con 0x7f12b813f6f0 2023-10-06T05:07:08.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.457+0000 7f12be017700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12b815ade0 con 0x7f12b806e340 2023-10-06T05:07:08.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.457+0000 7f12be017700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12b806ea20 con 0x7f12b80c7ac0 2023-10-06T05:07:08.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.457+0000 7f12b6ffd700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f12b806e340 0x7f12b815a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:08.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.457+0000 7f12b7fff700 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f12b813f6f0 0x7f12b80c66e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.98:60354/0 (socket says 172.21.15.98:60354) 2023-10-06T05:07:08.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.457+0000 7f12b7fff700 1 -- 172.21.15.98:0/1231051114 learned_addr learned my addr 172.21.15.98:0/1231051114 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:07:08.464 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.457+0000 7f12b77fe700 1 --2- 172.21.15.98:0/1231051114 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f12b80c7ac0 0x7f12b80c7e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:08.464 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.458+0000 7f12b6ffd700 1 -- 172.21.15.98:0/1231051114 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f12b80c7ac0 msgr2=0x7f12b80c7e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:08.464 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.458+0000 7f12b6ffd700 1 --2- 172.21.15.98:0/1231051114 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f12b80c7ac0 0x7f12b80c7e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:08.464 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.458+0000 7f12b6ffd700 1 -- 172.21.15.98:0/1231051114 >> v1:172.21.15.174:6789/0 conn(0x7f12b813f6f0 legacy=0x7f12b80c66e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:08.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.458+0000 7f12b6ffd700 1 -- 172.21.15.98:0/1231051114 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12b806ed00 con 0x7f12b806e340 2023-10-06T05:07:08.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.458+0000 7f12b67fc700 1 -- 172.21.15.98:0/1231051114 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2512036351 0 0) 0x7f12b806e790 con 0x7f12b813f6f0 2023-10-06T05:07:08.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.458+0000 7f12b6ffd700 1 --2- 172.21.15.98:0/1231051114 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f12b806e340 0x7f12b815a8a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f12ac0098e0 tx=0x7f12ac0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c879866f22ef0736 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:08.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.458+0000 7f12b67fc700 1 -- 172.21.15.98:0/1231051114 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12ac00b6d0 con 0x7f12b806e340 2023-10-06T05:07:08.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.458+0000 7f12b67fc700 1 -- 172.21.15.98:0/1231051114 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12ac00b830 con 0x7f12b806e340 2023-10-06T05:07:08.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.458+0000 7f12b67fc700 1 -- 172.21.15.98:0/1231051114 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12ac00bbd0 con 0x7f12b806e340 2023-10-06T05:07:08.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.459+0000 7f12be017700 1 -- 172.21.15.98:0/1231051114 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f12b806e340 msgr2=0x7f12b815a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:08.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.459+0000 7f12be017700 1 --2- 172.21.15.98:0/1231051114 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f12b806e340 0x7f12b815a8a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f12ac0098e0 tx=0x7f12ac0091a0 comp rx=0 tx=0).stop 2023-10-06T05:07:08.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.459+0000 7f12be017700 1 -- 172.21.15.98:0/1231051114 shutdown_connections 2023-10-06T05:07:08.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.459+0000 7f12be017700 1 --2- 172.21.15.98:0/1231051114 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f12b80c7ac0 0x7f12b80c7e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:08.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.459+0000 7f12be017700 1 --2- 172.21.15.98:0/1231051114 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f12b806e340 0x7f12b815a8a0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:08.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.459+0000 7f12be017700 1 -- 172.21.15.98:0/1231051114 >> 172.21.15.98:0/1231051114 conn(0x7f12b80cc0a0 msgr2=0x7f12b80cc4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:08.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.459+0000 7f12be017700 1 -- 172.21.15.98:0/1231051114 shutdown_connections 2023-10-06T05:07:08.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.459+0000 7f12be017700 1 -- 172.21.15.98:0/1231051114 wait complete. 2023-10-06T05:07:08.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.460+0000 7f12be017700 1 Processor -- start 2023-10-06T05:07:08.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.460+0000 7f12be017700 1 -- start start 2023-10-06T05:07:08.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.461+0000 7f12be017700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12b806e340 0x7f12b8151ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:08.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.461+0000 7f12be017700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f12b80c7ac0 0x7f12b8150750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:08.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.461+0000 7f12be017700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f12b813f6f0 0x7f12b813c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:08.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.461+0000 7f12be017700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b8152c30 con 0x7f12b806e340 2023-10-06T05:07:08.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.461+0000 7f12be017700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b8150c90 con 0x7f12b813f6f0 2023-10-06T05:07:08.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.461+0000 7f12be017700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12b8152950 con 0x7f12b80c7ac0 2023-10-06T05:07:08.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.461+0000 7f12b77fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12b806e340 0x7f12b8151ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:08.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.461+0000 7f12b77fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12b806e340 0x7f12b8151ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47912/0 (socket says 172.21.15.98:47912) 2023-10-06T05:07:08.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.461+0000 7f12b77fe700 1 -- 172.21.15.98:0/1883169768 learned_addr learned my addr 172.21.15.98:0/1883169768 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:08.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.461+0000 7f12b6ffd700 1 --2- 172.21.15.98:0/1883169768 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f12b80c7ac0 0x7f12b8150750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:08.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.461+0000 7f12b7fff700 1 --2- 172.21.15.98:0/1883169768 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f12b813f6f0 0x7f12b813c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:08.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.461+0000 7f12b77fe700 1 -- 172.21.15.98:0/1883169768 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f12b80c7ac0 msgr2=0x7f12b8150750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:08.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.461+0000 7f12b77fe700 1 --2- 172.21.15.98:0/1883169768 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f12b80c7ac0 0x7f12b8150750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:08.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.461+0000 7f12b77fe700 1 -- 172.21.15.98:0/1883169768 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f12b813f6f0 msgr2=0x7f12b813c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:08.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.461+0000 7f12b77fe700 1 --2- 172.21.15.98:0/1883169768 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f12b813f6f0 0x7f12b813c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:08.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.461+0000 7f12b77fe700 1 -- 172.21.15.98:0/1883169768 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12ac009420 con 0x7f12b806e340 2023-10-06T05:07:08.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.461+0000 7f12b77fe700 1 --2- 172.21.15.98:0/1883169768 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12b806e340 0x7f12b8151ec0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f12a8009f20 tx=0x7f12a800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:08.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.461+0000 7f129f7fe700 1 -- 172.21.15.98:0/1883169768 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a800cb70 con 0x7f12b806e340 2023-10-06T05:07:08.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.461+0000 7f129f7fe700 1 -- 172.21.15.98:0/1883169768 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12a800f040 con 0x7f12b806e340 2023-10-06T05:07:08.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.461+0000 7f129f7fe700 1 -- 172.21.15.98:0/1883169768 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a8014480 con 0x7f12b806e340 2023-10-06T05:07:08.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.461+0000 7f12be017700 1 -- 172.21.15.98:0/1883169768 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12b8153060 con 0x7f12b806e340 2023-10-06T05:07:08.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.462+0000 7f12be017700 1 -- 172.21.15.98:0/1883169768 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12b8153620 con 0x7f12b806e340 2023-10-06T05:07:08.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.463+0000 7f12be017700 1 -- 172.21.15.98:0/1883169768 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12b8152ec0 con 0x7f12b806e340 2023-10-06T05:07:08.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.463+0000 7f129f7fe700 1 -- 172.21.15.98:0/1883169768 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f12a80147f0 con 0x7f12b806e340 2023-10-06T05:07:08.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.463+0000 7f129f7fe700 1 --2- 172.21.15.98:0/1883169768 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f12a4066f30 0x7f12a40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:08.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.464+0000 7f129f7fe700 1 -- 172.21.15.98:0/1883169768 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f12a80852b0 con 0x7f12b806e340 2023-10-06T05:07:08.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.464+0000 7f12b6ffd700 1 --2- 172.21.15.98:0/1883169768 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f12a4066f30 0x7f12a40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:08.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.464+0000 7f12b6ffd700 1 --2- 172.21.15.98:0/1883169768 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f12a4066f30 0x7f12a40693e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f12ac0093f0 tx=0x7f12ac017000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:08.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.468+0000 7f129f7fe700 1 -- 172.21.15.98:0/1883169768 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f12b8152ec0 con 0x7f12b806e340 2023-10-06T05:07:08.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.697+0000 7f12be017700 1 -- 172.21.15.98:0/1883169768 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f12b816b5a0 con 0x7f12b806e340 2023-10-06T05:07:08.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.775+0000 7f129f7fe700 1 -- 172.21.15.98:0/1883169768 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v7) v1 ==== 118+0+0 (secure 0 0 0) 0x7f12a80520d0 con 0x7f12b806e340 2023-10-06T05:07:08.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.778+0000 7f12be017700 1 -- 172.21.15.98:0/1883169768 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f12a4066f30 msgr2=0x7f12a40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:08.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.778+0000 7f12be017700 1 --2- 172.21.15.98:0/1883169768 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f12a4066f30 0x7f12a40693e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f12ac0093f0 tx=0x7f12ac017000 comp rx=0 tx=0).stop 2023-10-06T05:07:08.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.778+0000 7f12be017700 1 -- 172.21.15.98:0/1883169768 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12b806e340 msgr2=0x7f12b8151ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:08.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.778+0000 7f12be017700 1 --2- 172.21.15.98:0/1883169768 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12b806e340 0x7f12b8151ec0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f12a8009f20 tx=0x7f12a800b040 comp rx=0 tx=0).stop 2023-10-06T05:07:08.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.778+0000 7f12be017700 1 -- 172.21.15.98:0/1883169768 shutdown_connections 2023-10-06T05:07:08.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.778+0000 7f12be017700 1 --2- 172.21.15.98:0/1883169768 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f12b813f6f0 0x7f12b813c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:08.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.778+0000 7f12be017700 1 --2- 172.21.15.98:0/1883169768 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f12a4066f30 0x7f12a40693e0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:08.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.778+0000 7f12be017700 1 --2- 172.21.15.98:0/1883169768 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12b806e340 0x7f12b8151ec0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:08.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.778+0000 7f12be017700 1 --2- 172.21.15.98:0/1883169768 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f12b80c7ac0 0x7f12b8150750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:08.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.778+0000 7f12be017700 1 -- 172.21.15.98:0/1883169768 >> 172.21.15.98:0/1883169768 conn(0x7f12b80cc0a0 msgr2=0x7f12b80c2130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:08.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.778+0000 7f12be017700 1 -- 172.21.15.98:0/1883169768 shutdown_connections 2023-10-06T05:07:08.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.778+0000 7f12be017700 1 -- 172.21.15.98:0/1883169768 wait complete. 2023-10-06T05:07:08.792 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-06T05:07:08.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.907+0000 7f37fabd5700 1 Processor -- start 2023-10-06T05:07:08.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.907+0000 7f37fabd5700 1 -- start start 2023-10-06T05:07:08.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.908+0000 7f37fabd5700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f37f40b0820 0x7f37f40686c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:08.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.908+0000 7f37fabd5700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f37f40cc740 0x7f37f40ccb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:08.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.908+0000 7f37fabd5700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37f40c9670 con 0x7f37f40c4450 2023-10-06T05:07:08.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.908+0000 7f37fabd5700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37f4159c00 con 0x7f37f40cc740 2023-10-06T05:07:08.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.908+0000 7f37fabd5700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37f40c8a60 con 0x7f37f40b0820 2023-10-06T05:07:08.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.908+0000 7f37f9172700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f37f40cc740 0x7f37f40ccb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:08.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.908+0000 7f37f9172700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f37f40cc740 0x7f37f40ccb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47922/0 (socket says 172.21.15.98:47922) 2023-10-06T05:07:08.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.908+0000 7f37f9172700 1 -- 172.21.15.98:0/4187107898 learned_addr learned my addr 172.21.15.98:0/4187107898 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:08.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.908+0000 7f37f8971700 1 --2- 172.21.15.98:0/4187107898 >> v2:172.21.15.174:3300/0 conn(0x7f37f40b0820 0x7f37f40686c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:08.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.908+0000 7f37f9172700 1 -- 172.21.15.98:0/4187107898 >> v2:172.21.15.174:3300/0 conn(0x7f37f40b0820 msgr2=0x7f37f40686c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:08.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.908+0000 7f37f9172700 1 --2- 172.21.15.98:0/4187107898 >> v2:172.21.15.174:3300/0 conn(0x7f37f40b0820 0x7f37f40686c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:08.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.908+0000 7f37f9172700 1 -- 172.21.15.98:0/4187107898 >> v1:172.21.15.174:6789/0 conn(0x7f37f40c4450 legacy=0x7f37f40cec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:08.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.908+0000 7f37f9172700 1 -- 172.21.15.98:0/4187107898 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37f40c8d40 con 0x7f37f40cc740 2023-10-06T05:07:08.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.908+0000 7f37f9172700 1 --2- 172.21.15.98:0/4187107898 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37f40cc740 0x7f37f40ccb60 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f37f406e180 tx=0x7f37dc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ed9f12a1169ee17 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:08.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.908+0000 7f37f37fe700 1 -- 172.21.15.98:0/4187107898 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37dc00b860 con 0x7f37f40cc740 2023-10-06T05:07:08.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.909+0000 7f37f37fe700 1 -- 172.21.15.98:0/4187107898 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37dc00b9c0 con 0x7f37f40cc740 2023-10-06T05:07:08.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.909+0000 7f37f37fe700 1 -- 172.21.15.98:0/4187107898 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37dc00bd60 con 0x7f37f40cc740 2023-10-06T05:07:08.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.909+0000 7f37fabd5700 1 -- 172.21.15.98:0/4187107898 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37f40cc740 msgr2=0x7f37f40ccb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:08.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.909+0000 7f37fabd5700 1 --2- 172.21.15.98:0/4187107898 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37f40cc740 0x7f37f40ccb60 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f37f406e180 tx=0x7f37dc009250 comp rx=0 tx=0).stop 2023-10-06T05:07:08.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.909+0000 7f37fabd5700 1 -- 172.21.15.98:0/4187107898 shutdown_connections 2023-10-06T05:07:08.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.909+0000 7f37fabd5700 1 --2- 172.21.15.98:0/4187107898 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37f40cc740 0x7f37f40ccb60 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:08.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.909+0000 7f37fabd5700 1 --2- 172.21.15.98:0/4187107898 >> v2:172.21.15.174:3300/0 conn(0x7f37f40b0820 0x7f37f40686c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:08.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.909+0000 7f37fabd5700 1 -- 172.21.15.98:0/4187107898 >> 172.21.15.98:0/4187107898 conn(0x7f37f4087fc0 msgr2=0x7f37f40ce310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:08.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.909+0000 7f37fabd5700 1 -- 172.21.15.98:0/4187107898 shutdown_connections 2023-10-06T05:07:08.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.909+0000 7f37fabd5700 1 -- 172.21.15.98:0/4187107898 wait complete. 2023-10-06T05:07:08.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.910+0000 7f37fabd5700 1 Processor -- start 2023-10-06T05:07:08.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.910+0000 7f37fabd5700 1 -- start start 2023-10-06T05:07:08.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.911+0000 7f37fabd5700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37f40b0820 0x7f37f414a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:08.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.911+0000 7f37fabd5700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f37f40c4450 0x7f37f40b3200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:08.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.911+0000 7f37fabd5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37f40cc740 0x7f37f40b3740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:08.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.911+0000 7f37fabd5700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37f415f5f0 con 0x7f37f40cc740 2023-10-06T05:07:08.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.911+0000 7f37fabd5700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37f415fa80 con 0x7f37f40b0820 2023-10-06T05:07:08.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.911+0000 7f37fabd5700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37f415e9e0 con 0x7f37f40c4450 2023-10-06T05:07:08.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.911+0000 7f37f9172700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37f40cc740 0x7f37f40b3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:08.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.911+0000 7f37f9172700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37f40cc740 0x7f37f40b3740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47926/0 (socket says 172.21.15.98:47926) 2023-10-06T05:07:08.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.911+0000 7f37f9172700 1 -- 172.21.15.98:0/523928396 learned_addr learned my addr 172.21.15.98:0/523928396 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:08.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.911+0000 7f37f8971700 1 --2- 172.21.15.98:0/523928396 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37f40b0820 0x7f37f414a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:08.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.911+0000 7f37f3fff700 1 --2- 172.21.15.98:0/523928396 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f37f40c4450 0x7f37f40b3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:08.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.912+0000 7f37f9172700 1 -- 172.21.15.98:0/523928396 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f37f40c4450 msgr2=0x7f37f40b3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:08.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.912+0000 7f37f9172700 1 --2- 172.21.15.98:0/523928396 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f37f40c4450 0x7f37f40b3200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:08.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.912+0000 7f37f9172700 1 -- 172.21.15.98:0/523928396 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37f40b0820 msgr2=0x7f37f414a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:08.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.912+0000 7f37f9172700 1 --2- 172.21.15.98:0/523928396 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37f40b0820 0x7f37f414a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:08.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.912+0000 7f37f9172700 1 -- 172.21.15.98:0/523928396 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37dc0094f0 con 0x7f37f40cc740 2023-10-06T05:07:08.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.912+0000 7f37f3fff700 1 --2- 172.21.15.98:0/523928396 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f37f40c4450 0x7f37f40b3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:07:08.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.912+0000 7f37f9172700 1 --2- 172.21.15.98:0/523928396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37f40cc740 0x7f37f40b3740 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f37dc009290 tx=0x7f37dc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:08.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.912+0000 7f37f0ff9700 1 -- 172.21.15.98:0/523928396 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37dc01a700 con 0x7f37f40cc740 2023-10-06T05:07:08.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.912+0000 7f37f0ff9700 1 -- 172.21.15.98:0/523928396 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37dc01a860 con 0x7f37f40cc740 2023-10-06T05:07:08.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.912+0000 7f37f0ff9700 1 -- 172.21.15.98:0/523928396 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37dc01ac20 con 0x7f37f40cc740 2023-10-06T05:07:08.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.912+0000 7f37fabd5700 1 -- 172.21.15.98:0/523928396 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37f415ecc0 con 0x7f37f40cc740 2023-10-06T05:07:08.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.912+0000 7f37fabd5700 1 -- 172.21.15.98:0/523928396 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37f415f280 con 0x7f37f40cc740 2023-10-06T05:07:08.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.913+0000 7f37fabd5700 1 -- 172.21.15.98:0/523928396 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37f4064a90 con 0x7f37f40cc740 2023-10-06T05:07:08.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.913+0000 7f37f0ff9700 1 -- 172.21.15.98:0/523928396 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f37dc02d440 con 0x7f37f40cc740 2023-10-06T05:07:08.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.914+0000 7f37f0ff9700 1 --2- 172.21.15.98:0/523928396 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f37d4066f30 0x7f37d40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:08.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.914+0000 7f37f8971700 1 --2- 172.21.15.98:0/523928396 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f37d4066f30 0x7f37d40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:08.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.914+0000 7f37f0ff9700 1 -- 172.21.15.98:0/523928396 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f37dc086650 con 0x7f37f40cc740 2023-10-06T05:07:08.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.914+0000 7f37f8971700 1 --2- 172.21.15.98:0/523928396 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f37d4066f30 0x7f37d40693e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f37ec006aa0 tx=0x7f37ec008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:08.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:08.918+0000 7f37f0ff9700 1 -- 172.21.15.98:0/523928396 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f37dc053470 con 0x7f37f40cc740 2023-10-06T05:07:09.148 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:09.146+0000 7f37fabd5700 1 -- 172.21.15.98:0/523928396 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f37f41598b0 con 0x7f37f40cc740 2023-10-06T05:07:09.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:09.781+0000 7f37f0ff9700 1 -- 172.21.15.98:0/523928396 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v8) v1 ==== 122+0+0 (secure 0 0 0) 0x7f37dc056a90 con 0x7f37f40cc740 2023-10-06T05:07:09.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:09.784+0000 7f37fabd5700 1 -- 172.21.15.98:0/523928396 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f37d4066f30 msgr2=0x7f37d40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:09.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:09.784+0000 7f37fabd5700 1 --2- 172.21.15.98:0/523928396 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f37d4066f30 0x7f37d40693e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f37ec006aa0 tx=0x7f37ec008040 comp rx=0 tx=0).stop 2023-10-06T05:07:09.787 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:09.784+0000 7f37fabd5700 1 -- 172.21.15.98:0/523928396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37f40cc740 msgr2=0x7f37f40b3740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:09.787 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:09.784+0000 7f37fabd5700 1 --2- 172.21.15.98:0/523928396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37f40cc740 0x7f37f40b3740 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f37dc009290 tx=0x7f37dc017000 comp rx=0 tx=0).stop 2023-10-06T05:07:09.787 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:09.784+0000 7f37fabd5700 1 -- 172.21.15.98:0/523928396 shutdown_connections 2023-10-06T05:07:09.787 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:09.784+0000 7f37fabd5700 1 --2- 172.21.15.98:0/523928396 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f37d4066f30 0x7f37d40693e0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:09.787 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:09.784+0000 7f37fabd5700 1 --2- 172.21.15.98:0/523928396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37f40cc740 0x7f37f40b3740 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:09.787 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:09.784+0000 7f37fabd5700 1 --2- 172.21.15.98:0/523928396 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37f40b0820 0x7f37f414a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:09.787 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:09.784+0000 7f37fabd5700 1 --2- 172.21.15.98:0/523928396 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f37f40c4450 0x7f37f40b3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:09.787 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:09.784+0000 7f37fabd5700 1 -- 172.21.15.98:0/523928396 >> 172.21.15.98:0/523928396 conn(0x7f37f4087fc0 msgr2=0x7f37f40c30b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:09.787 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:09.784+0000 7f37fabd5700 1 -- 172.21.15.98:0/523928396 shutdown_connections 2023-10-06T05:07:09.787 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:09.784+0000 7f37fabd5700 1 -- 172.21.15.98:0/523928396 wait complete. 2023-10-06T05:07:09.798 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-06T05:07:09.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:09.911+0000 7fef8759e700 1 Processor -- start 2023-10-06T05:07:09.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:09.911+0000 7fef8759e700 1 -- start start 2023-10-06T05:07:09.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:09.912+0000 7fef8759e700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fef880cd330 0x7fef880ce980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:09.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:09.912+0000 7fef8759e700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fef880c6220 0x7fef880ceec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:09.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:09.912+0000 7fef8759e700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef880cc800 con 0x7fef88159470 2023-10-06T05:07:09.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:09.912+0000 7fef8759e700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef880cbc30 con 0x7fef880cd330 2023-10-06T05:07:09.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:09.912+0000 7fef8759e700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef880cf400 con 0x7fef880c6220 2023-10-06T05:07:09.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:09.912+0000 7fef85d9b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fef880cd330 0x7fef880ce980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:09.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:09.912+0000 7fef85d9b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fef880cd330 0x7fef880ce980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45622/0 (socket says 172.21.15.98:45622) 2023-10-06T05:07:09.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:09.912+0000 7fef85d9b700 1 -- 172.21.15.98:0/13133103 learned_addr learned my addr 172.21.15.98:0/13133103 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:09.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:09.912+0000 7fef86d9d700 1 --2- 172.21.15.98:0/13133103 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fef880c6220 0x7fef880ceec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:09.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:09.912+0000 7fef85d9b700 1 -- 172.21.15.98:0/13133103 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fef880c6220 msgr2=0x7fef880ceec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:09.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:09.912+0000 7fef85d9b700 1 --2- 172.21.15.98:0/13133103 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fef880c6220 0x7fef880ceec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:09.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:09.912+0000 7fef85d9b700 1 -- 172.21.15.98:0/13133103 >> v1:172.21.15.174:6789/0 conn(0x7fef88159470 legacy=0x7fef880ccc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:09.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:09.913+0000 7fef85d9b700 1 -- 172.21.15.98:0/13133103 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef880cbed0 con 0x7fef880cd330 2023-10-06T05:07:09.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:09.913+0000 7fef85d9b700 1 --2- 172.21.15.98:0/13133103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef880cd330 0x7fef880ce980 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fef7c009990 tx=0x7fef7c009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c093ce434d65f8 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:09.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:09.913+0000 7fef8559a700 1 -- 172.21.15.98:0/13133103 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef7c005c10 con 0x7fef880cd330 2023-10-06T05:07:09.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:09.913+0000 7fef8559a700 1 -- 172.21.15.98:0/13133103 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef7c005d70 con 0x7fef880cd330 2023-10-06T05:07:09.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:09.913+0000 7fef8559a700 1 -- 172.21.15.98:0/13133103 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef7c00b8a0 con 0x7fef880cd330 2023-10-06T05:07:09.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:09.913+0000 7fef8759e700 1 -- 172.21.15.98:0/13133103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef880cd330 msgr2=0x7fef880ce980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:09.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:09.913+0000 7fef8759e700 1 --2- 172.21.15.98:0/13133103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef880cd330 0x7fef880ce980 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fef7c009990 tx=0x7fef7c009220 comp rx=0 tx=0).stop 2023-10-06T05:07:09.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:09.913+0000 7fef8759e700 1 -- 172.21.15.98:0/13133103 shutdown_connections 2023-10-06T05:07:09.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:09.913+0000 7fef8759e700 1 --2- 172.21.15.98:0/13133103 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fef880c6220 0x7fef880ceec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:09.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:09.913+0000 7fef8759e700 1 --2- 172.21.15.98:0/13133103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef880cd330 0x7fef880ce980 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:09.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:09.913+0000 7fef8759e700 1 -- 172.21.15.98:0/13133103 >> 172.21.15.98:0/13133103 conn(0x7fef880995c0 msgr2=0x7fef8806f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:09.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:09.913+0000 7fef8759e700 1 -- 172.21.15.98:0/13133103 shutdown_connections 2023-10-06T05:07:09.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:09.913+0000 7fef8759e700 1 -- 172.21.15.98:0/13133103 wait complete. 2023-10-06T05:07:09.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:09.914+0000 7fef8759e700 1 Processor -- start 2023-10-06T05:07:09.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:09.914+0000 7fef8759e700 1 -- start start 2023-10-06T05:07:09.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:09.915+0000 7fef8759e700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef880c6220 0x7fef88146f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:09.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:09.915+0000 7fef8759e700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fef880cd330 0x7fef8815bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:09.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:09.915+0000 7fef8759e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef88159470 0x7fef8815a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:09.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:09.915+0000 7fef8759e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef880c7890 con 0x7fef88159470 2023-10-06T05:07:09.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:09.915+0000 7fef8759e700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef880c7b70 con 0x7fef880c6220 2023-10-06T05:07:09.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:09.915+0000 7fef8759e700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef880c7e50 con 0x7fef880cd330 2023-10-06T05:07:09.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:09.915+0000 7fef86d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef88159470 0x7fef8815a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:09.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:09.915+0000 7fef86d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef88159470 0x7fef8815a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45628/0 (socket says 172.21.15.98:45628) 2023-10-06T05:07:09.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:09.915+0000 7fef86d9d700 1 -- 172.21.15.98:0/2174380699 learned_addr learned my addr 172.21.15.98:0/2174380699 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:09.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:09.915+0000 7fef85d9b700 1 --2- 172.21.15.98:0/2174380699 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fef880cd330 0x7fef8815bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:09.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:09.915+0000 7fef8659c700 1 --2- 172.21.15.98:0/2174380699 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef880c6220 0x7fef88146f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:09.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:09.916+0000 7fef86d9d700 1 -- 172.21.15.98:0/2174380699 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fef880cd330 msgr2=0x7fef8815bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:09.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:09.916+0000 7fef86d9d700 1 --2- 172.21.15.98:0/2174380699 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fef880cd330 0x7fef8815bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:09.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:09.916+0000 7fef86d9d700 1 -- 172.21.15.98:0/2174380699 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef880c6220 msgr2=0x7fef88146f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:09.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:09.916+0000 7fef86d9d700 1 --2- 172.21.15.98:0/2174380699 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef880c6220 0x7fef88146f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:09.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:09.916+0000 7fef86d9d700 1 -- 172.21.15.98:0/2174380699 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef7c009480 con 0x7fef88159470 2023-10-06T05:07:09.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:09.916+0000 7fef86d9d700 1 --2- 172.21.15.98:0/2174380699 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef88159470 0x7fef8815a070 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fef80007f20 tx=0x7fef80009860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:09.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:09.916+0000 7fef767fc700 1 -- 172.21.15.98:0/2174380699 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef8000eb90 con 0x7fef88159470 2023-10-06T05:07:09.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:09.916+0000 7fef767fc700 1 -- 172.21.15.98:0/2174380699 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef8000ed10 con 0x7fef88159470 2023-10-06T05:07:09.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:09.916+0000 7fef8759e700 1 -- 172.21.15.98:0/2174380699 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef880c8130 con 0x7fef88159470 2023-10-06T05:07:09.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:09.916+0000 7fef767fc700 1 -- 172.21.15.98:0/2174380699 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef80014670 con 0x7fef88159470 2023-10-06T05:07:09.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:09.916+0000 7fef8759e700 1 -- 172.21.15.98:0/2174380699 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef880c73b0 con 0x7fef88159470 2023-10-06T05:07:09.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:09.917+0000 7fef8759e700 1 -- 172.21.15.98:0/2174380699 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef880c8370 con 0x7fef88159470 2023-10-06T05:07:09.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:09.917+0000 7fef767fc700 1 -- 172.21.15.98:0/2174380699 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fef800149e0 con 0x7fef88159470 2023-10-06T05:07:09.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:09.918+0000 7fef767fc700 1 --2- 172.21.15.98:0/2174380699 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fef68066f30 0x7fef680693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:09.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:09.918+0000 7fef8659c700 1 --2- 172.21.15.98:0/2174380699 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fef68066f30 0x7fef680693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:09.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:09.918+0000 7fef767fc700 1 -- 172.21.15.98:0/2174380699 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fef80084f80 con 0x7fef88159470 2023-10-06T05:07:09.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:09.918+0000 7fef8659c700 1 --2- 172.21.15.98:0/2174380699 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fef68066f30 0x7fef680693e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fef88069860 tx=0x7fef78009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:09.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:09.922+0000 7fef767fc700 1 -- 172.21.15.98:0/2174380699 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fef880c8370 con 0x7fef88159470 2023-10-06T05:07:10.147 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.145+0000 7fef8759e700 1 -- 172.21.15.98:0/2174380699 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fef880c8370 con 0x7fef88159470 2023-10-06T05:07:10.148 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.146+0000 7fef767fc700 1 -- 172.21.15.98:0/2174380699 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4596 (secure 0 0 0) 0x7fef880c8370 con 0x7fef88159470 2023-10-06T05:07:10.148 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:07:10.148 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4285,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6834/925250632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":925250632},{"type":"v1","addr":"172.21.15.174:6834","nonce":925250632}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4288,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/1629449810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6833","nonce":1629449810},{"type":"v1","addr":"172.21.15.174:6835","nonce":1629449810}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4299,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/3961327990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3961327990},{"type":"v1","addr":"172.21.15.98:6836","nonce":3961327990}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-06T05:06:57.086683+0000","modified":"2023-10-06T05:07:09.773654+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.98:6837/2683239094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":2683239094},{"type":"v1","addr":"172.21.15.98:6837","nonce":2683239094}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-06T05:07:10.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.148+0000 7fef8759e700 1 -- 172.21.15.98:0/2174380699 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fef68066f30 msgr2=0x7fef680693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:10.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.148+0000 7fef8759e700 1 --2- 172.21.15.98:0/2174380699 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fef68066f30 0x7fef680693e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fef88069860 tx=0x7fef78009040 comp rx=0 tx=0).stop 2023-10-06T05:07:10.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.148+0000 7fef8759e700 1 -- 172.21.15.98:0/2174380699 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef88159470 msgr2=0x7fef8815a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:10.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.148+0000 7fef8759e700 1 --2- 172.21.15.98:0/2174380699 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef88159470 0x7fef8815a070 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fef80007f20 tx=0x7fef80009860 comp rx=0 tx=0).stop 2023-10-06T05:07:10.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.148+0000 7fef8759e700 1 -- 172.21.15.98:0/2174380699 shutdown_connections 2023-10-06T05:07:10.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.148+0000 7fef8759e700 1 --2- 172.21.15.98:0/2174380699 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fef68066f30 0x7fef680693e0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:10.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.148+0000 7fef8759e700 1 --2- 172.21.15.98:0/2174380699 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef88159470 0x7fef8815a070 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:10.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.148+0000 7fef8759e700 1 --2- 172.21.15.98:0/2174380699 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef880c6220 0x7fef88146f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:10.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.148+0000 7fef8759e700 1 --2- 172.21.15.98:0/2174380699 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fef880cd330 0x7fef8815bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:10.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.148+0000 7fef8759e700 1 -- 172.21.15.98:0/2174380699 >> 172.21.15.98:0/2174380699 conn(0x7fef880995c0 msgr2=0x7fef881582a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:10.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.148+0000 7fef8759e700 1 -- 172.21.15.98:0/2174380699 shutdown_connections 2023-10-06T05:07:10.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.148+0000 7fef8759e700 1 -- 172.21.15.98:0/2174380699 wait complete. 2023-10-06T05:07:10.156 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 8 2023-10-06T05:07:10.162 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T05:07:10.275 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.273+0000 7f981e627700 1 Processor -- start 2023-10-06T05:07:10.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.273+0000 7f981e627700 1 -- start start 2023-10-06T05:07:10.278 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.274+0000 7f981e627700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f98180b49a0 0x7f98180aac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:10.278 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.274+0000 7f981e627700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f98180b0760 0x7f98180b1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:10.278 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.274+0000 7f981e627700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98180b0c60 con 0x7f9818159560 2023-10-06T05:07:10.278 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.274+0000 7f981e627700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98180b0f40 con 0x7f98180b49a0 2023-10-06T05:07:10.278 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.274+0000 7f981e627700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98180b1220 con 0x7f98180b0760 2023-10-06T05:07:10.278 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.274+0000 7f9817fff700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f9818159560 0x7f98180aa530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:41100/0 (socket says 172.21.15.98:41100) 2023-10-06T05:07:10.278 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.274+0000 7f9817fff700 1 -- 172.21.15.98:0/1658616306 learned_addr learned my addr 172.21.15.98:0/1658616306 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:07:10.278 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.274+0000 7f981cbc4700 1 --2- 172.21.15.98:0/1658616306 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f98180b0760 0x7f98180b1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:10.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.274+0000 7f98177fe700 1 --2- 172.21.15.98:0/1658616306 >> v2:172.21.15.174:3300/0 conn(0x7f98180b49a0 0x7f98180aac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:10.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.275+0000 7f9816ffd700 1 -- 172.21.15.98:0/1658616306 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1563719887 0 0) 0x7f98180b0c60 con 0x7f9818159560 2023-10-06T05:07:10.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.275+0000 7f9816ffd700 1 -- 172.21.15.98:0/1658616306 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9800003570 con 0x7f9818159560 2023-10-06T05:07:10.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.275+0000 7f9816ffd700 1 -- 172.21.15.98:0/1658616306 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2710139582 0 0) 0x7f9800003570 con 0x7f9818159560 2023-10-06T05:07:10.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.275+0000 7f98177fe700 1 -- 172.21.15.98:0/1658616306 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f98180b0760 msgr2=0x7f98180b1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:10.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.275+0000 7f98177fe700 1 --2- 172.21.15.98:0/1658616306 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f98180b0760 0x7f98180b1bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:10.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.275+0000 7f98177fe700 1 -- 172.21.15.98:0/1658616306 >> v1:172.21.15.98:6789/0 conn(0x7f9818159560 legacy=0x7f98180aa530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:10.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.275+0000 7f98177fe700 1 -- 172.21.15.98:0/1658616306 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98180b1500 con 0x7f98180b49a0 2023-10-06T05:07:10.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.275+0000 7f981cbc4700 1 --2- 172.21.15.98:0/1658616306 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f98180b0760 0x7f98180b1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T05:07:10.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.275+0000 7f98177fe700 1 --2- 172.21.15.98:0/1658616306 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f98180b49a0 0x7f98180aac40 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f980c009910 tx=0x7f980c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f677ef599e29cb52 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:10.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.275+0000 7f9816ffd700 1 -- 172.21.15.98:0/1658616306 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98080015f0 con 0x7f98180b49a0 2023-10-06T05:07:10.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.275+0000 7f9816ffd700 1 -- 172.21.15.98:0/1658616306 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f980c011420 con 0x7f98180b49a0 2023-10-06T05:07:10.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.275+0000 7f9816ffd700 1 -- 172.21.15.98:0/1658616306 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f980c0117c0 con 0x7f98180b49a0 2023-10-06T05:07:10.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.276+0000 7f981e627700 1 -- 172.21.15.98:0/1658616306 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f98180b49a0 msgr2=0x7f98180aac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:10.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.276+0000 7f981e627700 1 --2- 172.21.15.98:0/1658616306 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f98180b49a0 0x7f98180aac40 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f980c009910 tx=0x7f980c0091a0 comp rx=0 tx=0).stop 2023-10-06T05:07:10.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.276+0000 7f981e627700 1 -- 172.21.15.98:0/1658616306 shutdown_connections 2023-10-06T05:07:10.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.276+0000 7f981e627700 1 --2- 172.21.15.98:0/1658616306 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f98180b0760 0x7f98180b1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:10.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.276+0000 7f981e627700 1 --2- 172.21.15.98:0/1658616306 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f98180b49a0 0x7f98180aac40 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:10.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.276+0000 7f981e627700 1 -- 172.21.15.98:0/1658616306 >> 172.21.15.98:0/1658616306 conn(0x7f98180997f0 msgr2=0x7f981806f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:10.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.276+0000 7f981e627700 1 -- 172.21.15.98:0/1658616306 shutdown_connections 2023-10-06T05:07:10.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.276+0000 7f981e627700 1 -- 172.21.15.98:0/1658616306 wait complete. 2023-10-06T05:07:10.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.277+0000 7f981e627700 1 Processor -- start 2023-10-06T05:07:10.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.277+0000 7f981e627700 1 -- start start 2023-10-06T05:07:10.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.278+0000 7f981e627700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f98180b0760 0x7f981814a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:10.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.278+0000 7f981e627700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98180b49a0 0x7f9818159930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:10.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.278+0000 7f981e627700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9818159560 0x7f98180ae550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:10.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.278+0000 7f981e627700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f981815aee0 con 0x7f98180b49a0 2023-10-06T05:07:10.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.278+0000 7f981e627700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f981815a2d0 con 0x7f98180b0760 2023-10-06T05:07:10.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.278+0000 7f981e627700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f981815a5b0 con 0x7f9818159560 2023-10-06T05:07:10.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.278+0000 7f98177fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98180b49a0 0x7f9818159930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:10.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.278+0000 7f98177fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98180b49a0 0x7f9818159930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45632/0 (socket says 172.21.15.98:45632) 2023-10-06T05:07:10.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.278+0000 7f98177fe700 1 -- 172.21.15.98:0/3778355509 learned_addr learned my addr 172.21.15.98:0/3778355509 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:10.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.278+0000 7f9817fff700 1 --2- 172.21.15.98:0/3778355509 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f98180b0760 0x7f981814a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:10.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.278+0000 7f981cbc4700 1 --2- 172.21.15.98:0/3778355509 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9818159560 0x7f98180ae550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:10.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.278+0000 7f98177fe700 1 -- 172.21.15.98:0/3778355509 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9818159560 msgr2=0x7f98180ae550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:10.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.278+0000 7f98177fe700 1 --2- 172.21.15.98:0/3778355509 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9818159560 0x7f98180ae550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:10.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.278+0000 7f98177fe700 1 -- 172.21.15.98:0/3778355509 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f98180b0760 msgr2=0x7f981814a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:10.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.278+0000 7f98177fe700 1 --2- 172.21.15.98:0/3778355509 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f98180b0760 0x7f981814a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:10.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.278+0000 7f98177fe700 1 -- 172.21.15.98:0/3778355509 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9810009a20 con 0x7f98180b49a0 2023-10-06T05:07:10.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.278+0000 7f98177fe700 1 --2- 172.21.15.98:0/3778355509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98180b49a0 0x7f9818159930 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f980c0093f0 tx=0x7f980c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:10.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.278+0000 7f97fffff700 1 -- 172.21.15.98:0/3778355509 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f980c01bbd0 con 0x7f98180b49a0 2023-10-06T05:07:10.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.278+0000 7f981e627700 1 -- 172.21.15.98:0/3778355509 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f980c009420 con 0x7f98180b49a0 2023-10-06T05:07:10.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.278+0000 7f97fffff700 1 -- 172.21.15.98:0/3778355509 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f980c01bd30 con 0x7f98180b49a0 2023-10-06T05:07:10.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.278+0000 7f97fffff700 1 -- 172.21.15.98:0/3778355509 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f980c01bea0 con 0x7f98180b49a0 2023-10-06T05:07:10.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.278+0000 7f981e627700 1 -- 172.21.15.98:0/3778355509 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f981815ab70 con 0x7f98180b49a0 2023-10-06T05:07:10.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.279+0000 7f981e627700 1 -- 172.21.15.98:0/3778355509 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9818064a90 con 0x7f98180b49a0 2023-10-06T05:07:10.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.280+0000 7f97fffff700 1 -- 172.21.15.98:0/3778355509 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f980c01c930 con 0x7f98180b49a0 2023-10-06T05:07:10.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.280+0000 7f97fffff700 1 --2- 172.21.15.98:0/3778355509 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f9804066f30 0x7f98040693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:10.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.280+0000 7f97fffff700 1 -- 172.21.15.98:0/3778355509 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f980c085f80 con 0x7f98180b49a0 2023-10-06T05:07:10.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.280+0000 7f9817fff700 1 --2- 172.21.15.98:0/3778355509 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f9804066f30 0x7f98040693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:10.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.280+0000 7f9817fff700 1 --2- 172.21.15.98:0/3778355509 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f9804066f30 0x7f98040693e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9808000ef0 tx=0x7f98080016a0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:10.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.284+0000 7f97fffff700 1 -- 172.21.15.98:0/3778355509 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f980c0b75a0 con 0x7f98180b49a0 2023-10-06T05:07:10.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.470+0000 7f981e627700 1 -- 172.21.15.98:0/3778355509 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f98180b17e0 con 0x7f98180b49a0 2023-10-06T05:07:10.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.471+0000 7f97fffff700 1 -- 172.21.15.98:0/3778355509 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+18707 (secure 0 0 0) 0x7f980c052e60 con 0x7f98180b49a0 2023-10-06T05:07:10.473 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:07:10.474 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":29,"fsid":"f50c5df5-cad5-47a4-9bb1-8c791d4311ec","created":"2023-10-06T05:06:32.971175+0000","modified":"2023-10-06T05:07:02.131672+0000","last_up_change":"2023-10-06T05:06:44.016489+0000","last_in_change":"2023-10-06T05:06:35.544511+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-06T05:06:44.559671+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-06T05:06:51.432161+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-10-06T05:06:55.589057+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-10-06T05:06:56.041687+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-10-06T05:06:59.754099+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d9495d95-da75-4c3b-8927-e9bae6a8b093","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":90627},{"type":"v1","addr":"172.21.15.98:6801","nonce":90627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":90627},{"type":"v1","addr":"172.21.15.98:6803","nonce":90627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":90627},{"type":"v1","addr":"172.21.15.98:6807","nonce":90627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":90627},{"type":"v1","addr":"172.21.15.98:6805","nonce":90627}]},"public_addr":"172.21.15.98:6801/90627","cluster_addr":"172.21.15.98:6803/90627","heartbeat_back_addr":"172.21.15.98:6807/90627","heartbeat_front_addr":"172.21.15.98:6805/90627","state":["exists","up"]},{"osd":1,"uuid":"f25ba5be-d65e-4707-a134-154092deb754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":90628},{"type":"v1","addr":"172.21.15.98:6809","nonce":90628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":90628},{"type":"v1","addr":"172.21.15.98:6811","nonce":90628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":90628},{"type":"v1","addr":"172.21.15.98:6815","nonce":90628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":90628},{"type":"v1","addr":"172.21.15.98:6813","nonce":90628}]},"public_addr":"172.21.15.98:6809/90628","cluster_addr":"172.21.15.98:6811/90628","heartbeat_back_addr":"172.21.15.98:6815/90628","heartbeat_front_addr":"172.21.15.98:6813/90628","state":["exists","up"]},{"osd":2,"uuid":"5ab9e028-945c-43e2-8c95-3b0b4cf38e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":90662},{"type":"v1","addr":"172.21.15.98:6818","nonce":90662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":90662},{"type":"v1","addr":"172.21.15.98:6822","nonce":90662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":90662},{"type":"v1","addr":"172.21.15.98:6830","nonce":90662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":90662},{"type":"v1","addr":"172.21.15.98:6826","nonce":90662}]},"public_addr":"172.21.15.98:6818/90662","cluster_addr":"172.21.15.98:6822/90662","heartbeat_back_addr":"172.21.15.98:6830/90662","heartbeat_front_addr":"172.21.15.98:6826/90662","state":["exists","up"]},{"osd":3,"uuid":"3bb803ff-1738-46bf-be62-2267bf378822","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6817","nonce":90663},{"type":"v1","addr":"172.21.15.98:6819","nonce":90663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6821","nonce":90663},{"type":"v1","addr":"172.21.15.98:6823","nonce":90663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6829","nonce":90663},{"type":"v1","addr":"172.21.15.98:6831","nonce":90663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6825","nonce":90663},{"type":"v1","addr":"172.21.15.98:6827","nonce":90663}]},"public_addr":"172.21.15.98:6819/90663","cluster_addr":"172.21.15.98:6823/90663","heartbeat_back_addr":"172.21.15.98:6831/90663","heartbeat_front_addr":"172.21.15.98:6827/90663","state":["exists","up"]},{"osd":4,"uuid":"7d57a4c9-6462-43f6-b64a-7f9e9b4353d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":90847},{"type":"v1","addr":"172.21.15.174:6801","nonce":90847}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":90847},{"type":"v1","addr":"172.21.15.174:6803","nonce":90847}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":90847},{"type":"v1","addr":"172.21.15.174:6807","nonce":90847}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":90847},{"type":"v1","addr":"172.21.15.174:6805","nonce":90847}]},"public_addr":"172.21.15.174:6801/90847","cluster_addr":"172.21.15.174:6803/90847","heartbeat_back_addr":"172.21.15.174:6807/90847","heartbeat_front_addr":"172.21.15.174:6805/90847","state":["exists","up"]},{"osd":5,"uuid":"83287ea4-e44d-4af9-bc77-36d47c60d4a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":90848},{"type":"v1","addr":"172.21.15.174:6809","nonce":90848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":90848},{"type":"v1","addr":"172.21.15.174:6811","nonce":90848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":90848},{"type":"v1","addr":"172.21.15.174:6815","nonce":90848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":90848},{"type":"v1","addr":"172.21.15.174:6813","nonce":90848}]},"public_addr":"172.21.15.174:6809/90848","cluster_addr":"172.21.15.174:6811/90848","heartbeat_back_addr":"172.21.15.174:6815/90848","heartbeat_front_addr":"172.21.15.174:6813/90848","state":["exists","up"]},{"osd":6,"uuid":"496a6272-892e-43c4-a145-206623c22f99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":90877},{"type":"v1","addr":"172.21.15.174:6817","nonce":90877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":90877},{"type":"v1","addr":"172.21.15.174:6819","nonce":90877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":90877},{"type":"v1","addr":"172.21.15.174:6823","nonce":90877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":90877},{"type":"v1","addr":"172.21.15.174:6821","nonce":90877}]},"public_addr":"172.21.15.174:6817/90877","cluster_addr":"172.21.15.174:6819/90877","heartbeat_back_addr":"172.21.15.174:6823/90877","heartbeat_front_addr":"172.21.15.174:6821/90877","state":["exists","up"]},{"osd":7,"uuid":"49967e34-8b07-4bab-a13e-0c16d6a0f3a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":90878},{"type":"v1","addr":"172.21.15.174:6825","nonce":90878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":90878},{"type":"v1","addr":"172.21.15.174:6827","nonce":90878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":90878},{"type":"v1","addr":"172.21.15.174:6831","nonce":90878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":90878},{"type":"v1","addr":"172.21.15.174:6829","nonce":90878}]},"public_addr":"172.21.15.174:6825/90878","cluster_addr":"172.21.15.174:6827/90878","heartbeat_back_addr":"172.21.15.174:6831/90878","heartbeat_front_addr":"172.21.15.174:6829/90878","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.446479+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.514266+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.794108+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.810894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.731919+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.717756+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.737435+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.508682+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T05:07:10.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.473+0000 7f981e627700 1 -- 172.21.15.98:0/3778355509 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f9804066f30 msgr2=0x7f98040693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:10.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.473+0000 7f981e627700 1 --2- 172.21.15.98:0/3778355509 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f9804066f30 0x7f98040693e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9808000ef0 tx=0x7f98080016a0 comp rx=0 tx=0).stop 2023-10-06T05:07:10.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.473+0000 7f981e627700 1 -- 172.21.15.98:0/3778355509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98180b49a0 msgr2=0x7f9818159930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:10.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.473+0000 7f981e627700 1 --2- 172.21.15.98:0/3778355509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98180b49a0 0x7f9818159930 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f980c0093f0 tx=0x7f980c016000 comp rx=0 tx=0).stop 2023-10-06T05:07:10.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.473+0000 7f981e627700 1 -- 172.21.15.98:0/3778355509 shutdown_connections 2023-10-06T05:07:10.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.473+0000 7f981e627700 1 --2- 172.21.15.98:0/3778355509 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9818159560 0x7f98180ae550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:10.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.473+0000 7f981e627700 1 --2- 172.21.15.98:0/3778355509 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f98180b0760 0x7f981814a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:10.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.473+0000 7f981e627700 1 --2- 172.21.15.98:0/3778355509 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f9804066f30 0x7f98040693e0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:10.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.473+0000 7f981e627700 1 --2- 172.21.15.98:0/3778355509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98180b49a0 0x7f9818159930 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:10.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.473+0000 7f981e627700 1 -- 172.21.15.98:0/3778355509 >> 172.21.15.98:0/3778355509 conn(0x7f98180997f0 msgr2=0x7f9818158860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:10.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.473+0000 7f981e627700 1 -- 172.21.15.98:0/3778355509 shutdown_connections 2023-10-06T05:07:10.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.473+0000 7f981e627700 1 -- 172.21.15.98:0/3778355509 wait complete. 2023-10-06T05:07:10.488 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-06T05:07:10.488 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-06T05:07:10.603 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.601+0000 7f9e26845700 1 Processor -- start 2023-10-06T05:07:10.604 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.601+0000 7f9e26845700 1 -- start start 2023-10-06T05:07:10.606 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.602+0000 7f9e26845700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f9e20156280 0x7f9e2015a060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:10.606 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.602+0000 7f9e26845700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9e20156650 0x7f9e200abb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:10.606 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.602+0000 7f9e26845700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e200c9700 con 0x7f9e200685c0 2023-10-06T05:07:10.606 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.602+0000 7f9e26845700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e2015b000 con 0x7f9e20156650 2023-10-06T05:07:10.606 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.602+0000 7f9e26845700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e200ad810 con 0x7f9e20156280 2023-10-06T05:07:10.606 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.602+0000 7f9e1f7fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9e20156650 0x7f9e200abb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:10.606 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.602+0000 7f9e24de2700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f9e200685c0 0x7f9e200af940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:41108/0 (socket says 172.21.15.98:41108) 2023-10-06T05:07:10.606 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.602+0000 7f9e24de2700 1 -- 172.21.15.98:0/705452277 learned_addr learned my addr 172.21.15.98:0/705452277 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:07:10.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.602+0000 7f9e1ffff700 1 --2- 172.21.15.98:0/705452277 >> v2:172.21.15.174:3300/0 conn(0x7f9e20156280 0x7f9e2015a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:10.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.603+0000 7f9e1f7fe700 1 -- 172.21.15.98:0/705452277 >> v2:172.21.15.174:3300/0 conn(0x7f9e20156280 msgr2=0x7f9e2015a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:10.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.603+0000 7f9e1f7fe700 1 --2- 172.21.15.98:0/705452277 >> v2:172.21.15.174:3300/0 conn(0x7f9e20156280 0x7f9e2015a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:10.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.603+0000 7f9e1f7fe700 1 -- 172.21.15.98:0/705452277 >> v1:172.21.15.98:6789/0 conn(0x7f9e200685c0 legacy=0x7f9e200af940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:10.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.603+0000 7f9e1f7fe700 1 -- 172.21.15.98:0/705452277 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e2015a760 con 0x7f9e20156650 2023-10-06T05:07:10.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.603+0000 7f9e1effd700 1 -- 172.21.15.98:0/705452277 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3349193374 0 0) 0x7f9e200c9700 con 0x7f9e200685c0 2023-10-06T05:07:10.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.603+0000 7f9e1f7fe700 1 --2- 172.21.15.98:0/705452277 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e20156650 0x7f9e200abb10 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f9e20070b70 tx=0x7f9e140091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=329f00a27f4a0db2 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:10.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.603+0000 7f9e1effd700 1 -- 172.21.15.98:0/705452277 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e14011630 con 0x7f9e20156650 2023-10-06T05:07:10.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.603+0000 7f9e1effd700 1 -- 172.21.15.98:0/705452277 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e14011790 con 0x7f9e20156650 2023-10-06T05:07:10.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.603+0000 7f9e1effd700 1 -- 172.21.15.98:0/705452277 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e14011b30 con 0x7f9e20156650 2023-10-06T05:07:10.608 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.603+0000 7f9e26845700 1 -- 172.21.15.98:0/705452277 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e20156650 msgr2=0x7f9e200abb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:10.608 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.603+0000 7f9e26845700 1 --2- 172.21.15.98:0/705452277 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e20156650 0x7f9e200abb10 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f9e20070b70 tx=0x7f9e140091a0 comp rx=0 tx=0).stop 2023-10-06T05:07:10.608 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.603+0000 7f9e26845700 1 -- 172.21.15.98:0/705452277 shutdown_connections 2023-10-06T05:07:10.608 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.603+0000 7f9e26845700 1 --2- 172.21.15.98:0/705452277 >> v2:172.21.15.174:3300/0 conn(0x7f9e20156280 0x7f9e2015a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:10.608 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.603+0000 7f9e26845700 1 --2- 172.21.15.98:0/705452277 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e20156650 0x7f9e200abb10 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:10.608 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.603+0000 7f9e26845700 1 -- 172.21.15.98:0/705452277 >> 172.21.15.98:0/705452277 conn(0x7f9e2006de40 msgr2=0x7f9e2006e240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:10.608 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.604+0000 7f9e26845700 1 -- 172.21.15.98:0/705452277 shutdown_connections 2023-10-06T05:07:10.608 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.604+0000 7f9e26845700 1 -- 172.21.15.98:0/705452277 wait complete. 2023-10-06T05:07:10.611 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.605+0000 7f9e26845700 1 Processor -- start 2023-10-06T05:07:10.611 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.605+0000 7f9e26845700 1 -- start start 2023-10-06T05:07:10.611 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.605+0000 7f9e26845700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e200685c0 0x7f9e2014d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:10.611 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.605+0000 7f9e26845700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9e20156280 0x7f9e200ca910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:10.611 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.605+0000 7f9e26845700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9e20156650 0x7f9e200cbb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:10.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.605+0000 7f9e26845700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e2014db50 con 0x7f9e200685c0 2023-10-06T05:07:10.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.605+0000 7f9e26845700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e2014de30 con 0x7f9e20156280 2023-10-06T05:07:10.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.605+0000 7f9e26845700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e2014e110 con 0x7f9e20156650 2023-10-06T05:07:10.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.606+0000 7f9e1ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e200685c0 0x7f9e2014d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:10.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.606+0000 7f9e1ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e200685c0 0x7f9e2014d100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45640/0 (socket says 172.21.15.98:45640) 2023-10-06T05:07:10.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.606+0000 7f9e1ffff700 1 -- 172.21.15.98:0/1073208412 learned_addr learned my addr 172.21.15.98:0/1073208412 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:10.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.606+0000 7f9e1f7fe700 1 --2- 172.21.15.98:0/1073208412 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9e20156280 0x7f9e200ca910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:10.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.606+0000 7f9e24de2700 1 --2- 172.21.15.98:0/1073208412 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9e20156650 0x7f9e200cbb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:10.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.606+0000 7f9e1ffff700 1 -- 172.21.15.98:0/1073208412 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9e20156650 msgr2=0x7f9e200cbb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:10.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.606+0000 7f9e1ffff700 1 --2- 172.21.15.98:0/1073208412 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9e20156650 0x7f9e200cbb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:10.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.606+0000 7f9e1ffff700 1 -- 172.21.15.98:0/1073208412 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9e20156280 msgr2=0x7f9e200ca910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:10.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.606+0000 7f9e1ffff700 1 --2- 172.21.15.98:0/1073208412 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9e20156280 0x7f9e200ca910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:10.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.606+0000 7f9e1ffff700 1 -- 172.21.15.98:0/1073208412 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e14009420 con 0x7f9e200685c0 2023-10-06T05:07:10.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.606+0000 7f9e1f7fe700 1 --2- 172.21.15.98:0/1073208412 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9e20156280 0x7f9e200ca910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:07:10.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.606+0000 7f9e1ffff700 1 --2- 172.21.15.98:0/1073208412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e200685c0 0x7f9e2014d100 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9e1000ec80 tx=0x7f9e10009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:10.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.606+0000 7f9e07fff700 1 -- 172.21.15.98:0/1073208412 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e1000c9d0 con 0x7f9e200685c0 2023-10-06T05:07:10.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.606+0000 7f9e07fff700 1 -- 172.21.15.98:0/1073208412 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e1000cb30 con 0x7f9e200685c0 2023-10-06T05:07:10.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.606+0000 7f9e26845700 1 -- 172.21.15.98:0/1073208412 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e2014e3f0 con 0x7f9e200685c0 2023-10-06T05:07:10.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.606+0000 7f9e07fff700 1 -- 172.21.15.98:0/1073208412 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e10013470 con 0x7f9e200685c0 2023-10-06T05:07:10.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.606+0000 7f9e26845700 1 -- 172.21.15.98:0/1073208412 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e20167170 con 0x7f9e200685c0 2023-10-06T05:07:10.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.607+0000 7f9e26845700 1 -- 172.21.15.98:0/1073208412 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e2014ea40 con 0x7f9e200685c0 2023-10-06T05:07:10.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.608+0000 7f9e07fff700 1 -- 172.21.15.98:0/1073208412 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f9e10013860 con 0x7f9e200685c0 2023-10-06T05:07:10.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.608+0000 7f9e07fff700 1 --2- 172.21.15.98:0/1073208412 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f9e0c066f30 0x7f9e0c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:10.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.608+0000 7f9e07fff700 1 -- 172.21.15.98:0/1073208412 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f9e100850c0 con 0x7f9e200685c0 2023-10-06T05:07:10.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.608+0000 7f9e1f7fe700 1 --2- 172.21.15.98:0/1073208412 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f9e0c066f30 0x7f9e0c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:10.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.608+0000 7f9e1f7fe700 1 --2- 172.21.15.98:0/1073208412 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f9e0c066f30 0x7f9e0c0693e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9e140093f0 tx=0x7f9e14017000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:10.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.612+0000 7f9e07fff700 1 -- 172.21.15.98:0/1073208412 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9e10051ee0 con 0x7f9e200685c0 2023-10-06T05:07:10.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.794+0000 7f9e26845700 1 -- 172.21.15.98:0/1073208412 --> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9e20064a90 con 0x7f9e0c066f30 2023-10-06T05:07:10.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.805+0000 7f9e07fff700 1 -- 172.21.15.98:0/1073208412 <== mgr.4104 v2:172.21.15.98:6832/90397 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+367360 (secure 0 0 0) 0x7f9e20064a90 con 0x7f9e0c066f30 2023-10-06T05:07:10.811 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:07:10.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.808+0000 7f9e26845700 1 -- 172.21.15.98:0/1073208412 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f9e0c066f30 msgr2=0x7f9e0c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:10.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.808+0000 7f9e26845700 1 --2- 172.21.15.98:0/1073208412 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f9e0c066f30 0x7f9e0c0693e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9e140093f0 tx=0x7f9e14017000 comp rx=0 tx=0).stop 2023-10-06T05:07:10.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.808+0000 7f9e26845700 1 -- 172.21.15.98:0/1073208412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e200685c0 msgr2=0x7f9e2014d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:10.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.808+0000 7f9e26845700 1 --2- 172.21.15.98:0/1073208412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e200685c0 0x7f9e2014d100 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9e1000ec80 tx=0x7f9e10009120 comp rx=0 tx=0).stop 2023-10-06T05:07:10.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.808+0000 7f9e26845700 1 -- 172.21.15.98:0/1073208412 shutdown_connections 2023-10-06T05:07:10.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.808+0000 7f9e26845700 1 --2- 172.21.15.98:0/1073208412 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9e20156650 0x7f9e200cbb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:10.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.808+0000 7f9e26845700 1 --2- 172.21.15.98:0/1073208412 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f9e0c066f30 0x7f9e0c0693e0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:10.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.808+0000 7f9e26845700 1 --2- 172.21.15.98:0/1073208412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e200685c0 0x7f9e2014d100 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:10.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.808+0000 7f9e26845700 1 --2- 172.21.15.98:0/1073208412 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9e20156280 0x7f9e200ca910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:10.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.808+0000 7f9e26845700 1 -- 172.21.15.98:0/1073208412 >> 172.21.15.98:0/1073208412 conn(0x7f9e2006de40 msgr2=0x7f9e20151230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:10.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.808+0000 7f9e26845700 1 -- 172.21.15.98:0/1073208412 shutdown_connections 2023-10-06T05:07:10.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.808+0000 7f9e26845700 1 -- 172.21.15.98:0/1073208412 wait complete. 2023-10-06T05:07:10.815 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2023-10-06T05:07:10.822 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2023-10-06T05:07:10.522196+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594852,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":338,"acting":338,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":338,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2381928,"kb_used_data":3624,"kb_used_omap":0,"kb_used_meta":2378240,"kb_avail":752592792,"statfs":{"total":773094113280,"available":770655019008,"internally_reserved":0,"allocated":3710976,"data_stored":2196771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2435317760},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2178,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.780017"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.753513+0000","last_change":"2023-10-06T05:06:57.619460+0000","last_active":"2023-10-06T05:07:01.753513+0000","last_peered":"2023-10-06T05:07:01.753513+0000","last_clean":"2023-10-06T05:07:01.753513+0000","last_became_active":"2023-10-06T05:06:57.619199+0000","last_became_peered":"2023-10-06T05:06:57.619199+0000","last_unstale":"2023-10-06T05:07:01.753513+0000","last_undegraded":"2023-10-06T05:07:01.753513+0000","last_fullsized":"2023-10-06T05:07:01.753513+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:49:28.024174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.747458+0000","last_change":"2023-10-06T05:06:57.610280+0000","last_active":"2023-10-06T05:07:01.747458+0000","last_peered":"2023-10-06T05:07:01.747458+0000","last_clean":"2023-10-06T05:07:01.747458+0000","last_became_active":"2023-10-06T05:06:57.609973+0000","last_became_peered":"2023-10-06T05:06:57.609973+0000","last_unstale":"2023-10-06T05:07:01.747458+0000","last_undegraded":"2023-10-06T05:07:01.747458+0000","last_fullsized":"2023-10-06T05:07:01.747458+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:22:11.051575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.762104+0000","last_change":"2023-10-06T05:06:57.615468+0000","last_active":"2023-10-06T05:07:01.762104+0000","last_peered":"2023-10-06T05:07:01.762104+0000","last_clean":"2023-10-06T05:07:01.762104+0000","last_became_active":"2023-10-06T05:06:57.615221+0000","last_became_peered":"2023-10-06T05:06:57.615221+0000","last_unstale":"2023-10-06T05:07:01.762104+0000","last_undegraded":"2023-10-06T05:07:01.762104+0000","last_fullsized":"2023-10-06T05:07:01.762104+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:49:48.309158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.754474+0000","last_change":"2023-10-06T05:06:57.614470+0000","last_active":"2023-10-06T05:07:01.754474+0000","last_peered":"2023-10-06T05:07:01.754474+0000","last_clean":"2023-10-06T05:07:01.754474+0000","last_became_active":"2023-10-06T05:06:57.614169+0000","last_became_peered":"2023-10-06T05:06:57.614169+0000","last_unstale":"2023-10-06T05:07:01.754474+0000","last_undegraded":"2023-10-06T05:07:01.754474+0000","last_fullsized":"2023-10-06T05:07:01.754474+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:23:03.302621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750277+0000","last_change":"2023-10-06T05:06:57.622050+0000","last_active":"2023-10-06T05:07:01.750277+0000","last_peered":"2023-10-06T05:07:01.750277+0000","last_clean":"2023-10-06T05:07:01.750277+0000","last_became_active":"2023-10-06T05:06:57.621777+0000","last_became_peered":"2023-10-06T05:06:57.621777+0000","last_unstale":"2023-10-06T05:07:01.750277+0000","last_undegraded":"2023-10-06T05:07:01.750277+0000","last_fullsized":"2023-10-06T05:07:01.750277+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:04:40.577463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.761680+0000","last_change":"2023-10-06T05:06:57.611449+0000","last_active":"2023-10-06T05:07:01.761680+0000","last_peered":"2023-10-06T05:07:01.761680+0000","last_clean":"2023-10-06T05:07:01.761680+0000","last_became_active":"2023-10-06T05:06:57.610301+0000","last_became_peered":"2023-10-06T05:06:57.610301+0000","last_unstale":"2023-10-06T05:07:01.761680+0000","last_undegraded":"2023-10-06T05:07:01.761680+0000","last_fullsized":"2023-10-06T05:07:01.761680+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:43:21.654853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.755118+0000","last_change":"2023-10-06T05:06:57.610678+0000","last_active":"2023-10-06T05:07:01.755118+0000","last_peered":"2023-10-06T05:07:01.755118+0000","last_clean":"2023-10-06T05:07:01.755118+0000","last_became_active":"2023-10-06T05:06:57.609746+0000","last_became_peered":"2023-10-06T05:06:57.609746+0000","last_unstale":"2023-10-06T05:07:01.755118+0000","last_undegraded":"2023-10-06T05:07:01.755118+0000","last_fullsized":"2023-10-06T05:07:01.755118+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:11:00.525585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.745457+0000","last_change":"2023-10-06T05:06:57.612559+0000","last_active":"2023-10-06T05:07:01.745457+0000","last_peered":"2023-10-06T05:07:01.745457+0000","last_clean":"2023-10-06T05:07:01.745457+0000","last_became_active":"2023-10-06T05:06:57.612199+0000","last_became_peered":"2023-10-06T05:06:57.612199+0000","last_unstale":"2023-10-06T05:07:01.745457+0000","last_undegraded":"2023-10-06T05:07:01.745457+0000","last_fullsized":"2023-10-06T05:07:01.745457+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:01:11.983432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.751122+0000","last_change":"2023-10-06T05:06:57.618496+0000","last_active":"2023-10-06T05:07:01.751122+0000","last_peered":"2023-10-06T05:07:01.751122+0000","last_clean":"2023-10-06T05:07:01.751122+0000","last_became_active":"2023-10-06T05:06:57.618261+0000","last_became_peered":"2023-10-06T05:06:57.618261+0000","last_unstale":"2023-10-06T05:07:01.751122+0000","last_undegraded":"2023-10-06T05:07:01.751122+0000","last_fullsized":"2023-10-06T05:07:01.751122+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:24:33.077189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.747636+0000","last_change":"2023-10-06T05:06:57.614417+0000","last_active":"2023-10-06T05:07:01.747636+0000","last_peered":"2023-10-06T05:07:01.747636+0000","last_clean":"2023-10-06T05:07:01.747636+0000","last_became_active":"2023-10-06T05:06:57.614144+0000","last_became_peered":"2023-10-06T05:06:57.614144+0000","last_unstale":"2023-10-06T05:07:01.747636+0000","last_undegraded":"2023-10-06T05:07:01.747636+0000","last_fullsized":"2023-10-06T05:07:01.747636+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:32:33.731449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.762013+0000","last_change":"2023-10-06T05:06:57.610272+0000","last_active":"2023-10-06T05:07:01.762013+0000","last_peered":"2023-10-06T05:07:01.762013+0000","last_clean":"2023-10-06T05:07:01.762013+0000","last_became_active":"2023-10-06T05:06:57.609985+0000","last_became_peered":"2023-10-06T05:06:57.609985+0000","last_unstale":"2023-10-06T05:07:01.762013+0000","last_undegraded":"2023-10-06T05:07:01.762013+0000","last_fullsized":"2023-10-06T05:07:01.762013+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:20:04.673613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.745990+0000","last_change":"2023-10-06T05:06:57.606852+0000","last_active":"2023-10-06T05:07:01.745990+0000","last_peered":"2023-10-06T05:07:01.745990+0000","last_clean":"2023-10-06T05:07:01.745990+0000","last_became_active":"2023-10-06T05:06:57.606582+0000","last_became_peered":"2023-10-06T05:06:57.606582+0000","last_unstale":"2023-10-06T05:07:01.745990+0000","last_undegraded":"2023-10-06T05:07:01.745990+0000","last_fullsized":"2023-10-06T05:07:01.745990+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:23:03.302621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.753767+0000","last_change":"2023-10-06T05:06:57.615706+0000","last_active":"2023-10-06T05:07:01.753767+0000","last_peered":"2023-10-06T05:07:01.753767+0000","last_clean":"2023-10-06T05:07:01.753767+0000","last_became_active":"2023-10-06T05:06:57.615408+0000","last_became_peered":"2023-10-06T05:06:57.615408+0000","last_unstale":"2023-10-06T05:07:01.753767+0000","last_undegraded":"2023-10-06T05:07:01.753767+0000","last_fullsized":"2023-10-06T05:07:01.753767+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:38:51.879514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.744412+0000","last_change":"2023-10-06T05:06:57.613222+0000","last_active":"2023-10-06T05:07:01.744412+0000","last_peered":"2023-10-06T05:07:01.744412+0000","last_clean":"2023-10-06T05:07:01.744412+0000","last_became_active":"2023-10-06T05:06:57.612924+0000","last_became_peered":"2023-10-06T05:06:57.612924+0000","last_unstale":"2023-10-06T05:07:01.744412+0000","last_undegraded":"2023-10-06T05:07:01.744412+0000","last_fullsized":"2023-10-06T05:07:01.744412+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:29:19.951056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.754947+0000","last_change":"2023-10-06T05:06:57.610023+0000","last_active":"2023-10-06T05:07:01.754947+0000","last_peered":"2023-10-06T05:07:01.754947+0000","last_clean":"2023-10-06T05:07:01.754947+0000","last_became_active":"2023-10-06T05:06:57.609367+0000","last_became_peered":"2023-10-06T05:06:57.609367+0000","last_unstale":"2023-10-06T05:07:01.754947+0000","last_undegraded":"2023-10-06T05:07:01.754947+0000","last_fullsized":"2023-10-06T05:07:01.754947+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:34:55.987076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.758813+0000","last_change":"2023-10-06T05:06:57.610369+0000","last_active":"2023-10-06T05:07:01.758813+0000","last_peered":"2023-10-06T05:07:01.758813+0000","last_clean":"2023-10-06T05:07:01.758813+0000","last_became_active":"2023-10-06T05:06:57.610044+0000","last_became_peered":"2023-10-06T05:06:57.610044+0000","last_unstale":"2023-10-06T05:07:01.758813+0000","last_undegraded":"2023-10-06T05:07:01.758813+0000","last_fullsized":"2023-10-06T05:07:01.758813+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:10:12.061112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.751198+0000","last_change":"2023-10-06T05:06:57.619477+0000","last_active":"2023-10-06T05:07:01.751198+0000","last_peered":"2023-10-06T05:07:01.751198+0000","last_clean":"2023-10-06T05:07:01.751198+0000","last_became_active":"2023-10-06T05:06:57.618021+0000","last_became_peered":"2023-10-06T05:06:57.618021+0000","last_unstale":"2023-10-06T05:07:01.751198+0000","last_undegraded":"2023-10-06T05:07:01.751198+0000","last_fullsized":"2023-10-06T05:07:01.751198+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:58:02.572423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.761093+0000","last_change":"2023-10-06T05:06:57.602802+0000","last_active":"2023-10-06T05:07:01.761093+0000","last_peered":"2023-10-06T05:07:01.761093+0000","last_clean":"2023-10-06T05:07:01.761093+0000","last_became_active":"2023-10-06T05:06:57.602513+0000","last_became_peered":"2023-10-06T05:06:57.602513+0000","last_unstale":"2023-10-06T05:07:01.761093+0000","last_undegraded":"2023-10-06T05:07:01.761093+0000","last_fullsized":"2023-10-06T05:07:01.761093+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T16:34:15.990651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.747139+0000","last_change":"2023-10-06T05:06:57.608190+0000","last_active":"2023-10-06T05:07:01.747139+0000","last_peered":"2023-10-06T05:07:01.747139+0000","last_clean":"2023-10-06T05:07:01.747139+0000","last_became_active":"2023-10-06T05:06:57.607936+0000","last_became_peered":"2023-10-06T05:06:57.607936+0000","last_unstale":"2023-10-06T05:07:01.747139+0000","last_undegraded":"2023-10-06T05:07:01.747139+0000","last_fullsized":"2023-10-06T05:07:01.747139+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:00:30.035770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.761424+0000","last_change":"2023-10-06T05:06:57.611226+0000","last_active":"2023-10-06T05:07:01.761424+0000","last_peered":"2023-10-06T05:07:01.761424+0000","last_clean":"2023-10-06T05:07:01.761424+0000","last_became_active":"2023-10-06T05:06:57.610931+0000","last_became_peered":"2023-10-06T05:06:57.610931+0000","last_unstale":"2023-10-06T05:07:01.761424+0000","last_undegraded":"2023-10-06T05:07:01.761424+0000","last_fullsized":"2023-10-06T05:07:01.761424+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:52:14.772801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.749471+0000","last_change":"2023-10-06T05:06:57.615682+0000","last_active":"2023-10-06T05:07:01.749471+0000","last_peered":"2023-10-06T05:07:01.749471+0000","last_clean":"2023-10-06T05:07:01.749471+0000","last_became_active":"2023-10-06T05:06:57.615408+0000","last_became_peered":"2023-10-06T05:06:57.615408+0000","last_unstale":"2023-10-06T05:07:01.749471+0000","last_undegraded":"2023-10-06T05:07:01.749471+0000","last_fullsized":"2023-10-06T05:07:01.749471+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:58:02.572423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.749868+0000","last_change":"2023-10-06T05:06:57.619795+0000","last_active":"2023-10-06T05:07:01.749868+0000","last_peered":"2023-10-06T05:07:01.749868+0000","last_clean":"2023-10-06T05:07:01.749868+0000","last_became_active":"2023-10-06T05:06:57.619533+0000","last_became_peered":"2023-10-06T05:06:57.619533+0000","last_unstale":"2023-10-06T05:07:01.749868+0000","last_undegraded":"2023-10-06T05:07:01.749868+0000","last_fullsized":"2023-10-06T05:07:01.749868+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:11:00.525585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750332+0000","last_change":"2023-10-06T05:06:57.617926+0000","last_active":"2023-10-06T05:07:01.750332+0000","last_peered":"2023-10-06T05:07:01.750332+0000","last_clean":"2023-10-06T05:07:01.750332+0000","last_became_active":"2023-10-06T05:06:57.617128+0000","last_became_peered":"2023-10-06T05:06:57.617128+0000","last_unstale":"2023-10-06T05:07:01.750332+0000","last_undegraded":"2023-10-06T05:07:01.750332+0000","last_fullsized":"2023-10-06T05:07:01.750332+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:38:16.871694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.748479+0000","last_change":"2023-10-06T05:06:57.620090+0000","last_active":"2023-10-06T05:07:01.748479+0000","last_peered":"2023-10-06T05:07:01.748479+0000","last_clean":"2023-10-06T05:07:01.748479+0000","last_became_active":"2023-10-06T05:06:57.619892+0000","last_became_peered":"2023-10-06T05:06:57.619892+0000","last_unstale":"2023-10-06T05:07:01.748479+0000","last_undegraded":"2023-10-06T05:07:01.748479+0000","last_fullsized":"2023-10-06T05:07:01.748479+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:45:23.272110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750371+0000","last_change":"2023-10-06T05:06:57.613844+0000","last_active":"2023-10-06T05:07:01.750371+0000","last_peered":"2023-10-06T05:07:01.750371+0000","last_clean":"2023-10-06T05:07:01.750371+0000","last_became_active":"2023-10-06T05:06:57.613513+0000","last_became_peered":"2023-10-06T05:06:57.613513+0000","last_unstale":"2023-10-06T05:07:01.750371+0000","last_undegraded":"2023-10-06T05:07:01.750371+0000","last_fullsized":"2023-10-06T05:07:01.750371+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:53:03.171108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.748759+0000","last_change":"2023-10-06T05:06:57.612464+0000","last_active":"2023-10-06T05:07:01.748759+0000","last_peered":"2023-10-06T05:07:01.748759+0000","last_clean":"2023-10-06T05:07:01.748759+0000","last_became_active":"2023-10-06T05:06:57.612179+0000","last_became_peered":"2023-10-06T05:06:57.612179+0000","last_unstale":"2023-10-06T05:07:01.748759+0000","last_undegraded":"2023-10-06T05:07:01.748759+0000","last_fullsized":"2023-10-06T05:07:01.748759+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:12:34.550298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.744832+0000","last_change":"2023-10-06T05:06:57.612625+0000","last_active":"2023-10-06T05:07:01.744832+0000","last_peered":"2023-10-06T05:07:01.744832+0000","last_clean":"2023-10-06T05:07:01.744832+0000","last_became_active":"2023-10-06T05:06:57.611582+0000","last_became_peered":"2023-10-06T05:06:57.611582+0000","last_unstale":"2023-10-06T05:07:01.744832+0000","last_undegraded":"2023-10-06T05:07:01.744832+0000","last_fullsized":"2023-10-06T05:07:01.744832+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:59:39.994965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.768197+0000","last_change":"2023-10-06T05:07:01.765965+0000","last_active":"2023-10-06T05:07:01.768197+0000","last_peered":"2023-10-06T05:07:01.768197+0000","last_clean":"2023-10-06T05:07:01.768197+0000","last_became_active":"2023-10-06T05:07:01.764635+0000","last_became_peered":"2023-10-06T05:07:01.764635+0000","last_unstale":"2023-10-06T05:07:01.768197+0000","last_undegraded":"2023-10-06T05:07:01.768197+0000","last_fullsized":"2023-10-06T05:07:01.768197+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:18:43.042335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2,0],"acting":[4,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"24'1","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.760983+0000","last_change":"2023-10-06T05:06:56.716498+0000","last_active":"2023-10-06T05:07:01.760983+0000","last_peered":"2023-10-06T05:07:01.760983+0000","last_clean":"2023-10-06T05:07:01.760983+0000","last_became_active":"2023-10-06T05:06:56.714771+0000","last_became_peered":"2023-10-06T05:06:56.714771+0000","last_unstale":"2023-10-06T05:07:01.760983+0000","last_undegraded":"2023-10-06T05:07:01.760983+0000","last_fullsized":"2023-10-06T05:07:01.760983+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:53:56.989908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.752793+0000","last_change":"2023-10-06T05:06:57.617712+0000","last_active":"2023-10-06T05:07:01.752793+0000","last_peered":"2023-10-06T05:07:01.752793+0000","last_clean":"2023-10-06T05:07:01.752793+0000","last_became_active":"2023-10-06T05:06:57.616946+0000","last_became_peered":"2023-10-06T05:06:57.616946+0000","last_unstale":"2023-10-06T05:07:01.752793+0000","last_undegraded":"2023-10-06T05:07:01.752793+0000","last_fullsized":"2023-10-06T05:07:01.752793+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:21:13.192156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.767576+0000","last_change":"2023-10-06T05:07:01.763425+0000","last_active":"2023-10-06T05:07:01.767576+0000","last_peered":"2023-10-06T05:07:01.767576+0000","last_clean":"2023-10-06T05:07:01.767576+0000","last_became_active":"2023-10-06T05:07:01.763035+0000","last_became_peered":"2023-10-06T05:07:01.763035+0000","last_unstale":"2023-10-06T05:07:01.767576+0000","last_undegraded":"2023-10-06T05:07:01.767576+0000","last_fullsized":"2023-10-06T05:07:01.767576+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:51:19.615655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3,7],"acting":[6,4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.745972+0000","last_change":"2023-10-06T05:06:56.714205+0000","last_active":"2023-10-06T05:07:01.745972+0000","last_peered":"2023-10-06T05:07:01.745972+0000","last_clean":"2023-10-06T05:07:01.745972+0000","last_became_active":"2023-10-06T05:06:56.713947+0000","last_became_peered":"2023-10-06T05:06:56.713947+0000","last_unstale":"2023-10-06T05:07:01.745972+0000","last_undegraded":"2023-10-06T05:07:01.745972+0000","last_fullsized":"2023-10-06T05:07:01.745972+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:54:07.584005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750046+0000","last_change":"2023-10-06T05:06:57.616019+0000","last_active":"2023-10-06T05:07:01.750046+0000","last_peered":"2023-10-06T05:07:01.750046+0000","last_clean":"2023-10-06T05:07:01.750046+0000","last_became_active":"2023-10-06T05:06:57.615747+0000","last_became_peered":"2023-10-06T05:06:57.615747+0000","last_unstale":"2023-10-06T05:07:01.750046+0000","last_undegraded":"2023-10-06T05:07:01.750046+0000","last_fullsized":"2023-10-06T05:07:01.750046+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:07:52.020683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.770991+0000","last_change":"2023-10-06T05:07:01.767383+0000","last_active":"2023-10-06T05:07:01.770991+0000","last_peered":"2023-10-06T05:07:01.770991+0000","last_clean":"2023-10-06T05:07:01.770991+0000","last_became_active":"2023-10-06T05:07:01.766997+0000","last_became_peered":"2023-10-06T05:07:01.766997+0000","last_unstale":"2023-10-06T05:07:01.770991+0000","last_undegraded":"2023-10-06T05:07:01.770991+0000","last_fullsized":"2023-10-06T05:07:01.770991+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:45:03.209108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1,2],"acting":[0,7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.749601+0000","last_change":"2023-10-06T05:06:56.706133+0000","last_active":"2023-10-06T05:07:01.749601+0000","last_peered":"2023-10-06T05:07:01.749601+0000","last_clean":"2023-10-06T05:07:01.749601+0000","last_became_active":"2023-10-06T05:06:56.705867+0000","last_became_peered":"2023-10-06T05:06:56.705867+0000","last_unstale":"2023-10-06T05:07:01.749601+0000","last_undegraded":"2023-10-06T05:07:01.749601+0000","last_fullsized":"2023-10-06T05:07:01.749601+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:55:08.124129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.751243+0000","last_change":"2023-10-06T05:06:57.608813+0000","last_active":"2023-10-06T05:07:01.751243+0000","last_peered":"2023-10-06T05:07:01.751243+0000","last_clean":"2023-10-06T05:07:01.751243+0000","last_became_active":"2023-10-06T05:06:57.608497+0000","last_became_peered":"2023-10-06T05:06:57.608497+0000","last_unstale":"2023-10-06T05:07:01.751243+0000","last_undegraded":"2023-10-06T05:07:01.751243+0000","last_fullsized":"2023-10-06T05:07:01.751243+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:30:19.509164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.770179+0000","last_change":"2023-10-06T05:07:01.765801+0000","last_active":"2023-10-06T05:07:01.770179+0000","last_peered":"2023-10-06T05:07:01.770179+0000","last_clean":"2023-10-06T05:07:01.770179+0000","last_became_active":"2023-10-06T05:07:01.765422+0000","last_became_peered":"2023-10-06T05:07:01.765422+0000","last_unstale":"2023-10-06T05:07:01.770179+0000","last_undegraded":"2023-10-06T05:07:01.770179+0000","last_fullsized":"2023-10-06T05:07:01.770179+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:20:08.719247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7,1],"acting":[6,4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.751276+0000","last_change":"2023-10-06T05:06:56.714826+0000","last_active":"2023-10-06T05:07:01.751276+0000","last_peered":"2023-10-06T05:07:01.751276+0000","last_clean":"2023-10-06T05:07:01.751276+0000","last_became_active":"2023-10-06T05:06:56.714564+0000","last_became_peered":"2023-10-06T05:06:56.714564+0000","last_unstale":"2023-10-06T05:07:01.751276+0000","last_undegraded":"2023-10-06T05:07:01.751276+0000","last_fullsized":"2023-10-06T05:07:01.751276+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:19:13.930707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.749520+0000","last_change":"2023-10-06T05:06:57.618934+0000","last_active":"2023-10-06T05:07:01.749520+0000","last_peered":"2023-10-06T05:07:01.749520+0000","last_clean":"2023-10-06T05:07:01.749520+0000","last_became_active":"2023-10-06T05:06:57.618410+0000","last_became_peered":"2023-10-06T05:06:57.618410+0000","last_unstale":"2023-10-06T05:07:01.749520+0000","last_undegraded":"2023-10-06T05:07:01.749520+0000","last_fullsized":"2023-10-06T05:07:01.749520+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:51:15.570020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.775608+0000","last_change":"2023-10-06T05:07:01.766370+0000","last_active":"2023-10-06T05:07:01.775608+0000","last_peered":"2023-10-06T05:07:01.775608+0000","last_clean":"2023-10-06T05:07:01.775608+0000","last_became_active":"2023-10-06T05:07:01.765994+0000","last_became_peered":"2023-10-06T05:07:01.765994+0000","last_unstale":"2023-10-06T05:07:01.775608+0000","last_undegraded":"2023-10-06T05:07:01.775608+0000","last_fullsized":"2023-10-06T05:07:01.775608+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:20:46.564009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6,0],"acting":[7,4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.747549+0000","last_change":"2023-10-06T05:06:56.708479+0000","last_active":"2023-10-06T05:07:01.747549+0000","last_peered":"2023-10-06T05:07:01.747549+0000","last_clean":"2023-10-06T05:07:01.747549+0000","last_became_active":"2023-10-06T05:06:56.708155+0000","last_became_peered":"2023-10-06T05:06:56.708155+0000","last_unstale":"2023-10-06T05:07:01.747549+0000","last_undegraded":"2023-10-06T05:07:01.747549+0000","last_fullsized":"2023-10-06T05:07:01.747549+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:30:18.507322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750417+0000","last_change":"2023-10-06T05:06:57.617995+0000","last_active":"2023-10-06T05:07:01.750417+0000","last_peered":"2023-10-06T05:07:01.750417+0000","last_clean":"2023-10-06T05:07:01.750417+0000","last_became_active":"2023-10-06T05:06:57.617577+0000","last_became_peered":"2023-10-06T05:06:57.617577+0000","last_unstale":"2023-10-06T05:07:01.750417+0000","last_undegraded":"2023-10-06T05:07:01.750417+0000","last_fullsized":"2023-10-06T05:07:01.750417+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:01:11.983432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.763161+0000","last_change":"2023-10-06T05:07:01.761436+0000","last_active":"2023-10-06T05:07:01.763161+0000","last_peered":"2023-10-06T05:07:01.763161+0000","last_clean":"2023-10-06T05:07:01.763161+0000","last_became_active":"2023-10-06T05:07:01.761049+0000","last_became_peered":"2023-10-06T05:07:01.761049+0000","last_unstale":"2023-10-06T05:07:01.763161+0000","last_undegraded":"2023-10-06T05:07:01.763161+0000","last_fullsized":"2023-10-06T05:07:01.763161+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:09:13.387478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,5],"acting":[3,0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"24'3","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.753446+0000","last_change":"2023-10-06T05:06:56.717320+0000","last_active":"2023-10-06T05:07:01.753446+0000","last_peered":"2023-10-06T05:07:01.753446+0000","last_clean":"2023-10-06T05:07:01.753446+0000","last_became_active":"2023-10-06T05:06:56.716985+0000","last_became_peered":"2023-10-06T05:06:56.716985+0000","last_unstale":"2023-10-06T05:07:01.753446+0000","last_undegraded":"2023-10-06T05:07:01.753446+0000","last_fullsized":"2023-10-06T05:07:01.753446+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:53:02.169267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.747499+0000","last_change":"2023-10-06T05:06:57.613655+0000","last_active":"2023-10-06T05:07:01.747499+0000","last_peered":"2023-10-06T05:07:01.747499+0000","last_clean":"2023-10-06T05:07:01.747499+0000","last_became_active":"2023-10-06T05:06:57.613378+0000","last_became_peered":"2023-10-06T05:06:57.613378+0000","last_unstale":"2023-10-06T05:07:01.747499+0000","last_undegraded":"2023-10-06T05:07:01.747499+0000","last_fullsized":"2023-10-06T05:07:01.747499+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:23:03.919621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.772389+0000","last_change":"2023-10-06T05:07:01.766400+0000","last_active":"2023-10-06T05:07:01.772389+0000","last_peered":"2023-10-06T05:07:01.772389+0000","last_clean":"2023-10-06T05:07:01.772389+0000","last_became_active":"2023-10-06T05:07:01.766034+0000","last_became_peered":"2023-10-06T05:07:01.766034+0000","last_unstale":"2023-10-06T05:07:01.772389+0000","last_undegraded":"2023-10-06T05:07:01.772389+0000","last_fullsized":"2023-10-06T05:07:01.772389+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:04:14.761320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3,7],"acting":[1,5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"29'9","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-06T05:07:07.500759+0000","last_change":"2023-10-06T05:06:56.716406+0000","last_active":"2023-10-06T05:07:07.500759+0000","last_peered":"2023-10-06T05:07:07.500759+0000","last_clean":"2023-10-06T05:07:07.500759+0000","last_became_active":"2023-10-06T05:06:56.716075+0000","last_became_peered":"2023-10-06T05:06:56.716075+0000","last_unstale":"2023-10-06T05:07:07.500759+0000","last_undegraded":"2023-10-06T05:07:07.500759+0000","last_fullsized":"2023-10-06T05:07:07.500759+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:36:39.323759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3714,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.761355+0000","last_change":"2023-10-06T05:06:57.605798+0000","last_active":"2023-10-06T05:07:01.761355+0000","last_peered":"2023-10-06T05:07:01.761355+0000","last_clean":"2023-10-06T05:07:01.761355+0000","last_became_active":"2023-10-06T05:06:57.605523+0000","last_became_peered":"2023-10-06T05:06:57.605523+0000","last_unstale":"2023-10-06T05:07:01.761355+0000","last_undegraded":"2023-10-06T05:07:01.761355+0000","last_fullsized":"2023-10-06T05:07:01.761355+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:32:33.731449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.765132+0000","last_change":"2023-10-06T05:07:01.760576+0000","last_active":"2023-10-06T05:07:01.765132+0000","last_peered":"2023-10-06T05:07:01.765132+0000","last_clean":"2023-10-06T05:07:01.765132+0000","last_became_active":"2023-10-06T05:07:01.760146+0000","last_became_peered":"2023-10-06T05:07:01.760146+0000","last_unstale":"2023-10-06T05:07:01.765132+0000","last_undegraded":"2023-10-06T05:07:01.765132+0000","last_fullsized":"2023-10-06T05:07:01.765132+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:30:21.024442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3,0],"acting":[5,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"24'2","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.748025+0000","last_change":"2023-10-06T05:06:56.722076+0000","last_active":"2023-10-06T05:07:01.748025+0000","last_peered":"2023-10-06T05:07:01.748025+0000","last_clean":"2023-10-06T05:07:01.748025+0000","last_became_active":"2023-10-06T05:06:56.721786+0000","last_became_peered":"2023-10-06T05:06:56.721786+0000","last_unstale":"2023-10-06T05:07:01.748025+0000","last_undegraded":"2023-10-06T05:07:01.748025+0000","last_fullsized":"2023-10-06T05:07:01.748025+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:15:54.616807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.749669+0000","last_change":"2023-10-06T05:06:57.620208+0000","last_active":"2023-10-06T05:07:01.749669+0000","last_peered":"2023-10-06T05:07:01.749669+0000","last_clean":"2023-10-06T05:07:01.749669+0000","last_became_active":"2023-10-06T05:06:57.619747+0000","last_became_peered":"2023-10-06T05:06:57.619747+0000","last_unstale":"2023-10-06T05:07:01.749669+0000","last_undegraded":"2023-10-06T05:07:01.749669+0000","last_fullsized":"2023-10-06T05:07:01.749669+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:52:14.772801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.764815+0000","last_change":"2023-10-06T05:07:01.762230+0000","last_active":"2023-10-06T05:07:01.764815+0000","last_peered":"2023-10-06T05:07:01.764815+0000","last_clean":"2023-10-06T05:07:01.764815+0000","last_became_active":"2023-10-06T05:07:01.761878+0000","last_became_peered":"2023-10-06T05:07:01.761878+0000","last_unstale":"2023-10-06T05:07:01.764815+0000","last_undegraded":"2023-10-06T05:07:01.764815+0000","last_fullsized":"2023-10-06T05:07:01.764815+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:18:24.586582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2,1],"acting":[3,7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.749791+0000","last_change":"2023-10-06T05:06:56.715016+0000","last_active":"2023-10-06T05:07:01.749791+0000","last_peered":"2023-10-06T05:07:01.749791+0000","last_clean":"2023-10-06T05:07:01.749791+0000","last_became_active":"2023-10-06T05:06:56.714513+0000","last_became_peered":"2023-10-06T05:06:56.714513+0000","last_unstale":"2023-10-06T05:07:01.749791+0000","last_undegraded":"2023-10-06T05:07:01.749791+0000","last_fullsized":"2023-10-06T05:07:01.749791+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T16:11:31.404755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.744725+0000","last_change":"2023-10-06T05:06:57.613062+0000","last_active":"2023-10-06T05:07:01.744725+0000","last_peered":"2023-10-06T05:07:01.744725+0000","last_clean":"2023-10-06T05:07:01.744725+0000","last_became_active":"2023-10-06T05:06:57.611524+0000","last_became_peered":"2023-10-06T05:06:57.611524+0000","last_unstale":"2023-10-06T05:07:01.744725+0000","last_undegraded":"2023-10-06T05:07:01.744725+0000","last_fullsized":"2023-10-06T05:07:01.744725+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:01:11.480811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.770644+0000","last_change":"2023-10-06T05:07:01.764987+0000","last_active":"2023-10-06T05:07:01.770644+0000","last_peered":"2023-10-06T05:07:01.770644+0000","last_clean":"2023-10-06T05:07:01.770644+0000","last_became_active":"2023-10-06T05:07:01.764475+0000","last_became_peered":"2023-10-06T05:07:01.764475+0000","last_unstale":"2023-10-06T05:07:01.770644+0000","last_undegraded":"2023-10-06T05:07:01.770644+0000","last_fullsized":"2023-10-06T05:07:01.770644+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:25:41.281713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,0],"acting":[3,1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.755037+0000","last_change":"2023-10-06T05:06:56.719174+0000","last_active":"2023-10-06T05:07:01.755037+0000","last_peered":"2023-10-06T05:07:01.755037+0000","last_clean":"2023-10-06T05:07:01.755037+0000","last_became_active":"2023-10-06T05:06:56.718930+0000","last_became_peered":"2023-10-06T05:06:56.718930+0000","last_unstale":"2023-10-06T05:07:01.755037+0000","last_undegraded":"2023-10-06T05:07:01.755037+0000","last_fullsized":"2023-10-06T05:07:01.755037+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:51:30.871635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750114+0000","last_change":"2023-10-06T05:06:57.617389+0000","last_active":"2023-10-06T05:07:01.750114+0000","last_peered":"2023-10-06T05:07:01.750114+0000","last_clean":"2023-10-06T05:07:01.750114+0000","last_became_active":"2023-10-06T05:06:57.617099+0000","last_became_peered":"2023-10-06T05:06:57.617099+0000","last_unstale":"2023-10-06T05:07:01.750114+0000","last_undegraded":"2023-10-06T05:07:01.750114+0000","last_fullsized":"2023-10-06T05:07:01.750114+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:51:15.929322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.759618+0000","last_change":"2023-10-06T05:07:01.756721+0000","last_active":"2023-10-06T05:07:01.759618+0000","last_peered":"2023-10-06T05:07:01.759618+0000","last_clean":"2023-10-06T05:07:01.759618+0000","last_became_active":"2023-10-06T05:07:01.756335+0000","last_became_peered":"2023-10-06T05:07:01.756335+0000","last_unstale":"2023-10-06T05:07:01.759618+0000","last_undegraded":"2023-10-06T05:07:01.759618+0000","last_fullsized":"2023-10-06T05:07:01.759618+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:20:29.759167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1,6],"acting":[5,3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"24'8","reported_seq":22,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750615+0000","last_change":"2023-10-06T05:06:56.713625+0000","last_active":"2023-10-06T05:07:01.750615+0000","last_peered":"2023-10-06T05:07:01.750615+0000","last_clean":"2023-10-06T05:07:01.750615+0000","last_became_active":"2023-10-06T05:06:56.713346+0000","last_became_peered":"2023-10-06T05:06:56.713346+0000","last_unstale":"2023-10-06T05:07:01.750615+0000","last_undegraded":"2023-10-06T05:07:01.750615+0000","last_fullsized":"2023-10-06T05:07:01.750615+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:57:59.037083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750303+0000","last_change":"2023-10-06T05:06:57.617670+0000","last_active":"2023-10-06T05:07:01.750303+0000","last_peered":"2023-10-06T05:07:01.750303+0000","last_clean":"2023-10-06T05:07:01.750303+0000","last_became_active":"2023-10-06T05:06:57.617259+0000","last_became_peered":"2023-10-06T05:06:57.617259+0000","last_unstale":"2023-10-06T05:07:01.750303+0000","last_undegraded":"2023-10-06T05:07:01.750303+0000","last_fullsized":"2023-10-06T05:07:01.750303+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:22:11.051575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.775868+0000","last_change":"2023-10-06T05:07:01.771047+0000","last_active":"2023-10-06T05:07:01.775868+0000","last_peered":"2023-10-06T05:07:01.775868+0000","last_clean":"2023-10-06T05:07:01.775868+0000","last_became_active":"2023-10-06T05:07:01.770692+0000","last_became_peered":"2023-10-06T05:07:01.770692+0000","last_unstale":"2023-10-06T05:07:01.775868+0000","last_undegraded":"2023-10-06T05:07:01.775868+0000","last_fullsized":"2023-10-06T05:07:01.775868+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:36:04.280555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3,4],"acting":[7,6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"24'1","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.752603+0000","last_change":"2023-10-06T05:06:56.722082+0000","last_active":"2023-10-06T05:07:01.752603+0000","last_peered":"2023-10-06T05:07:01.752603+0000","last_clean":"2023-10-06T05:07:01.752603+0000","last_became_active":"2023-10-06T05:06:56.721758+0000","last_became_peered":"2023-10-06T05:06:56.721758+0000","last_unstale":"2023-10-06T05:07:01.752603+0000","last_undegraded":"2023-10-06T05:07:01.752603+0000","last_fullsized":"2023-10-06T05:07:01.752603+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:10:26.413910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.745354+0000","last_change":"2023-10-06T05:06:57.611665+0000","last_active":"2023-10-06T05:07:01.745354+0000","last_peered":"2023-10-06T05:07:01.745354+0000","last_clean":"2023-10-06T05:07:01.745354+0000","last_became_active":"2023-10-06T05:06:57.611226+0000","last_became_peered":"2023-10-06T05:06:57.611226+0000","last_unstale":"2023-10-06T05:07:01.745354+0000","last_undegraded":"2023-10-06T05:07:01.745354+0000","last_fullsized":"2023-10-06T05:07:01.745354+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:26:29.348498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.761343+0000","last_change":"2023-10-06T05:07:01.760161+0000","last_active":"2023-10-06T05:07:01.761343+0000","last_peered":"2023-10-06T05:07:01.761343+0000","last_clean":"2023-10-06T05:07:01.761343+0000","last_became_active":"2023-10-06T05:07:01.759742+0000","last_became_peered":"2023-10-06T05:07:01.759742+0000","last_unstale":"2023-10-06T05:07:01.761343+0000","last_undegraded":"2023-10-06T05:07:01.761343+0000","last_fullsized":"2023-10-06T05:07:01.761343+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:10:16.106747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1,5],"acting":[2,0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"24'2","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.751770+0000","last_change":"2023-10-06T05:06:56.722004+0000","last_active":"2023-10-06T05:07:01.751770+0000","last_peered":"2023-10-06T05:07:01.751770+0000","last_clean":"2023-10-06T05:07:01.751770+0000","last_became_active":"2023-10-06T05:06:56.721717+0000","last_became_peered":"2023-10-06T05:06:56.721717+0000","last_unstale":"2023-10-06T05:07:01.751770+0000","last_undegraded":"2023-10-06T05:07:01.751770+0000","last_fullsized":"2023-10-06T05:07:01.751770+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:50:35.786032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.761552+0000","last_change":"2023-10-06T05:06:57.609279+0000","last_active":"2023-10-06T05:07:01.761552+0000","last_peered":"2023-10-06T05:07:01.761552+0000","last_clean":"2023-10-06T05:07:01.761552+0000","last_became_active":"2023-10-06T05:06:57.608982+0000","last_became_peered":"2023-10-06T05:06:57.608982+0000","last_unstale":"2023-10-06T05:07:01.761552+0000","last_undegraded":"2023-10-06T05:07:01.761552+0000","last_fullsized":"2023-10-06T05:07:01.761552+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:38:16.871694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.765310+0000","last_change":"2023-10-06T05:07:01.762960+0000","last_active":"2023-10-06T05:07:01.765310+0000","last_peered":"2023-10-06T05:07:01.765310+0000","last_clean":"2023-10-06T05:07:01.765310+0000","last_became_active":"2023-10-06T05:07:01.762308+0000","last_became_peered":"2023-10-06T05:07:01.762308+0000","last_unstale":"2023-10-06T05:07:01.765310+0000","last_undegraded":"2023-10-06T05:07:01.765310+0000","last_fullsized":"2023-10-06T05:07:01.765310+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:49:52.354792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5,6],"acting":[2,0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"24'1","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.754352+0000","last_change":"2023-10-06T05:06:56.710907+0000","last_active":"2023-10-06T05:07:01.754352+0000","last_peered":"2023-10-06T05:07:01.754352+0000","last_clean":"2023-10-06T05:07:01.754352+0000","last_became_active":"2023-10-06T05:06:56.710595+0000","last_became_peered":"2023-10-06T05:06:56.710595+0000","last_unstale":"2023-10-06T05:07:01.754352+0000","last_undegraded":"2023-10-06T05:07:01.754352+0000","last_fullsized":"2023-10-06T05:07:01.754352+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:17:31.213115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.751968+0000","last_change":"2023-10-06T05:06:57.611226+0000","last_active":"2023-10-06T05:07:01.751968+0000","last_peered":"2023-10-06T05:07:01.751968+0000","last_clean":"2023-10-06T05:07:01.751968+0000","last_became_active":"2023-10-06T05:06:57.610966+0000","last_became_peered":"2023-10-06T05:06:57.610966+0000","last_unstale":"2023-10-06T05:07:01.751968+0000","last_undegraded":"2023-10-06T05:07:01.751968+0000","last_fullsized":"2023-10-06T05:07:01.751968+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:15:48.746279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.768101+0000","last_change":"2023-10-06T05:07:01.764757+0000","last_active":"2023-10-06T05:07:01.768101+0000","last_peered":"2023-10-06T05:07:01.768101+0000","last_clean":"2023-10-06T05:07:01.768101+0000","last_became_active":"2023-10-06T05:07:01.762819+0000","last_became_peered":"2023-10-06T05:07:01.762819+0000","last_unstale":"2023-10-06T05:07:01.768101+0000","last_undegraded":"2023-10-06T05:07:01.768101+0000","last_fullsized":"2023-10-06T05:07:01.768101+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:21:17.237790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3,6],"acting":[2,4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.745471+0000","last_change":"2023-10-06T05:06:56.704818+0000","last_active":"2023-10-06T05:07:01.745471+0000","last_peered":"2023-10-06T05:07:01.745471+0000","last_clean":"2023-10-06T05:07:01.745471+0000","last_became_active":"2023-10-06T05:06:56.704596+0000","last_became_peered":"2023-10-06T05:06:56.704596+0000","last_unstale":"2023-10-06T05:07:01.745471+0000","last_undegraded":"2023-10-06T05:07:01.745471+0000","last_fullsized":"2023-10-06T05:07:01.745471+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:23:02.300780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750614+0000","last_change":"2023-10-06T05:06:57.615733+0000","last_active":"2023-10-06T05:07:01.750614+0000","last_peered":"2023-10-06T05:07:01.750614+0000","last_clean":"2023-10-06T05:07:01.750614+0000","last_became_active":"2023-10-06T05:06:57.615061+0000","last_became_peered":"2023-10-06T05:06:57.615061+0000","last_unstale":"2023-10-06T05:07:01.750614+0000","last_undegraded":"2023-10-06T05:07:01.750614+0000","last_fullsized":"2023-10-06T05:07:01.750614+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:45:23.272110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.768073+0000","last_change":"2023-10-06T05:07:01.763754+0000","last_active":"2023-10-06T05:07:01.768073+0000","last_peered":"2023-10-06T05:07:01.768073+0000","last_clean":"2023-10-06T05:07:01.768073+0000","last_became_active":"2023-10-06T05:07:01.763392+0000","last_became_peered":"2023-10-06T05:07:01.763392+0000","last_unstale":"2023-10-06T05:07:01.768073+0000","last_undegraded":"2023-10-06T05:07:01.768073+0000","last_fullsized":"2023-10-06T05:07:01.768073+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:38:20.917328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5,6],"acting":[2,7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"24'2","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750935+0000","last_change":"2023-10-06T05:06:56.717936+0000","last_active":"2023-10-06T05:07:01.750935+0000","last_peered":"2023-10-06T05:07:01.750935+0000","last_clean":"2023-10-06T05:07:01.750935+0000","last_became_active":"2023-10-06T05:06:56.717683+0000","last_became_peered":"2023-10-06T05:06:56.717683+0000","last_unstale":"2023-10-06T05:07:01.750935+0000","last_undegraded":"2023-10-06T05:07:01.750935+0000","last_fullsized":"2023-10-06T05:07:01.750935+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:23:02.300780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.760808+0000","last_change":"2023-10-06T05:06:57.617318+0000","last_active":"2023-10-06T05:07:01.760808+0000","last_peered":"2023-10-06T05:07:01.760808+0000","last_clean":"2023-10-06T05:07:01.760808+0000","last_became_active":"2023-10-06T05:06:57.617044+0000","last_became_peered":"2023-10-06T05:06:57.617044+0000","last_unstale":"2023-10-06T05:07:01.760808+0000","last_undegraded":"2023-10-06T05:07:01.760808+0000","last_fullsized":"2023-10-06T05:07:01.760808+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:51:15.570020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.772045+0000","last_change":"2023-10-06T05:07:01.762386+0000","last_active":"2023-10-06T05:07:01.772045+0000","last_peered":"2023-10-06T05:07:01.772045+0000","last_clean":"2023-10-06T05:07:01.772045+0000","last_became_active":"2023-10-06T05:07:01.761040+0000","last_became_peered":"2023-10-06T05:07:01.761040+0000","last_unstale":"2023-10-06T05:07:01.772045+0000","last_undegraded":"2023-10-06T05:07:01.772045+0000","last_fullsized":"2023-10-06T05:07:01.772045+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:11:22.468821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0,6],"acting":[1,4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.752055+0000","last_change":"2023-10-06T05:06:56.723332+0000","last_active":"2023-10-06T05:07:01.752055+0000","last_peered":"2023-10-06T05:07:01.752055+0000","last_clean":"2023-10-06T05:07:01.752055+0000","last_became_active":"2023-10-06T05:06:56.723023+0000","last_became_peered":"2023-10-06T05:06:56.723023+0000","last_unstale":"2023-10-06T05:07:01.752055+0000","last_undegraded":"2023-10-06T05:07:01.752055+0000","last_fullsized":"2023-10-06T05:07:01.752055+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:36:39.323759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.761806+0000","last_change":"2023-10-06T05:06:57.609386+0000","last_active":"2023-10-06T05:07:01.761806+0000","last_peered":"2023-10-06T05:07:01.761806+0000","last_clean":"2023-10-06T05:07:01.761806+0000","last_became_active":"2023-10-06T05:06:57.609019+0000","last_became_peered":"2023-10-06T05:06:57.609019+0000","last_unstale":"2023-10-06T05:07:01.761806+0000","last_undegraded":"2023-10-06T05:07:01.761806+0000","last_fullsized":"2023-10-06T05:07:01.761806+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:20:18.722541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.767691+0000","last_change":"2023-10-06T05:07:01.763186+0000","last_active":"2023-10-06T05:07:01.767691+0000","last_peered":"2023-10-06T05:07:01.767691+0000","last_clean":"2023-10-06T05:07:01.767691+0000","last_became_active":"2023-10-06T05:07:01.762259+0000","last_became_peered":"2023-10-06T05:07:01.762259+0000","last_unstale":"2023-10-06T05:07:01.767691+0000","last_undegraded":"2023-10-06T05:07:01.767691+0000","last_fullsized":"2023-10-06T05:07:01.767691+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:46:13.640987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6,2],"acting":[5,4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.761404+0000","last_change":"2023-10-06T05:06:56.722507+0000","last_active":"2023-10-06T05:07:01.761404+0000","last_peered":"2023-10-06T05:07:01.761404+0000","last_clean":"2023-10-06T05:07:01.761404+0000","last_became_active":"2023-10-06T05:06:56.722243+0000","last_became_peered":"2023-10-06T05:06:56.722243+0000","last_unstale":"2023-10-06T05:07:01.761404+0000","last_undegraded":"2023-10-06T05:07:01.761404+0000","last_fullsized":"2023-10-06T05:07:01.761404+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:57:59.037083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.761170+0000","last_change":"2023-10-06T05:06:57.606979+0000","last_active":"2023-10-06T05:07:01.761170+0000","last_peered":"2023-10-06T05:07:01.761170+0000","last_clean":"2023-10-06T05:07:01.761170+0000","last_became_active":"2023-10-06T05:06:57.606633+0000","last_became_peered":"2023-10-06T05:06:57.606633+0000","last_unstale":"2023-10-06T05:07:01.761170+0000","last_undegraded":"2023-10-06T05:07:01.761170+0000","last_fullsized":"2023-10-06T05:07:01.761170+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:21:13.192156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.768126+0000","last_change":"2023-10-06T05:07:01.765104+0000","last_active":"2023-10-06T05:07:01.768126+0000","last_peered":"2023-10-06T05:07:01.768126+0000","last_clean":"2023-10-06T05:07:01.768126+0000","last_became_active":"2023-10-06T05:07:01.764692+0000","last_became_peered":"2023-10-06T05:07:01.764692+0000","last_unstale":"2023-10-06T05:07:01.768126+0000","last_undegraded":"2023-10-06T05:07:01.768126+0000","last_fullsized":"2023-10-06T05:07:01.768126+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:40:47.847563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0,2],"acting":[4,5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.749439+0000","last_change":"2023-10-06T05:06:56.708403+0000","last_active":"2023-10-06T05:07:01.749439+0000","last_peered":"2023-10-06T05:07:01.749439+0000","last_clean":"2023-10-06T05:07:01.749439+0000","last_became_active":"2023-10-06T05:06:56.708144+0000","last_became_peered":"2023-10-06T05:06:56.708144+0000","last_unstale":"2023-10-06T05:07:01.749439+0000","last_undegraded":"2023-10-06T05:07:01.749439+0000","last_fullsized":"2023-10-06T05:07:01.749439+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:17:02.349663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750522+0000","last_change":"2023-10-06T05:06:57.620557+0000","last_active":"2023-10-06T05:07:01.750522+0000","last_peered":"2023-10-06T05:07:01.750522+0000","last_clean":"2023-10-06T05:07:01.750522+0000","last_became_active":"2023-10-06T05:06:57.620286+0000","last_became_peered":"2023-10-06T05:06:57.620286+0000","last_unstale":"2023-10-06T05:07:01.750522+0000","last_undegraded":"2023-10-06T05:07:01.750522+0000","last_fullsized":"2023-10-06T05:07:01.750522+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T16:14:57.399087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.765266+0000","last_change":"2023-10-06T05:07:01.762346+0000","last_active":"2023-10-06T05:07:01.765266+0000","last_peered":"2023-10-06T05:07:01.765266+0000","last_clean":"2023-10-06T05:07:01.765266+0000","last_became_active":"2023-10-06T05:07:01.761861+0000","last_became_peered":"2023-10-06T05:07:01.761861+0000","last_unstale":"2023-10-06T05:07:01.765266+0000","last_undegraded":"2023-10-06T05:07:01.765266+0000","last_fullsized":"2023-10-06T05:07:01.765266+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:41:09.047631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6,5],"acting":[0,1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.748453+0000","last_change":"2023-10-06T05:06:52.627232+0000","last_active":"2023-10-06T05:07:01.748453+0000","last_peered":"2023-10-06T05:07:01.748453+0000","last_clean":"2023-10-06T05:07:01.748453+0000","last_became_active":"2023-10-06T05:06:52.626936+0000","last_became_peered":"2023-10-06T05:06:52.626936+0000","last_unstale":"2023-10-06T05:07:01.748453+0000","last_undegraded":"2023-10-06T05:07:01.748453+0000","last_fullsized":"2023-10-06T05:07:01.748453+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:20:13.925170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.744621+0000","last_change":"2023-10-06T05:06:56.712027+0000","last_active":"2023-10-06T05:07:01.744621+0000","last_peered":"2023-10-06T05:07:01.744621+0000","last_clean":"2023-10-06T05:07:01.744621+0000","last_became_active":"2023-10-06T05:06:56.711779+0000","last_became_peered":"2023-10-06T05:06:56.711779+0000","last_unstale":"2023-10-06T05:07:01.744621+0000","last_undegraded":"2023-10-06T05:07:01.744621+0000","last_fullsized":"2023-10-06T05:07:01.744621+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:49:27.022333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.747306+0000","last_change":"2023-10-06T05:06:57.613847+0000","last_active":"2023-10-06T05:07:01.747306+0000","last_peered":"2023-10-06T05:07:01.747306+0000","last_clean":"2023-10-06T05:07:01.747306+0000","last_became_active":"2023-10-06T05:06:57.613560+0000","last_became_peered":"2023-10-06T05:06:57.613560+0000","last_unstale":"2023-10-06T05:07:01.747306+0000","last_undegraded":"2023-10-06T05:07:01.747306+0000","last_fullsized":"2023-10-06T05:07:01.747306+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:50:36.787873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.766219+0000","last_change":"2023-10-06T05:07:01.764522+0000","last_active":"2023-10-06T05:07:01.766219+0000","last_peered":"2023-10-06T05:07:01.766219+0000","last_clean":"2023-10-06T05:07:01.766219+0000","last_became_active":"2023-10-06T05:07:01.764199+0000","last_became_peered":"2023-10-06T05:07:01.764199+0000","last_unstale":"2023-10-06T05:07:01.766219+0000","last_undegraded":"2023-10-06T05:07:01.766219+0000","last_fullsized":"2023-10-06T05:07:01.766219+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:13:19.947679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,6],"acting":[0,1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.744573+0000","last_change":"2023-10-06T05:06:52.587795+0000","last_active":"2023-10-06T05:07:01.744573+0000","last_peered":"2023-10-06T05:07:01.744573+0000","last_clean":"2023-10-06T05:07:01.744573+0000","last_became_active":"2023-10-06T05:06:52.587407+0000","last_became_peered":"2023-10-06T05:06:52.587407+0000","last_unstale":"2023-10-06T05:07:01.744573+0000","last_undegraded":"2023-10-06T05:07:01.744573+0000","last_fullsized":"2023-10-06T05:07:01.744573+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:36:31.526532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.761857+0000","last_change":"2023-10-06T05:06:56.721976+0000","last_active":"2023-10-06T05:07:01.761857+0000","last_peered":"2023-10-06T05:07:01.761857+0000","last_clean":"2023-10-06T05:07:01.761857+0000","last_became_active":"2023-10-06T05:06:56.721679+0000","last_became_peered":"2023-10-06T05:06:56.721679+0000","last_unstale":"2023-10-06T05:07:01.761857+0000","last_undegraded":"2023-10-06T05:07:01.761857+0000","last_fullsized":"2023-10-06T05:07:01.761857+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:38:50.877673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.748383+0000","last_change":"2023-10-06T05:06:57.612621+0000","last_active":"2023-10-06T05:07:01.748383+0000","last_peered":"2023-10-06T05:07:01.748383+0000","last_clean":"2023-10-06T05:07:01.748383+0000","last_became_active":"2023-10-06T05:06:57.612305+0000","last_became_peered":"2023-10-06T05:06:57.612305+0000","last_unstale":"2023-10-06T05:07:01.748383+0000","last_undegraded":"2023-10-06T05:07:01.748383+0000","last_fullsized":"2023-10-06T05:07:01.748383+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:35:18.828024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.768737+0000","last_change":"2023-10-06T05:07:01.764514+0000","last_active":"2023-10-06T05:07:01.768737+0000","last_peered":"2023-10-06T05:07:01.768737+0000","last_clean":"2023-10-06T05:07:01.768737+0000","last_became_active":"2023-10-06T05:07:01.764176+0000","last_became_peered":"2023-10-06T05:07:01.764176+0000","last_unstale":"2023-10-06T05:07:01.768737+0000","last_undegraded":"2023-10-06T05:07:01.768737+0000","last_fullsized":"2023-10-06T05:07:01.768737+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:20:08.719247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1,7],"acting":[2,5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.746279+0000","last_change":"2023-10-06T05:06:52.675238+0000","last_active":"2023-10-06T05:07:01.746279+0000","last_peered":"2023-10-06T05:07:01.746279+0000","last_clean":"2023-10-06T05:07:01.746279+0000","last_became_active":"2023-10-06T05:06:52.674952+0000","last_became_peered":"2023-10-06T05:06:52.674952+0000","last_unstale":"2023-10-06T05:07:01.746279+0000","last_undegraded":"2023-10-06T05:07:01.746279+0000","last_fullsized":"2023-10-06T05:07:01.746279+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:27:51.141184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.746657+0000","last_change":"2023-10-06T05:06:56.711280+0000","last_active":"2023-10-06T05:07:01.746657+0000","last_peered":"2023-10-06T05:07:01.746657+0000","last_clean":"2023-10-06T05:07:01.746657+0000","last_became_active":"2023-10-06T05:06:56.711033+0000","last_became_peered":"2023-10-06T05:06:56.711033+0000","last_unstale":"2023-10-06T05:07:01.746657+0000","last_undegraded":"2023-10-06T05:07:01.746657+0000","last_fullsized":"2023-10-06T05:07:01.746657+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:38:12.815936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750448+0000","last_change":"2023-10-06T05:06:57.614460+0000","last_active":"2023-10-06T05:07:01.750448+0000","last_peered":"2023-10-06T05:07:01.750448+0000","last_clean":"2023-10-06T05:07:01.750448+0000","last_became_active":"2023-10-06T05:06:57.614245+0000","last_became_peered":"2023-10-06T05:06:57.614245+0000","last_unstale":"2023-10-06T05:07:01.750448+0000","last_undegraded":"2023-10-06T05:07:01.750448+0000","last_fullsized":"2023-10-06T05:07:01.750448+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:51:31.873476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.775792+0000","last_change":"2023-10-06T05:07:01.766672+0000","last_active":"2023-10-06T05:07:01.775792+0000","last_peered":"2023-10-06T05:07:01.775792+0000","last_clean":"2023-10-06T05:07:01.775792+0000","last_became_active":"2023-10-06T05:07:01.766118+0000","last_became_peered":"2023-10-06T05:07:01.766118+0000","last_unstale":"2023-10-06T05:07:01.775792+0000","last_undegraded":"2023-10-06T05:07:01.775792+0000","last_fullsized":"2023-10-06T05:07:01.775792+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:55:02.566598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5,3],"acting":[7,2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.745050+0000","last_change":"2023-10-06T05:06:52.587778+0000","last_active":"2023-10-06T05:07:01.745050+0000","last_peered":"2023-10-06T05:07:01.745050+0000","last_clean":"2023-10-06T05:07:01.745050+0000","last_became_active":"2023-10-06T05:06:52.587388+0000","last_became_peered":"2023-10-06T05:06:52.587388+0000","last_unstale":"2023-10-06T05:07:01.745050+0000","last_undegraded":"2023-10-06T05:07:01.745050+0000","last_fullsized":"2023-10-06T05:07:01.745050+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:45:52.502298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":169,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.748339+0000","last_change":"2023-10-06T05:06:46.307169+0000","last_active":"2023-10-06T05:07:01.748339+0000","last_peered":"2023-10-06T05:07:01.748339+0000","last_clean":"2023-10-06T05:07:01.748339+0000","last_became_active":"2023-10-06T05:06:46.306107+0000","last_became_peered":"2023-10-06T05:06:46.306107+0000","last_unstale":"2023-10-06T05:07:01.748339+0000","last_undegraded":"2023-10-06T05:07:01.748339+0000","last_fullsized":"2023-10-06T05:07:01.748339+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:45.018006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:45.018006+0000","last_clean_scrub_stamp":"2023-10-06T05:06:45.018006+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:38:17.064131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750303+0000","last_change":"2023-10-06T05:06:56.706870+0000","last_active":"2023-10-06T05:07:01.750303+0000","last_peered":"2023-10-06T05:07:01.750303+0000","last_clean":"2023-10-06T05:07:01.750303+0000","last_became_active":"2023-10-06T05:06:56.706636+0000","last_became_peered":"2023-10-06T05:06:56.706636+0000","last_unstale":"2023-10-06T05:07:01.750303+0000","last_undegraded":"2023-10-06T05:07:01.750303+0000","last_fullsized":"2023-10-06T05:07:01.750303+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:00:29.033929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.753036+0000","last_change":"2023-10-06T05:06:57.604630+0000","last_active":"2023-10-06T05:07:01.753036+0000","last_peered":"2023-10-06T05:07:01.753036+0000","last_clean":"2023-10-06T05:07:01.753036+0000","last_became_active":"2023-10-06T05:06:57.604341+0000","last_became_peered":"2023-10-06T05:06:57.604341+0000","last_unstale":"2023-10-06T05:07:01.753036+0000","last_undegraded":"2023-10-06T05:07:01.753036+0000","last_fullsized":"2023-10-06T05:07:01.753036+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:57:12.615864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.767715+0000","last_change":"2023-10-06T05:07:01.763448+0000","last_active":"2023-10-06T05:07:01.767715+0000","last_peered":"2023-10-06T05:07:01.767715+0000","last_clean":"2023-10-06T05:07:01.767715+0000","last_became_active":"2023-10-06T05:07:01.762832+0000","last_became_peered":"2023-10-06T05:07:01.762832+0000","last_unstale":"2023-10-06T05:07:01.767715+0000","last_undegraded":"2023-10-06T05:07:01.767715+0000","last_fullsized":"2023-10-06T05:07:01.767715+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:49:53.682865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0,4],"acting":[5,1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.748590+0000","last_change":"2023-10-06T05:06:52.588475+0000","last_active":"2023-10-06T05:07:01.748590+0000","last_peered":"2023-10-06T05:07:01.748590+0000","last_clean":"2023-10-06T05:07:01.748590+0000","last_became_active":"2023-10-06T05:06:52.588134+0000","last_became_peered":"2023-10-06T05:06:52.588134+0000","last_unstale":"2023-10-06T05:07:01.748590+0000","last_undegraded":"2023-10-06T05:07:01.748590+0000","last_fullsized":"2023-10-06T05:07:01.748590+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:35:05.269778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.749699+0000","last_change":"2023-10-06T05:06:56.723240+0000","last_active":"2023-10-06T05:07:01.749699+0000","last_peered":"2023-10-06T05:07:01.749699+0000","last_clean":"2023-10-06T05:07:01.749699+0000","last_became_active":"2023-10-06T05:06:56.722965+0000","last_became_peered":"2023-10-06T05:06:56.722965+0000","last_unstale":"2023-10-06T05:07:01.749699+0000","last_undegraded":"2023-10-06T05:07:01.749699+0000","last_fullsized":"2023-10-06T05:07:01.749699+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:33:23.298771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.749910+0000","last_change":"2023-10-06T05:06:57.616494+0000","last_active":"2023-10-06T05:07:01.749910+0000","last_peered":"2023-10-06T05:07:01.749910+0000","last_clean":"2023-10-06T05:07:01.749910+0000","last_became_active":"2023-10-06T05:06:57.616234+0000","last_became_peered":"2023-10-06T05:06:57.616234+0000","last_unstale":"2023-10-06T05:07:01.749910+0000","last_undegraded":"2023-10-06T05:07:01.749910+0000","last_fullsized":"2023-10-06T05:07:01.749910+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:34:55.987076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.773039+0000","last_change":"2023-10-06T05:07:01.768822+0000","last_active":"2023-10-06T05:07:01.773039+0000","last_peered":"2023-10-06T05:07:01.773039+0000","last_clean":"2023-10-06T05:07:01.773039+0000","last_became_active":"2023-10-06T05:07:01.768425+0000","last_became_peered":"2023-10-06T05:07:01.768425+0000","last_unstale":"2023-10-06T05:07:01.773039+0000","last_undegraded":"2023-10-06T05:07:01.773039+0000","last_fullsized":"2023-10-06T05:07:01.773039+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:52:18.818435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5,2],"acting":[6,0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.754250+0000","last_change":"2023-10-06T05:06:52.627289+0000","last_active":"2023-10-06T05:07:01.754250+0000","last_peered":"2023-10-06T05:07:01.754250+0000","last_clean":"2023-10-06T05:07:01.754250+0000","last_became_active":"2023-10-06T05:06:52.626937+0000","last_became_peered":"2023-10-06T05:06:52.626937+0000","last_unstale":"2023-10-06T05:07:01.754250+0000","last_undegraded":"2023-10-06T05:07:01.754250+0000","last_fullsized":"2023-10-06T05:07:01.754250+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:46:59.070674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.748611+0000","last_change":"2023-10-06T05:06:56.720424+0000","last_active":"2023-10-06T05:07:01.748611+0000","last_peered":"2023-10-06T05:07:01.748611+0000","last_clean":"2023-10-06T05:07:01.748611+0000","last_became_active":"2023-10-06T05:06:56.720133+0000","last_became_peered":"2023-10-06T05:06:56.720133+0000","last_unstale":"2023-10-06T05:07:01.748611+0000","last_undegraded":"2023-10-06T05:07:01.748611+0000","last_fullsized":"2023-10-06T05:07:01.748611+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T16:11:31.404755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750404+0000","last_change":"2023-10-06T05:06:57.619048+0000","last_active":"2023-10-06T05:07:01.750404+0000","last_peered":"2023-10-06T05:07:01.750404+0000","last_clean":"2023-10-06T05:07:01.750404+0000","last_became_active":"2023-10-06T05:06:57.618823+0000","last_became_peered":"2023-10-06T05:06:57.618823+0000","last_unstale":"2023-10-06T05:07:01.750404+0000","last_undegraded":"2023-10-06T05:07:01.750404+0000","last_fullsized":"2023-10-06T05:07:01.750404+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:25:56.905027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.768167+0000","last_change":"2023-10-06T05:07:01.765125+0000","last_active":"2023-10-06T05:07:01.768167+0000","last_peered":"2023-10-06T05:07:01.768167+0000","last_clean":"2023-10-06T05:07:01.768167+0000","last_became_active":"2023-10-06T05:07:01.764733+0000","last_became_peered":"2023-10-06T05:07:01.764733+0000","last_unstale":"2023-10-06T05:07:01.768167+0000","last_undegraded":"2023-10-06T05:07:01.768167+0000","last_fullsized":"2023-10-06T05:07:01.768167+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T16:29:47.226299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7,0],"acting":[4,3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"24'2","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750393+0000","last_change":"2023-10-06T05:06:56.710924+0000","last_active":"2023-10-06T05:07:01.750393+0000","last_peered":"2023-10-06T05:07:01.750393+0000","last_clean":"2023-10-06T05:07:01.750393+0000","last_became_active":"2023-10-06T05:06:56.710518+0000","last_became_peered":"2023-10-06T05:06:56.710518+0000","last_unstale":"2023-10-06T05:07:01.750393+0000","last_undegraded":"2023-10-06T05:07:01.750393+0000","last_fullsized":"2023-10-06T05:07:01.750393+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:49:27.022333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.748508+0000","last_change":"2023-10-06T05:06:52.588853+0000","last_active":"2023-10-06T05:07:01.748508+0000","last_peered":"2023-10-06T05:07:01.748508+0000","last_clean":"2023-10-06T05:07:01.748508+0000","last_became_active":"2023-10-06T05:06:52.588561+0000","last_became_peered":"2023-10-06T05:06:52.588561+0000","last_unstale":"2023-10-06T05:07:01.748508+0000","last_undegraded":"2023-10-06T05:07:01.748508+0000","last_fullsized":"2023-10-06T05:07:01.748508+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:06:28.729493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.749357+0000","last_change":"2023-10-06T05:06:57.619145+0000","last_active":"2023-10-06T05:07:01.749357+0000","last_peered":"2023-10-06T05:07:01.749357+0000","last_clean":"2023-10-06T05:07:01.749357+0000","last_became_active":"2023-10-06T05:06:57.618666+0000","last_became_peered":"2023-10-06T05:06:57.618666+0000","last_unstale":"2023-10-06T05:07:01.749357+0000","last_undegraded":"2023-10-06T05:07:01.749357+0000","last_fullsized":"2023-10-06T05:07:01.749357+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:29:19.951056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.765168+0000","last_change":"2023-10-06T05:07:01.762410+0000","last_active":"2023-10-06T05:07:01.765168+0000","last_peered":"2023-10-06T05:07:01.765168+0000","last_clean":"2023-10-06T05:07:01.765168+0000","last_became_active":"2023-10-06T05:07:01.762060+0000","last_became_peered":"2023-10-06T05:07:01.762060+0000","last_unstale":"2023-10-06T05:07:01.765168+0000","last_undegraded":"2023-10-06T05:07:01.765168+0000","last_fullsized":"2023-10-06T05:07:01.765168+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:49:53.682865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,0],"acting":[3,1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.751181+0000","last_change":"2023-10-06T05:06:52.589519+0000","last_active":"2023-10-06T05:07:01.751181+0000","last_peered":"2023-10-06T05:07:01.751181+0000","last_clean":"2023-10-06T05:07:01.751181+0000","last_became_active":"2023-10-06T05:06:52.589246+0000","last_became_peered":"2023-10-06T05:06:52.589246+0000","last_unstale":"2023-10-06T05:07:01.751181+0000","last_undegraded":"2023-10-06T05:07:01.751181+0000","last_fullsized":"2023-10-06T05:07:01.751181+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:54:23.352764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"28'1","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:02.685849+0000","last_change":"2023-10-06T05:06:56.714058+0000","last_active":"2023-10-06T05:07:02.685849+0000","last_peered":"2023-10-06T05:07:02.685849+0000","last_clean":"2023-10-06T05:07:02.685849+0000","last_became_active":"2023-10-06T05:06:56.713772+0000","last_became_peered":"2023-10-06T05:06:56.713772+0000","last_unstale":"2023-10-06T05:07:02.685849+0000","last_undegraded":"2023-10-06T05:07:02.685849+0000","last_fullsized":"2023-10-06T05:07:02.685849+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:53:47.748350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.761483+0000","last_change":"2023-10-06T05:06:57.608832+0000","last_active":"2023-10-06T05:07:01.761483+0000","last_peered":"2023-10-06T05:07:01.761483+0000","last_clean":"2023-10-06T05:07:01.761483+0000","last_became_active":"2023-10-06T05:06:57.608499+0000","last_became_peered":"2023-10-06T05:06:57.608499+0000","last_unstale":"2023-10-06T05:07:01.761483+0000","last_undegraded":"2023-10-06T05:07:01.761483+0000","last_fullsized":"2023-10-06T05:07:01.761483+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:29:19.951056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.772123+0000","last_change":"2023-10-06T05:07:01.765615+0000","last_active":"2023-10-06T05:07:01.772123+0000","last_peered":"2023-10-06T05:07:01.772123+0000","last_clean":"2023-10-06T05:07:01.772123+0000","last_became_active":"2023-10-06T05:07:01.762552+0000","last_became_peered":"2023-10-06T05:07:01.762552+0000","last_unstale":"2023-10-06T05:07:01.772123+0000","last_undegraded":"2023-10-06T05:07:01.772123+0000","last_fullsized":"2023-10-06T05:07:01.772123+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:30:14.995001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,0],"acting":[1,3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"24'1","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750483+0000","last_change":"2023-10-06T05:06:56.723224+0000","last_active":"2023-10-06T05:07:01.750483+0000","last_peered":"2023-10-06T05:07:01.750483+0000","last_clean":"2023-10-06T05:07:01.750483+0000","last_became_active":"2023-10-06T05:06:56.722971+0000","last_became_peered":"2023-10-06T05:06:56.722971+0000","last_unstale":"2023-10-06T05:07:01.750483+0000","last_undegraded":"2023-10-06T05:07:01.750483+0000","last_fullsized":"2023-10-06T05:07:01.750483+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:10:32.670079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.746067+0000","last_change":"2023-10-06T05:06:57.610613+0000","last_active":"2023-10-06T05:07:01.746067+0000","last_peered":"2023-10-06T05:07:01.746067+0000","last_clean":"2023-10-06T05:07:01.746067+0000","last_became_active":"2023-10-06T05:06:57.610363+0000","last_became_peered":"2023-10-06T05:06:57.610363+0000","last_unstale":"2023-10-06T05:07:01.746067+0000","last_undegraded":"2023-10-06T05:07:01.746067+0000","last_fullsized":"2023-10-06T05:07:01.746067+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:55:09.125970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.775891+0000","last_change":"2023-10-06T05:07:01.771319+0000","last_active":"2023-10-06T05:07:01.775891+0000","last_peered":"2023-10-06T05:07:01.775891+0000","last_clean":"2023-10-06T05:07:01.775891+0000","last_became_active":"2023-10-06T05:07:01.770953+0000","last_became_peered":"2023-10-06T05:07:01.770953+0000","last_unstale":"2023-10-06T05:07:01.775891+0000","last_undegraded":"2023-10-06T05:07:01.775891+0000","last_fullsized":"2023-10-06T05:07:01.775891+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:13:19.947679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1,6],"acting":[7,4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"24'2","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.745566+0000","last_change":"2023-10-06T05:06:56.723033+0000","last_active":"2023-10-06T05:07:01.745566+0000","last_peered":"2023-10-06T05:07:01.745566+0000","last_clean":"2023-10-06T05:07:01.745566+0000","last_became_active":"2023-10-06T05:06:56.722747+0000","last_became_peered":"2023-10-06T05:06:56.722747+0000","last_unstale":"2023-10-06T05:07:01.745566+0000","last_undegraded":"2023-10-06T05:07:01.745566+0000","last_fullsized":"2023-10-06T05:07:01.745566+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:17:02.349663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.761541+0000","last_change":"2023-10-06T05:06:57.621769+0000","last_active":"2023-10-06T05:07:01.761541+0000","last_peered":"2023-10-06T05:07:01.761541+0000","last_clean":"2023-10-06T05:07:01.761541+0000","last_became_active":"2023-10-06T05:06:57.621489+0000","last_became_peered":"2023-10-06T05:06:57.621489+0000","last_unstale":"2023-10-06T05:07:01.761541+0000","last_undegraded":"2023-10-06T05:07:01.761541+0000","last_fullsized":"2023-10-06T05:07:01.761541+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:59:39.994965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.773062+0000","last_change":"2023-10-06T05:07:01.769584+0000","last_active":"2023-10-06T05:07:01.773062+0000","last_peered":"2023-10-06T05:07:01.773062+0000","last_clean":"2023-10-06T05:07:01.773062+0000","last_became_active":"2023-10-06T05:07:01.769168+0000","last_became_peered":"2023-10-06T05:07:01.769168+0000","last_unstale":"2023-10-06T05:07:01.773062+0000","last_undegraded":"2023-10-06T05:07:01.773062+0000","last_fullsized":"2023-10-06T05:07:01.773062+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:25:41.281713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7,4],"acting":[5,0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"24'2","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.744323+0000","last_change":"2023-10-06T05:06:56.721940+0000","last_active":"2023-10-06T05:07:01.744323+0000","last_peered":"2023-10-06T05:07:01.744323+0000","last_clean":"2023-10-06T05:07:01.744323+0000","last_became_active":"2023-10-06T05:06:56.721668+0000","last_became_peered":"2023-10-06T05:06:56.721668+0000","last_unstale":"2023-10-06T05:07:01.744323+0000","last_undegraded":"2023-10-06T05:07:01.744323+0000","last_fullsized":"2023-10-06T05:07:01.744323+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:55:08.124129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.760815+0000","last_change":"2023-10-06T05:06:57.614557+0000","last_active":"2023-10-06T05:07:01.760815+0000","last_peered":"2023-10-06T05:07:01.760815+0000","last_clean":"2023-10-06T05:07:01.760815+0000","last_became_active":"2023-10-06T05:06:57.614113+0000","last_became_peered":"2023-10-06T05:06:57.614113+0000","last_unstale":"2023-10-06T05:07:01.760815+0000","last_undegraded":"2023-10-06T05:07:01.760815+0000","last_fullsized":"2023-10-06T05:07:01.760815+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:57:12.615864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.772225+0000","last_change":"2023-10-06T05:07:01.766934+0000","last_active":"2023-10-06T05:07:01.772225+0000","last_peered":"2023-10-06T05:07:01.772225+0000","last_clean":"2023-10-06T05:07:01.772225+0000","last_became_active":"2023-10-06T05:07:01.766522+0000","last_became_peered":"2023-10-06T05:07:01.766522+0000","last_unstale":"2023-10-06T05:07:01.772225+0000","last_undegraded":"2023-10-06T05:07:01.772225+0000","last_fullsized":"2023-10-06T05:07:01.772225+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:15:30.724218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,2],"acting":[4,6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"24'1","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.753608+0000","last_change":"2023-10-06T05:06:56.717832+0000","last_active":"2023-10-06T05:07:01.753608+0000","last_peered":"2023-10-06T05:07:01.753608+0000","last_clean":"2023-10-06T05:07:01.753608+0000","last_became_active":"2023-10-06T05:06:56.717568+0000","last_became_peered":"2023-10-06T05:06:56.717568+0000","last_unstale":"2023-10-06T05:07:01.753608+0000","last_undegraded":"2023-10-06T05:07:01.753608+0000","last_fullsized":"2023-10-06T05:07:01.753608+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:30:18.507322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750496+0000","last_change":"2023-10-06T05:06:57.620391+0000","last_active":"2023-10-06T05:07:01.750496+0000","last_peered":"2023-10-06T05:07:01.750496+0000","last_clean":"2023-10-06T05:07:01.750496+0000","last_became_active":"2023-10-06T05:06:57.619822+0000","last_became_peered":"2023-10-06T05:06:57.619822+0000","last_unstale":"2023-10-06T05:07:01.750496+0000","last_undegraded":"2023-10-06T05:07:01.750496+0000","last_fullsized":"2023-10-06T05:07:01.750496+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:59:39.994965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.772089+0000","last_change":"2023-10-06T05:07:01.762139+0000","last_active":"2023-10-06T05:07:01.772089+0000","last_peered":"2023-10-06T05:07:01.772089+0000","last_clean":"2023-10-06T05:07:01.772089+0000","last_became_active":"2023-10-06T05:07:01.761672+0000","last_became_peered":"2023-10-06T05:07:01.761672+0000","last_unstale":"2023-10-06T05:07:01.772089+0000","last_undegraded":"2023-10-06T05:07:01.772089+0000","last_fullsized":"2023-10-06T05:07:01.772089+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:07:51.743165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7,2],"acting":[1,5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"24'4","reported_seq":18,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.751360+0000","last_change":"2023-10-06T05:06:56.714871+0000","last_active":"2023-10-06T05:07:01.751360+0000","last_peered":"2023-10-06T05:07:01.751360+0000","last_clean":"2023-10-06T05:07:01.751360+0000","last_became_active":"2023-10-06T05:06:56.714616+0000","last_became_peered":"2023-10-06T05:06:56.714616+0000","last_unstale":"2023-10-06T05:07:01.751360+0000","last_undegraded":"2023-10-06T05:07:01.751360+0000","last_fullsized":"2023-10-06T05:07:01.751360+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:35:42.402195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.758731+0000","last_change":"2023-10-06T05:06:57.605198+0000","last_active":"2023-10-06T05:07:01.758731+0000","last_peered":"2023-10-06T05:07:01.758731+0000","last_clean":"2023-10-06T05:07:01.758731+0000","last_became_active":"2023-10-06T05:06:57.604924+0000","last_became_peered":"2023-10-06T05:06:57.604924+0000","last_unstale":"2023-10-06T05:07:01.758731+0000","last_undegraded":"2023-10-06T05:07:01.758731+0000","last_fullsized":"2023-10-06T05:07:01.758731+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:01:11.983432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750721+0000","last_change":"2023-10-06T05:06:57.608286+0000","last_active":"2023-10-06T05:07:01.750721+0000","last_peered":"2023-10-06T05:07:01.750721+0000","last_clean":"2023-10-06T05:07:01.750721+0000","last_became_active":"2023-10-06T05:06:57.607986+0000","last_became_peered":"2023-10-06T05:06:57.607986+0000","last_unstale":"2023-10-06T05:07:01.750721+0000","last_undegraded":"2023-10-06T05:07:01.750721+0000","last_fullsized":"2023-10-06T05:07:01.750721+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:51:31.370159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750626+0000","last_change":"2023-10-06T05:06:57.623937+0000","last_active":"2023-10-06T05:07:01.750626+0000","last_peered":"2023-10-06T05:07:01.750626+0000","last_clean":"2023-10-06T05:07:01.750626+0000","last_became_active":"2023-10-06T05:06:57.623705+0000","last_became_peered":"2023-10-06T05:06:57.623705+0000","last_unstale":"2023-10-06T05:07:01.750626+0000","last_undegraded":"2023-10-06T05:07:01.750626+0000","last_fullsized":"2023-10-06T05:07:01.750626+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:20:04.673613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.761577+0000","last_change":"2023-10-06T05:06:57.611066+0000","last_active":"2023-10-06T05:07:01.761577+0000","last_peered":"2023-10-06T05:07:01.761577+0000","last_clean":"2023-10-06T05:07:01.761577+0000","last_became_active":"2023-10-06T05:06:57.610827+0000","last_became_peered":"2023-10-06T05:06:57.610827+0000","last_unstale":"2023-10-06T05:07:01.761577+0000","last_undegraded":"2023-10-06T05:07:01.761577+0000","last_fullsized":"2023-10-06T05:07:01.761577+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:43:18.121629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750840+0000","last_change":"2023-10-06T05:06:57.615191+0000","last_active":"2023-10-06T05:07:01.750840+0000","last_peered":"2023-10-06T05:07:01.750840+0000","last_clean":"2023-10-06T05:07:01.750840+0000","last_became_active":"2023-10-06T05:06:57.614632+0000","last_became_peered":"2023-10-06T05:06:57.614632+0000","last_unstale":"2023-10-06T05:07:01.750840+0000","last_undegraded":"2023-10-06T05:07:01.750840+0000","last_fullsized":"2023-10-06T05:07:01.750840+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:32:33.731449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.748531+0000","last_change":"2023-10-06T05:06:57.616743+0000","last_active":"2023-10-06T05:07:01.748531+0000","last_peered":"2023-10-06T05:07:01.748531+0000","last_clean":"2023-10-06T05:07:01.748531+0000","last_became_active":"2023-10-06T05:06:57.616487+0000","last_became_peered":"2023-10-06T05:06:57.616487+0000","last_unstale":"2023-10-06T05:07:01.748531+0000","last_undegraded":"2023-10-06T05:07:01.748531+0000","last_fullsized":"2023-10-06T05:07:01.748531+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:15:55.618648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":4484,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":8968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509448,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297588,"kb_used_data":300,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074252,"statfs":{"total":96636764160,"available":96332034048,"internally_reserved":0,"allocated":307200,"data_stored":126099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.376}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.498}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42799999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64500000000000002}]}]},{"osd":5,"up_from":15,"seq":64424509448,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297596,"kb_used_data":308,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074244,"statfs":{"total":96636764160,"available":96332025856,"internally_reserved":0,"allocated":315392,"data_stored":126723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56799999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48799999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55500000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66800000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63200000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542152,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298172,"kb_used_data":884,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073668,"statfs":{"total":96636764160,"available":96331436032,"internally_reserved":0,"allocated":905216,"data_stored":717021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33700000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]}]},{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298196,"kb_used_data":908,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073644,"statfs":{"total":96636764160,"available":96331411456,"internally_reserved":0,"allocated":929792,"data_stored":718672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56599999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57599999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53300000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64300000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69599999999999995}]}]},{"osd":1,"up_from":14,"seq":60129542151,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297588,"kb_used_data":300,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074252,"statfs":{"total":96636764160,"available":96332034048,"internally_reserved":0,"allocated":307200,"data_stored":126099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29999999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33200000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54900000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72399999999999998}]}]},{"osd":2,"up_from":15,"seq":64424509447,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297588,"kb_used_data":300,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074252,"statfs":{"total":96636764160,"available":96332034048,"internally_reserved":0,"allocated":307200,"data_stored":126099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26400000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30599999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28000000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39400000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42299999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59799999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.504}]}]},{"osd":3,"up_from":15,"seq":64424509447,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297592,"kb_used_data":304,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074248,"statfs":{"total":96636764160,"available":96332029952,"internally_reserved":0,"allocated":311296,"data_stored":126189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26700000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28599999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32700000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66900000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68000000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.625}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56299999999999994}]}]},{"osd":4,"up_from":15,"seq":64424509448,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297608,"kb_used_data":320,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074232,"statfs":{"total":96636764160,"available":96332013568,"internally_reserved":0,"allocated":327680,"data_stored":129869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40100000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.379}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":3770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":3930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-06T05:07:10.834 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-06T05:07:10.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.975+0000 7f07a6ac9700 1 Processor -- start 2023-10-06T05:07:10.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.975+0000 7f07a6ac9700 1 -- start start 2023-10-06T05:07:10.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.976+0000 7f07a6ac9700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f07a0159560 0x7f07a00aa530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:10.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.976+0000 7f07a6ac9700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f07a00b49a0 0x7f07a00aaa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:10.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.976+0000 7f07a6ac9700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07a00b0c60 con 0x7f07a00b0760 2023-10-06T05:07:10.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.976+0000 7f07a6ac9700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07a00b0f40 con 0x7f07a00b49a0 2023-10-06T05:07:10.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.976+0000 7f07a6ac9700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07a00b1220 con 0x7f07a0159560 2023-10-06T05:07:10.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.976+0000 7f07a5066700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f07a00b0760 0x7f07a00b3890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:41124/0 (socket says 172.21.15.98:41124) 2023-10-06T05:07:10.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.976+0000 7f07a5066700 1 -- 172.21.15.98:0/1159515711 learned_addr learned my addr 172.21.15.98:0/1159515711 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:07:10.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.976+0000 7f079ffff700 1 --2- 172.21.15.98:0/1159515711 >> v2:172.21.15.174:3300/0 conn(0x7f07a00b49a0 0x7f07a00aaa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:10.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.976+0000 7f07a4865700 1 --2- 172.21.15.98:0/1159515711 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f07a0159560 0x7f07a00aa530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:10.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.976+0000 7f079f7fe700 1 -- 172.21.15.98:0/1159515711 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1370367203 0 0) 0x7f07a00b0c60 con 0x7f07a00b0760 2023-10-06T05:07:10.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.976+0000 7f079f7fe700 1 -- 172.21.15.98:0/1159515711 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0788003570 con 0x7f07a00b0760 2023-10-06T05:07:10.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.976+0000 7f079ffff700 1 -- 172.21.15.98:0/1159515711 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f07a0159560 msgr2=0x7f07a00aa530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:10.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.976+0000 7f079ffff700 1 --2- 172.21.15.98:0/1159515711 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f07a0159560 0x7f07a00aa530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:10.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.976+0000 7f079ffff700 1 -- 172.21.15.98:0/1159515711 >> v1:172.21.15.98:6789/0 conn(0x7f07a00b0760 legacy=0x7f07a00b3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:10.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.976+0000 7f079ffff700 1 -- 172.21.15.98:0/1159515711 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07a00b1500 con 0x7f07a00b49a0 2023-10-06T05:07:10.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.976+0000 7f07a4865700 1 --2- 172.21.15.98:0/1159515711 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f07a0159560 0x7f07a00aa530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T05:07:10.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.977+0000 7f079ffff700 1 --2- 172.21.15.98:0/1159515711 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f07a00b49a0 0x7f07a00aaa70 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f07940098e0 tx=0x7f07940091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=136c2e21f8e11167 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:10.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.977+0000 7f079f7fe700 1 -- 172.21.15.98:0/1159515711 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0794011630 con 0x7f07a00b49a0 2023-10-06T05:07:10.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.977+0000 7f079f7fe700 1 -- 172.21.15.98:0/1159515711 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0794011790 con 0x7f07a00b49a0 2023-10-06T05:07:10.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.977+0000 7f079f7fe700 1 -- 172.21.15.98:0/1159515711 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0794011b30 con 0x7f07a00b49a0 2023-10-06T05:07:10.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.977+0000 7f07a6ac9700 1 -- 172.21.15.98:0/1159515711 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f07a00b49a0 msgr2=0x7f07a00aaa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:10.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.977+0000 7f07a6ac9700 1 --2- 172.21.15.98:0/1159515711 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f07a00b49a0 0x7f07a00aaa70 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f07940098e0 tx=0x7f07940091a0 comp rx=0 tx=0).stop 2023-10-06T05:07:10.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.977+0000 7f07a6ac9700 1 -- 172.21.15.98:0/1159515711 shutdown_connections 2023-10-06T05:07:10.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.977+0000 7f07a6ac9700 1 --2- 172.21.15.98:0/1159515711 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f07a0159560 0x7f07a00aa530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:10.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.977+0000 7f07a6ac9700 1 --2- 172.21.15.98:0/1159515711 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f07a00b49a0 0x7f07a00aaa70 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:10.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.977+0000 7f07a6ac9700 1 -- 172.21.15.98:0/1159515711 >> 172.21.15.98:0/1159515711 conn(0x7f07a00997f0 msgr2=0x7f07a006f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:10.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.977+0000 7f07a6ac9700 1 -- 172.21.15.98:0/1159515711 shutdown_connections 2023-10-06T05:07:10.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.977+0000 7f07a6ac9700 1 -- 172.21.15.98:0/1159515711 wait complete. 2023-10-06T05:07:10.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.978+0000 7f07a6ac9700 1 Processor -- start 2023-10-06T05:07:10.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.979+0000 7f07a6ac9700 1 -- start start 2023-10-06T05:07:10.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.979+0000 7f07a6ac9700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f07a00b0760 0x7f07a009a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:10.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.979+0000 7f07a6ac9700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f07a00b49a0 0x7f07a00cbab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:10.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.979+0000 7f07a6ac9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07a0159560 0x7f07a01579c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:10.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.979+0000 7f07a6ac9700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07a0158480 con 0x7f07a0159560 2023-10-06T05:07:10.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.979+0000 7f07a6ac9700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07a009c070 con 0x7f07a00b49a0 2023-10-06T05:07:10.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.979+0000 7f07a6ac9700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07a009a590 con 0x7f07a00b0760 2023-10-06T05:07:10.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.979+0000 7f07a5066700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07a0159560 0x7f07a01579c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:10.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.979+0000 7f07a5066700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07a0159560 0x7f07a01579c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45644/0 (socket says 172.21.15.98:45644) 2023-10-06T05:07:10.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.979+0000 7f07a5066700 1 -- 172.21.15.98:0/3483930205 learned_addr learned my addr 172.21.15.98:0/3483930205 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:10.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.980+0000 7f079ffff700 1 --2- 172.21.15.98:0/3483930205 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f07a00b49a0 0x7f07a00cbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:10.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.980+0000 7f07a4865700 1 --2- 172.21.15.98:0/3483930205 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f07a00b0760 0x7f07a009a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:10.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.980+0000 7f07a5066700 1 -- 172.21.15.98:0/3483930205 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f07a00b0760 msgr2=0x7f07a009a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:10.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.980+0000 7f07a5066700 1 --2- 172.21.15.98:0/3483930205 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f07a00b0760 0x7f07a009a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:10.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.980+0000 7f07a5066700 1 -- 172.21.15.98:0/3483930205 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f07a00b49a0 msgr2=0x7f07a00cbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:10.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.980+0000 7f07a5066700 1 --2- 172.21.15.98:0/3483930205 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f07a00b49a0 0x7f07a00cbab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:10.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.980+0000 7f07a5066700 1 -- 172.21.15.98:0/3483930205 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0790009a20 con 0x7f07a0159560 2023-10-06T05:07:10.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.980+0000 7f07a4865700 1 --2- 172.21.15.98:0/3483930205 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f07a00b0760 0x7f07a009a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:07:10.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.980+0000 7f07a5066700 1 --2- 172.21.15.98:0/3483930205 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07a0159560 0x7f07a01579c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f079800c7c0 tx=0x7f0798007dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:10.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.980+0000 7f079cff9700 1 -- 172.21.15.98:0/3483930205 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0798005c00 con 0x7f07a0159560 2023-10-06T05:07:10.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.980+0000 7f079cff9700 1 -- 172.21.15.98:0/3483930205 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0798005d60 con 0x7f07a0159560 2023-10-06T05:07:10.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.980+0000 7f07a6ac9700 1 -- 172.21.15.98:0/3483930205 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0794009420 con 0x7f07a0159560 2023-10-06T05:07:10.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.980+0000 7f079cff9700 1 -- 172.21.15.98:0/3483930205 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07980072e0 con 0x7f07a0159560 2023-10-06T05:07:10.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.980+0000 7f07a6ac9700 1 -- 172.21.15.98:0/3483930205 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07a009ba20 con 0x7f07a0159560 2023-10-06T05:07:10.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.981+0000 7f07a6ac9700 1 -- 172.21.15.98:0/3483930205 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07a016cc20 con 0x7f07a0159560 2023-10-06T05:07:10.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.981+0000 7f079cff9700 1 -- 172.21.15.98:0/3483930205 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f0798007650 con 0x7f07a0159560 2023-10-06T05:07:10.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.982+0000 7f079cff9700 1 --2- 172.21.15.98:0/3483930205 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f078c066f30 0x7f078c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:10.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.982+0000 7f079cff9700 1 -- 172.21.15.98:0/3483930205 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f0798083e60 con 0x7f07a0159560 2023-10-06T05:07:10.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.982+0000 7f07a4865700 1 --2- 172.21.15.98:0/3483930205 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f078c066f30 0x7f078c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:10.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.982+0000 7f07a4865700 1 --2- 172.21.15.98:0/3483930205 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f078c066f30 0x7f078c0693e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f07900099f0 tx=0x7f079000b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:10.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:10.986+0000 7f079cff9700 1 -- 172.21.15.98:0/3483930205 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f079804b5a0 con 0x7f07a0159560 2023-10-06T05:07:11.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.164+0000 7f07a6ac9700 1 -- 172.21.15.98:0/3483930205 --> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f07a0064a90 con 0x7f078c066f30 2023-10-06T05:07:11.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.174+0000 7f079cff9700 1 -- 172.21.15.98:0/3483930205 <== mgr.4104 v2:172.21.15.98:6832/90397 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+367360 (secure 0 0 0) 0x7f07a0064a90 con 0x7f078c066f30 2023-10-06T05:07:11.181 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:07:11.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.178+0000 7f07a6ac9700 1 -- 172.21.15.98:0/3483930205 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f078c066f30 msgr2=0x7f078c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:11.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.178+0000 7f07a6ac9700 1 --2- 172.21.15.98:0/3483930205 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f078c066f30 0x7f078c0693e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f07900099f0 tx=0x7f079000b040 comp rx=0 tx=0).stop 2023-10-06T05:07:11.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.178+0000 7f07a6ac9700 1 -- 172.21.15.98:0/3483930205 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07a0159560 msgr2=0x7f07a01579c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:11.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.178+0000 7f07a6ac9700 1 --2- 172.21.15.98:0/3483930205 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07a0159560 0x7f07a01579c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f079800c7c0 tx=0x7f0798007dd0 comp rx=0 tx=0).stop 2023-10-06T05:07:11.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.178+0000 7f07a6ac9700 1 -- 172.21.15.98:0/3483930205 shutdown_connections 2023-10-06T05:07:11.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.178+0000 7f07a6ac9700 1 --2- 172.21.15.98:0/3483930205 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f078c066f30 0x7f078c0693e0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:11.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.178+0000 7f07a6ac9700 1 --2- 172.21.15.98:0/3483930205 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07a0159560 0x7f07a01579c0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:11.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.178+0000 7f07a6ac9700 1 --2- 172.21.15.98:0/3483930205 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f07a00b0760 0x7f07a009a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:11.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.178+0000 7f07a6ac9700 1 --2- 172.21.15.98:0/3483930205 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f07a00b49a0 0x7f07a00cbab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:11.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.178+0000 7f07a6ac9700 1 -- 172.21.15.98:0/3483930205 >> 172.21.15.98:0/3483930205 conn(0x7f07a00997f0 msgr2=0x7f07a0158a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:11.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.178+0000 7f07a6ac9700 1 -- 172.21.15.98:0/3483930205 shutdown_connections 2023-10-06T05:07:11.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.178+0000 7f07a6ac9700 1 -- 172.21.15.98:0/3483930205 wait complete. 2023-10-06T05:07:11.191 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2023-10-06T05:07:11.193 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2023-10-06T05:07:10.522196+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594852,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":338,"acting":338,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":338,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2381928,"kb_used_data":3624,"kb_used_omap":0,"kb_used_meta":2378240,"kb_avail":752592792,"statfs":{"total":773094113280,"available":770655019008,"internally_reserved":0,"allocated":3710976,"data_stored":2196771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2435317760},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2178,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.780017"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.753513+0000","last_change":"2023-10-06T05:06:57.619460+0000","last_active":"2023-10-06T05:07:01.753513+0000","last_peered":"2023-10-06T05:07:01.753513+0000","last_clean":"2023-10-06T05:07:01.753513+0000","last_became_active":"2023-10-06T05:06:57.619199+0000","last_became_peered":"2023-10-06T05:06:57.619199+0000","last_unstale":"2023-10-06T05:07:01.753513+0000","last_undegraded":"2023-10-06T05:07:01.753513+0000","last_fullsized":"2023-10-06T05:07:01.753513+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:49:28.024174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.747458+0000","last_change":"2023-10-06T05:06:57.610280+0000","last_active":"2023-10-06T05:07:01.747458+0000","last_peered":"2023-10-06T05:07:01.747458+0000","last_clean":"2023-10-06T05:07:01.747458+0000","last_became_active":"2023-10-06T05:06:57.609973+0000","last_became_peered":"2023-10-06T05:06:57.609973+0000","last_unstale":"2023-10-06T05:07:01.747458+0000","last_undegraded":"2023-10-06T05:07:01.747458+0000","last_fullsized":"2023-10-06T05:07:01.747458+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:22:11.051575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.762104+0000","last_change":"2023-10-06T05:06:57.615468+0000","last_active":"2023-10-06T05:07:01.762104+0000","last_peered":"2023-10-06T05:07:01.762104+0000","last_clean":"2023-10-06T05:07:01.762104+0000","last_became_active":"2023-10-06T05:06:57.615221+0000","last_became_peered":"2023-10-06T05:06:57.615221+0000","last_unstale":"2023-10-06T05:07:01.762104+0000","last_undegraded":"2023-10-06T05:07:01.762104+0000","last_fullsized":"2023-10-06T05:07:01.762104+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:49:48.309158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.754474+0000","last_change":"2023-10-06T05:06:57.614470+0000","last_active":"2023-10-06T05:07:01.754474+0000","last_peered":"2023-10-06T05:07:01.754474+0000","last_clean":"2023-10-06T05:07:01.754474+0000","last_became_active":"2023-10-06T05:06:57.614169+0000","last_became_peered":"2023-10-06T05:06:57.614169+0000","last_unstale":"2023-10-06T05:07:01.754474+0000","last_undegraded":"2023-10-06T05:07:01.754474+0000","last_fullsized":"2023-10-06T05:07:01.754474+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:23:03.302621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750277+0000","last_change":"2023-10-06T05:06:57.622050+0000","last_active":"2023-10-06T05:07:01.750277+0000","last_peered":"2023-10-06T05:07:01.750277+0000","last_clean":"2023-10-06T05:07:01.750277+0000","last_became_active":"2023-10-06T05:06:57.621777+0000","last_became_peered":"2023-10-06T05:06:57.621777+0000","last_unstale":"2023-10-06T05:07:01.750277+0000","last_undegraded":"2023-10-06T05:07:01.750277+0000","last_fullsized":"2023-10-06T05:07:01.750277+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:04:40.577463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.761680+0000","last_change":"2023-10-06T05:06:57.611449+0000","last_active":"2023-10-06T05:07:01.761680+0000","last_peered":"2023-10-06T05:07:01.761680+0000","last_clean":"2023-10-06T05:07:01.761680+0000","last_became_active":"2023-10-06T05:06:57.610301+0000","last_became_peered":"2023-10-06T05:06:57.610301+0000","last_unstale":"2023-10-06T05:07:01.761680+0000","last_undegraded":"2023-10-06T05:07:01.761680+0000","last_fullsized":"2023-10-06T05:07:01.761680+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:43:21.654853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.755118+0000","last_change":"2023-10-06T05:06:57.610678+0000","last_active":"2023-10-06T05:07:01.755118+0000","last_peered":"2023-10-06T05:07:01.755118+0000","last_clean":"2023-10-06T05:07:01.755118+0000","last_became_active":"2023-10-06T05:06:57.609746+0000","last_became_peered":"2023-10-06T05:06:57.609746+0000","last_unstale":"2023-10-06T05:07:01.755118+0000","last_undegraded":"2023-10-06T05:07:01.755118+0000","last_fullsized":"2023-10-06T05:07:01.755118+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:11:00.525585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.745457+0000","last_change":"2023-10-06T05:06:57.612559+0000","last_active":"2023-10-06T05:07:01.745457+0000","last_peered":"2023-10-06T05:07:01.745457+0000","last_clean":"2023-10-06T05:07:01.745457+0000","last_became_active":"2023-10-06T05:06:57.612199+0000","last_became_peered":"2023-10-06T05:06:57.612199+0000","last_unstale":"2023-10-06T05:07:01.745457+0000","last_undegraded":"2023-10-06T05:07:01.745457+0000","last_fullsized":"2023-10-06T05:07:01.745457+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:01:11.983432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.751122+0000","last_change":"2023-10-06T05:06:57.618496+0000","last_active":"2023-10-06T05:07:01.751122+0000","last_peered":"2023-10-06T05:07:01.751122+0000","last_clean":"2023-10-06T05:07:01.751122+0000","last_became_active":"2023-10-06T05:06:57.618261+0000","last_became_peered":"2023-10-06T05:06:57.618261+0000","last_unstale":"2023-10-06T05:07:01.751122+0000","last_undegraded":"2023-10-06T05:07:01.751122+0000","last_fullsized":"2023-10-06T05:07:01.751122+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:24:33.077189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.747636+0000","last_change":"2023-10-06T05:06:57.614417+0000","last_active":"2023-10-06T05:07:01.747636+0000","last_peered":"2023-10-06T05:07:01.747636+0000","last_clean":"2023-10-06T05:07:01.747636+0000","last_became_active":"2023-10-06T05:06:57.614144+0000","last_became_peered":"2023-10-06T05:06:57.614144+0000","last_unstale":"2023-10-06T05:07:01.747636+0000","last_undegraded":"2023-10-06T05:07:01.747636+0000","last_fullsized":"2023-10-06T05:07:01.747636+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:32:33.731449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.762013+0000","last_change":"2023-10-06T05:06:57.610272+0000","last_active":"2023-10-06T05:07:01.762013+0000","last_peered":"2023-10-06T05:07:01.762013+0000","last_clean":"2023-10-06T05:07:01.762013+0000","last_became_active":"2023-10-06T05:06:57.609985+0000","last_became_peered":"2023-10-06T05:06:57.609985+0000","last_unstale":"2023-10-06T05:07:01.762013+0000","last_undegraded":"2023-10-06T05:07:01.762013+0000","last_fullsized":"2023-10-06T05:07:01.762013+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:20:04.673613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.745990+0000","last_change":"2023-10-06T05:06:57.606852+0000","last_active":"2023-10-06T05:07:01.745990+0000","last_peered":"2023-10-06T05:07:01.745990+0000","last_clean":"2023-10-06T05:07:01.745990+0000","last_became_active":"2023-10-06T05:06:57.606582+0000","last_became_peered":"2023-10-06T05:06:57.606582+0000","last_unstale":"2023-10-06T05:07:01.745990+0000","last_undegraded":"2023-10-06T05:07:01.745990+0000","last_fullsized":"2023-10-06T05:07:01.745990+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:23:03.302621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.753767+0000","last_change":"2023-10-06T05:06:57.615706+0000","last_active":"2023-10-06T05:07:01.753767+0000","last_peered":"2023-10-06T05:07:01.753767+0000","last_clean":"2023-10-06T05:07:01.753767+0000","last_became_active":"2023-10-06T05:06:57.615408+0000","last_became_peered":"2023-10-06T05:06:57.615408+0000","last_unstale":"2023-10-06T05:07:01.753767+0000","last_undegraded":"2023-10-06T05:07:01.753767+0000","last_fullsized":"2023-10-06T05:07:01.753767+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:38:51.879514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.744412+0000","last_change":"2023-10-06T05:06:57.613222+0000","last_active":"2023-10-06T05:07:01.744412+0000","last_peered":"2023-10-06T05:07:01.744412+0000","last_clean":"2023-10-06T05:07:01.744412+0000","last_became_active":"2023-10-06T05:06:57.612924+0000","last_became_peered":"2023-10-06T05:06:57.612924+0000","last_unstale":"2023-10-06T05:07:01.744412+0000","last_undegraded":"2023-10-06T05:07:01.744412+0000","last_fullsized":"2023-10-06T05:07:01.744412+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:29:19.951056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.754947+0000","last_change":"2023-10-06T05:06:57.610023+0000","last_active":"2023-10-06T05:07:01.754947+0000","last_peered":"2023-10-06T05:07:01.754947+0000","last_clean":"2023-10-06T05:07:01.754947+0000","last_became_active":"2023-10-06T05:06:57.609367+0000","last_became_peered":"2023-10-06T05:06:57.609367+0000","last_unstale":"2023-10-06T05:07:01.754947+0000","last_undegraded":"2023-10-06T05:07:01.754947+0000","last_fullsized":"2023-10-06T05:07:01.754947+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:34:55.987076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.758813+0000","last_change":"2023-10-06T05:06:57.610369+0000","last_active":"2023-10-06T05:07:01.758813+0000","last_peered":"2023-10-06T05:07:01.758813+0000","last_clean":"2023-10-06T05:07:01.758813+0000","last_became_active":"2023-10-06T05:06:57.610044+0000","last_became_peered":"2023-10-06T05:06:57.610044+0000","last_unstale":"2023-10-06T05:07:01.758813+0000","last_undegraded":"2023-10-06T05:07:01.758813+0000","last_fullsized":"2023-10-06T05:07:01.758813+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:10:12.061112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.751198+0000","last_change":"2023-10-06T05:06:57.619477+0000","last_active":"2023-10-06T05:07:01.751198+0000","last_peered":"2023-10-06T05:07:01.751198+0000","last_clean":"2023-10-06T05:07:01.751198+0000","last_became_active":"2023-10-06T05:06:57.618021+0000","last_became_peered":"2023-10-06T05:06:57.618021+0000","last_unstale":"2023-10-06T05:07:01.751198+0000","last_undegraded":"2023-10-06T05:07:01.751198+0000","last_fullsized":"2023-10-06T05:07:01.751198+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:58:02.572423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.761093+0000","last_change":"2023-10-06T05:06:57.602802+0000","last_active":"2023-10-06T05:07:01.761093+0000","last_peered":"2023-10-06T05:07:01.761093+0000","last_clean":"2023-10-06T05:07:01.761093+0000","last_became_active":"2023-10-06T05:06:57.602513+0000","last_became_peered":"2023-10-06T05:06:57.602513+0000","last_unstale":"2023-10-06T05:07:01.761093+0000","last_undegraded":"2023-10-06T05:07:01.761093+0000","last_fullsized":"2023-10-06T05:07:01.761093+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T16:34:15.990651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.747139+0000","last_change":"2023-10-06T05:06:57.608190+0000","last_active":"2023-10-06T05:07:01.747139+0000","last_peered":"2023-10-06T05:07:01.747139+0000","last_clean":"2023-10-06T05:07:01.747139+0000","last_became_active":"2023-10-06T05:06:57.607936+0000","last_became_peered":"2023-10-06T05:06:57.607936+0000","last_unstale":"2023-10-06T05:07:01.747139+0000","last_undegraded":"2023-10-06T05:07:01.747139+0000","last_fullsized":"2023-10-06T05:07:01.747139+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:00:30.035770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.761424+0000","last_change":"2023-10-06T05:06:57.611226+0000","last_active":"2023-10-06T05:07:01.761424+0000","last_peered":"2023-10-06T05:07:01.761424+0000","last_clean":"2023-10-06T05:07:01.761424+0000","last_became_active":"2023-10-06T05:06:57.610931+0000","last_became_peered":"2023-10-06T05:06:57.610931+0000","last_unstale":"2023-10-06T05:07:01.761424+0000","last_undegraded":"2023-10-06T05:07:01.761424+0000","last_fullsized":"2023-10-06T05:07:01.761424+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:52:14.772801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.749471+0000","last_change":"2023-10-06T05:06:57.615682+0000","last_active":"2023-10-06T05:07:01.749471+0000","last_peered":"2023-10-06T05:07:01.749471+0000","last_clean":"2023-10-06T05:07:01.749471+0000","last_became_active":"2023-10-06T05:06:57.615408+0000","last_became_peered":"2023-10-06T05:06:57.615408+0000","last_unstale":"2023-10-06T05:07:01.749471+0000","last_undegraded":"2023-10-06T05:07:01.749471+0000","last_fullsized":"2023-10-06T05:07:01.749471+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:58:02.572423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.749868+0000","last_change":"2023-10-06T05:06:57.619795+0000","last_active":"2023-10-06T05:07:01.749868+0000","last_peered":"2023-10-06T05:07:01.749868+0000","last_clean":"2023-10-06T05:07:01.749868+0000","last_became_active":"2023-10-06T05:06:57.619533+0000","last_became_peered":"2023-10-06T05:06:57.619533+0000","last_unstale":"2023-10-06T05:07:01.749868+0000","last_undegraded":"2023-10-06T05:07:01.749868+0000","last_fullsized":"2023-10-06T05:07:01.749868+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:11:00.525585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750332+0000","last_change":"2023-10-06T05:06:57.617926+0000","last_active":"2023-10-06T05:07:01.750332+0000","last_peered":"2023-10-06T05:07:01.750332+0000","last_clean":"2023-10-06T05:07:01.750332+0000","last_became_active":"2023-10-06T05:06:57.617128+0000","last_became_peered":"2023-10-06T05:06:57.617128+0000","last_unstale":"2023-10-06T05:07:01.750332+0000","last_undegraded":"2023-10-06T05:07:01.750332+0000","last_fullsized":"2023-10-06T05:07:01.750332+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:38:16.871694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.748479+0000","last_change":"2023-10-06T05:06:57.620090+0000","last_active":"2023-10-06T05:07:01.748479+0000","last_peered":"2023-10-06T05:07:01.748479+0000","last_clean":"2023-10-06T05:07:01.748479+0000","last_became_active":"2023-10-06T05:06:57.619892+0000","last_became_peered":"2023-10-06T05:06:57.619892+0000","last_unstale":"2023-10-06T05:07:01.748479+0000","last_undegraded":"2023-10-06T05:07:01.748479+0000","last_fullsized":"2023-10-06T05:07:01.748479+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:45:23.272110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750371+0000","last_change":"2023-10-06T05:06:57.613844+0000","last_active":"2023-10-06T05:07:01.750371+0000","last_peered":"2023-10-06T05:07:01.750371+0000","last_clean":"2023-10-06T05:07:01.750371+0000","last_became_active":"2023-10-06T05:06:57.613513+0000","last_became_peered":"2023-10-06T05:06:57.613513+0000","last_unstale":"2023-10-06T05:07:01.750371+0000","last_undegraded":"2023-10-06T05:07:01.750371+0000","last_fullsized":"2023-10-06T05:07:01.750371+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:53:03.171108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.748759+0000","last_change":"2023-10-06T05:06:57.612464+0000","last_active":"2023-10-06T05:07:01.748759+0000","last_peered":"2023-10-06T05:07:01.748759+0000","last_clean":"2023-10-06T05:07:01.748759+0000","last_became_active":"2023-10-06T05:06:57.612179+0000","last_became_peered":"2023-10-06T05:06:57.612179+0000","last_unstale":"2023-10-06T05:07:01.748759+0000","last_undegraded":"2023-10-06T05:07:01.748759+0000","last_fullsized":"2023-10-06T05:07:01.748759+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:12:34.550298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.744832+0000","last_change":"2023-10-06T05:06:57.612625+0000","last_active":"2023-10-06T05:07:01.744832+0000","last_peered":"2023-10-06T05:07:01.744832+0000","last_clean":"2023-10-06T05:07:01.744832+0000","last_became_active":"2023-10-06T05:06:57.611582+0000","last_became_peered":"2023-10-06T05:06:57.611582+0000","last_unstale":"2023-10-06T05:07:01.744832+0000","last_undegraded":"2023-10-06T05:07:01.744832+0000","last_fullsized":"2023-10-06T05:07:01.744832+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:59:39.994965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.768197+0000","last_change":"2023-10-06T05:07:01.765965+0000","last_active":"2023-10-06T05:07:01.768197+0000","last_peered":"2023-10-06T05:07:01.768197+0000","last_clean":"2023-10-06T05:07:01.768197+0000","last_became_active":"2023-10-06T05:07:01.764635+0000","last_became_peered":"2023-10-06T05:07:01.764635+0000","last_unstale":"2023-10-06T05:07:01.768197+0000","last_undegraded":"2023-10-06T05:07:01.768197+0000","last_fullsized":"2023-10-06T05:07:01.768197+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:18:43.042335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2,0],"acting":[4,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"24'1","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.760983+0000","last_change":"2023-10-06T05:06:56.716498+0000","last_active":"2023-10-06T05:07:01.760983+0000","last_peered":"2023-10-06T05:07:01.760983+0000","last_clean":"2023-10-06T05:07:01.760983+0000","last_became_active":"2023-10-06T05:06:56.714771+0000","last_became_peered":"2023-10-06T05:06:56.714771+0000","last_unstale":"2023-10-06T05:07:01.760983+0000","last_undegraded":"2023-10-06T05:07:01.760983+0000","last_fullsized":"2023-10-06T05:07:01.760983+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:53:56.989908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.752793+0000","last_change":"2023-10-06T05:06:57.617712+0000","last_active":"2023-10-06T05:07:01.752793+0000","last_peered":"2023-10-06T05:07:01.752793+0000","last_clean":"2023-10-06T05:07:01.752793+0000","last_became_active":"2023-10-06T05:06:57.616946+0000","last_became_peered":"2023-10-06T05:06:57.616946+0000","last_unstale":"2023-10-06T05:07:01.752793+0000","last_undegraded":"2023-10-06T05:07:01.752793+0000","last_fullsized":"2023-10-06T05:07:01.752793+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:21:13.192156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.767576+0000","last_change":"2023-10-06T05:07:01.763425+0000","last_active":"2023-10-06T05:07:01.767576+0000","last_peered":"2023-10-06T05:07:01.767576+0000","last_clean":"2023-10-06T05:07:01.767576+0000","last_became_active":"2023-10-06T05:07:01.763035+0000","last_became_peered":"2023-10-06T05:07:01.763035+0000","last_unstale":"2023-10-06T05:07:01.767576+0000","last_undegraded":"2023-10-06T05:07:01.767576+0000","last_fullsized":"2023-10-06T05:07:01.767576+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:51:19.615655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3,7],"acting":[6,4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.745972+0000","last_change":"2023-10-06T05:06:56.714205+0000","last_active":"2023-10-06T05:07:01.745972+0000","last_peered":"2023-10-06T05:07:01.745972+0000","last_clean":"2023-10-06T05:07:01.745972+0000","last_became_active":"2023-10-06T05:06:56.713947+0000","last_became_peered":"2023-10-06T05:06:56.713947+0000","last_unstale":"2023-10-06T05:07:01.745972+0000","last_undegraded":"2023-10-06T05:07:01.745972+0000","last_fullsized":"2023-10-06T05:07:01.745972+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:54:07.584005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750046+0000","last_change":"2023-10-06T05:06:57.616019+0000","last_active":"2023-10-06T05:07:01.750046+0000","last_peered":"2023-10-06T05:07:01.750046+0000","last_clean":"2023-10-06T05:07:01.750046+0000","last_became_active":"2023-10-06T05:06:57.615747+0000","last_became_peered":"2023-10-06T05:06:57.615747+0000","last_unstale":"2023-10-06T05:07:01.750046+0000","last_undegraded":"2023-10-06T05:07:01.750046+0000","last_fullsized":"2023-10-06T05:07:01.750046+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:07:52.020683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.770991+0000","last_change":"2023-10-06T05:07:01.767383+0000","last_active":"2023-10-06T05:07:01.770991+0000","last_peered":"2023-10-06T05:07:01.770991+0000","last_clean":"2023-10-06T05:07:01.770991+0000","last_became_active":"2023-10-06T05:07:01.766997+0000","last_became_peered":"2023-10-06T05:07:01.766997+0000","last_unstale":"2023-10-06T05:07:01.770991+0000","last_undegraded":"2023-10-06T05:07:01.770991+0000","last_fullsized":"2023-10-06T05:07:01.770991+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:45:03.209108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1,2],"acting":[0,7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.749601+0000","last_change":"2023-10-06T05:06:56.706133+0000","last_active":"2023-10-06T05:07:01.749601+0000","last_peered":"2023-10-06T05:07:01.749601+0000","last_clean":"2023-10-06T05:07:01.749601+0000","last_became_active":"2023-10-06T05:06:56.705867+0000","last_became_peered":"2023-10-06T05:06:56.705867+0000","last_unstale":"2023-10-06T05:07:01.749601+0000","last_undegraded":"2023-10-06T05:07:01.749601+0000","last_fullsized":"2023-10-06T05:07:01.749601+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:55:08.124129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.751243+0000","last_change":"2023-10-06T05:06:57.608813+0000","last_active":"2023-10-06T05:07:01.751243+0000","last_peered":"2023-10-06T05:07:01.751243+0000","last_clean":"2023-10-06T05:07:01.751243+0000","last_became_active":"2023-10-06T05:06:57.608497+0000","last_became_peered":"2023-10-06T05:06:57.608497+0000","last_unstale":"2023-10-06T05:07:01.751243+0000","last_undegraded":"2023-10-06T05:07:01.751243+0000","last_fullsized":"2023-10-06T05:07:01.751243+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:30:19.509164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.770179+0000","last_change":"2023-10-06T05:07:01.765801+0000","last_active":"2023-10-06T05:07:01.770179+0000","last_peered":"2023-10-06T05:07:01.770179+0000","last_clean":"2023-10-06T05:07:01.770179+0000","last_became_active":"2023-10-06T05:07:01.765422+0000","last_became_peered":"2023-10-06T05:07:01.765422+0000","last_unstale":"2023-10-06T05:07:01.770179+0000","last_undegraded":"2023-10-06T05:07:01.770179+0000","last_fullsized":"2023-10-06T05:07:01.770179+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:20:08.719247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7,1],"acting":[6,4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.751276+0000","last_change":"2023-10-06T05:06:56.714826+0000","last_active":"2023-10-06T05:07:01.751276+0000","last_peered":"2023-10-06T05:07:01.751276+0000","last_clean":"2023-10-06T05:07:01.751276+0000","last_became_active":"2023-10-06T05:06:56.714564+0000","last_became_peered":"2023-10-06T05:06:56.714564+0000","last_unstale":"2023-10-06T05:07:01.751276+0000","last_undegraded":"2023-10-06T05:07:01.751276+0000","last_fullsized":"2023-10-06T05:07:01.751276+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:19:13.930707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.749520+0000","last_change":"2023-10-06T05:06:57.618934+0000","last_active":"2023-10-06T05:07:01.749520+0000","last_peered":"2023-10-06T05:07:01.749520+0000","last_clean":"2023-10-06T05:07:01.749520+0000","last_became_active":"2023-10-06T05:06:57.618410+0000","last_became_peered":"2023-10-06T05:06:57.618410+0000","last_unstale":"2023-10-06T05:07:01.749520+0000","last_undegraded":"2023-10-06T05:07:01.749520+0000","last_fullsized":"2023-10-06T05:07:01.749520+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:51:15.570020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.775608+0000","last_change":"2023-10-06T05:07:01.766370+0000","last_active":"2023-10-06T05:07:01.775608+0000","last_peered":"2023-10-06T05:07:01.775608+0000","last_clean":"2023-10-06T05:07:01.775608+0000","last_became_active":"2023-10-06T05:07:01.765994+0000","last_became_peered":"2023-10-06T05:07:01.765994+0000","last_unstale":"2023-10-06T05:07:01.775608+0000","last_undegraded":"2023-10-06T05:07:01.775608+0000","last_fullsized":"2023-10-06T05:07:01.775608+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:20:46.564009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6,0],"acting":[7,4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.747549+0000","last_change":"2023-10-06T05:06:56.708479+0000","last_active":"2023-10-06T05:07:01.747549+0000","last_peered":"2023-10-06T05:07:01.747549+0000","last_clean":"2023-10-06T05:07:01.747549+0000","last_became_active":"2023-10-06T05:06:56.708155+0000","last_became_peered":"2023-10-06T05:06:56.708155+0000","last_unstale":"2023-10-06T05:07:01.747549+0000","last_undegraded":"2023-10-06T05:07:01.747549+0000","last_fullsized":"2023-10-06T05:07:01.747549+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:30:18.507322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750417+0000","last_change":"2023-10-06T05:06:57.617995+0000","last_active":"2023-10-06T05:07:01.750417+0000","last_peered":"2023-10-06T05:07:01.750417+0000","last_clean":"2023-10-06T05:07:01.750417+0000","last_became_active":"2023-10-06T05:06:57.617577+0000","last_became_peered":"2023-10-06T05:06:57.617577+0000","last_unstale":"2023-10-06T05:07:01.750417+0000","last_undegraded":"2023-10-06T05:07:01.750417+0000","last_fullsized":"2023-10-06T05:07:01.750417+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:01:11.983432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.763161+0000","last_change":"2023-10-06T05:07:01.761436+0000","last_active":"2023-10-06T05:07:01.763161+0000","last_peered":"2023-10-06T05:07:01.763161+0000","last_clean":"2023-10-06T05:07:01.763161+0000","last_became_active":"2023-10-06T05:07:01.761049+0000","last_became_peered":"2023-10-06T05:07:01.761049+0000","last_unstale":"2023-10-06T05:07:01.763161+0000","last_undegraded":"2023-10-06T05:07:01.763161+0000","last_fullsized":"2023-10-06T05:07:01.763161+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:09:13.387478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,5],"acting":[3,0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"24'3","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.753446+0000","last_change":"2023-10-06T05:06:56.717320+0000","last_active":"2023-10-06T05:07:01.753446+0000","last_peered":"2023-10-06T05:07:01.753446+0000","last_clean":"2023-10-06T05:07:01.753446+0000","last_became_active":"2023-10-06T05:06:56.716985+0000","last_became_peered":"2023-10-06T05:06:56.716985+0000","last_unstale":"2023-10-06T05:07:01.753446+0000","last_undegraded":"2023-10-06T05:07:01.753446+0000","last_fullsized":"2023-10-06T05:07:01.753446+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:53:02.169267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.747499+0000","last_change":"2023-10-06T05:06:57.613655+0000","last_active":"2023-10-06T05:07:01.747499+0000","last_peered":"2023-10-06T05:07:01.747499+0000","last_clean":"2023-10-06T05:07:01.747499+0000","last_became_active":"2023-10-06T05:06:57.613378+0000","last_became_peered":"2023-10-06T05:06:57.613378+0000","last_unstale":"2023-10-06T05:07:01.747499+0000","last_undegraded":"2023-10-06T05:07:01.747499+0000","last_fullsized":"2023-10-06T05:07:01.747499+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:23:03.919621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.772389+0000","last_change":"2023-10-06T05:07:01.766400+0000","last_active":"2023-10-06T05:07:01.772389+0000","last_peered":"2023-10-06T05:07:01.772389+0000","last_clean":"2023-10-06T05:07:01.772389+0000","last_became_active":"2023-10-06T05:07:01.766034+0000","last_became_peered":"2023-10-06T05:07:01.766034+0000","last_unstale":"2023-10-06T05:07:01.772389+0000","last_undegraded":"2023-10-06T05:07:01.772389+0000","last_fullsized":"2023-10-06T05:07:01.772389+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:04:14.761320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3,7],"acting":[1,5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"29'9","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-06T05:07:07.500759+0000","last_change":"2023-10-06T05:06:56.716406+0000","last_active":"2023-10-06T05:07:07.500759+0000","last_peered":"2023-10-06T05:07:07.500759+0000","last_clean":"2023-10-06T05:07:07.500759+0000","last_became_active":"2023-10-06T05:06:56.716075+0000","last_became_peered":"2023-10-06T05:06:56.716075+0000","last_unstale":"2023-10-06T05:07:07.500759+0000","last_undegraded":"2023-10-06T05:07:07.500759+0000","last_fullsized":"2023-10-06T05:07:07.500759+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:36:39.323759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3714,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.761355+0000","last_change":"2023-10-06T05:06:57.605798+0000","last_active":"2023-10-06T05:07:01.761355+0000","last_peered":"2023-10-06T05:07:01.761355+0000","last_clean":"2023-10-06T05:07:01.761355+0000","last_became_active":"2023-10-06T05:06:57.605523+0000","last_became_peered":"2023-10-06T05:06:57.605523+0000","last_unstale":"2023-10-06T05:07:01.761355+0000","last_undegraded":"2023-10-06T05:07:01.761355+0000","last_fullsized":"2023-10-06T05:07:01.761355+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:32:33.731449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.765132+0000","last_change":"2023-10-06T05:07:01.760576+0000","last_active":"2023-10-06T05:07:01.765132+0000","last_peered":"2023-10-06T05:07:01.765132+0000","last_clean":"2023-10-06T05:07:01.765132+0000","last_became_active":"2023-10-06T05:07:01.760146+0000","last_became_peered":"2023-10-06T05:07:01.760146+0000","last_unstale":"2023-10-06T05:07:01.765132+0000","last_undegraded":"2023-10-06T05:07:01.765132+0000","last_fullsized":"2023-10-06T05:07:01.765132+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:30:21.024442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3,0],"acting":[5,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"24'2","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.748025+0000","last_change":"2023-10-06T05:06:56.722076+0000","last_active":"2023-10-06T05:07:01.748025+0000","last_peered":"2023-10-06T05:07:01.748025+0000","last_clean":"2023-10-06T05:07:01.748025+0000","last_became_active":"2023-10-06T05:06:56.721786+0000","last_became_peered":"2023-10-06T05:06:56.721786+0000","last_unstale":"2023-10-06T05:07:01.748025+0000","last_undegraded":"2023-10-06T05:07:01.748025+0000","last_fullsized":"2023-10-06T05:07:01.748025+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:15:54.616807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.749669+0000","last_change":"2023-10-06T05:06:57.620208+0000","last_active":"2023-10-06T05:07:01.749669+0000","last_peered":"2023-10-06T05:07:01.749669+0000","last_clean":"2023-10-06T05:07:01.749669+0000","last_became_active":"2023-10-06T05:06:57.619747+0000","last_became_peered":"2023-10-06T05:06:57.619747+0000","last_unstale":"2023-10-06T05:07:01.749669+0000","last_undegraded":"2023-10-06T05:07:01.749669+0000","last_fullsized":"2023-10-06T05:07:01.749669+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:52:14.772801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.764815+0000","last_change":"2023-10-06T05:07:01.762230+0000","last_active":"2023-10-06T05:07:01.764815+0000","last_peered":"2023-10-06T05:07:01.764815+0000","last_clean":"2023-10-06T05:07:01.764815+0000","last_became_active":"2023-10-06T05:07:01.761878+0000","last_became_peered":"2023-10-06T05:07:01.761878+0000","last_unstale":"2023-10-06T05:07:01.764815+0000","last_undegraded":"2023-10-06T05:07:01.764815+0000","last_fullsized":"2023-10-06T05:07:01.764815+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:18:24.586582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2,1],"acting":[3,7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.749791+0000","last_change":"2023-10-06T05:06:56.715016+0000","last_active":"2023-10-06T05:07:01.749791+0000","last_peered":"2023-10-06T05:07:01.749791+0000","last_clean":"2023-10-06T05:07:01.749791+0000","last_became_active":"2023-10-06T05:06:56.714513+0000","last_became_peered":"2023-10-06T05:06:56.714513+0000","last_unstale":"2023-10-06T05:07:01.749791+0000","last_undegraded":"2023-10-06T05:07:01.749791+0000","last_fullsized":"2023-10-06T05:07:01.749791+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T16:11:31.404755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.744725+0000","last_change":"2023-10-06T05:06:57.613062+0000","last_active":"2023-10-06T05:07:01.744725+0000","last_peered":"2023-10-06T05:07:01.744725+0000","last_clean":"2023-10-06T05:07:01.744725+0000","last_became_active":"2023-10-06T05:06:57.611524+0000","last_became_peered":"2023-10-06T05:06:57.611524+0000","last_unstale":"2023-10-06T05:07:01.744725+0000","last_undegraded":"2023-10-06T05:07:01.744725+0000","last_fullsized":"2023-10-06T05:07:01.744725+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:01:11.480811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.770644+0000","last_change":"2023-10-06T05:07:01.764987+0000","last_active":"2023-10-06T05:07:01.770644+0000","last_peered":"2023-10-06T05:07:01.770644+0000","last_clean":"2023-10-06T05:07:01.770644+0000","last_became_active":"2023-10-06T05:07:01.764475+0000","last_became_peered":"2023-10-06T05:07:01.764475+0000","last_unstale":"2023-10-06T05:07:01.770644+0000","last_undegraded":"2023-10-06T05:07:01.770644+0000","last_fullsized":"2023-10-06T05:07:01.770644+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:25:41.281713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,0],"acting":[3,1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.755037+0000","last_change":"2023-10-06T05:06:56.719174+0000","last_active":"2023-10-06T05:07:01.755037+0000","last_peered":"2023-10-06T05:07:01.755037+0000","last_clean":"2023-10-06T05:07:01.755037+0000","last_became_active":"2023-10-06T05:06:56.718930+0000","last_became_peered":"2023-10-06T05:06:56.718930+0000","last_unstale":"2023-10-06T05:07:01.755037+0000","last_undegraded":"2023-10-06T05:07:01.755037+0000","last_fullsized":"2023-10-06T05:07:01.755037+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:51:30.871635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750114+0000","last_change":"2023-10-06T05:06:57.617389+0000","last_active":"2023-10-06T05:07:01.750114+0000","last_peered":"2023-10-06T05:07:01.750114+0000","last_clean":"2023-10-06T05:07:01.750114+0000","last_became_active":"2023-10-06T05:06:57.617099+0000","last_became_peered":"2023-10-06T05:06:57.617099+0000","last_unstale":"2023-10-06T05:07:01.750114+0000","last_undegraded":"2023-10-06T05:07:01.750114+0000","last_fullsized":"2023-10-06T05:07:01.750114+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:51:15.929322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.759618+0000","last_change":"2023-10-06T05:07:01.756721+0000","last_active":"2023-10-06T05:07:01.759618+0000","last_peered":"2023-10-06T05:07:01.759618+0000","last_clean":"2023-10-06T05:07:01.759618+0000","last_became_active":"2023-10-06T05:07:01.756335+0000","last_became_peered":"2023-10-06T05:07:01.756335+0000","last_unstale":"2023-10-06T05:07:01.759618+0000","last_undegraded":"2023-10-06T05:07:01.759618+0000","last_fullsized":"2023-10-06T05:07:01.759618+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:20:29.759167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1,6],"acting":[5,3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"24'8","reported_seq":22,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750615+0000","last_change":"2023-10-06T05:06:56.713625+0000","last_active":"2023-10-06T05:07:01.750615+0000","last_peered":"2023-10-06T05:07:01.750615+0000","last_clean":"2023-10-06T05:07:01.750615+0000","last_became_active":"2023-10-06T05:06:56.713346+0000","last_became_peered":"2023-10-06T05:06:56.713346+0000","last_unstale":"2023-10-06T05:07:01.750615+0000","last_undegraded":"2023-10-06T05:07:01.750615+0000","last_fullsized":"2023-10-06T05:07:01.750615+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:57:59.037083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750303+0000","last_change":"2023-10-06T05:06:57.617670+0000","last_active":"2023-10-06T05:07:01.750303+0000","last_peered":"2023-10-06T05:07:01.750303+0000","last_clean":"2023-10-06T05:07:01.750303+0000","last_became_active":"2023-10-06T05:06:57.617259+0000","last_became_peered":"2023-10-06T05:06:57.617259+0000","last_unstale":"2023-10-06T05:07:01.750303+0000","last_undegraded":"2023-10-06T05:07:01.750303+0000","last_fullsized":"2023-10-06T05:07:01.750303+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:22:11.051575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.775868+0000","last_change":"2023-10-06T05:07:01.771047+0000","last_active":"2023-10-06T05:07:01.775868+0000","last_peered":"2023-10-06T05:07:01.775868+0000","last_clean":"2023-10-06T05:07:01.775868+0000","last_became_active":"2023-10-06T05:07:01.770692+0000","last_became_peered":"2023-10-06T05:07:01.770692+0000","last_unstale":"2023-10-06T05:07:01.775868+0000","last_undegraded":"2023-10-06T05:07:01.775868+0000","last_fullsized":"2023-10-06T05:07:01.775868+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:36:04.280555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3,4],"acting":[7,6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"24'1","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.752603+0000","last_change":"2023-10-06T05:06:56.722082+0000","last_active":"2023-10-06T05:07:01.752603+0000","last_peered":"2023-10-06T05:07:01.752603+0000","last_clean":"2023-10-06T05:07:01.752603+0000","last_became_active":"2023-10-06T05:06:56.721758+0000","last_became_peered":"2023-10-06T05:06:56.721758+0000","last_unstale":"2023-10-06T05:07:01.752603+0000","last_undegraded":"2023-10-06T05:07:01.752603+0000","last_fullsized":"2023-10-06T05:07:01.752603+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:10:26.413910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.745354+0000","last_change":"2023-10-06T05:06:57.611665+0000","last_active":"2023-10-06T05:07:01.745354+0000","last_peered":"2023-10-06T05:07:01.745354+0000","last_clean":"2023-10-06T05:07:01.745354+0000","last_became_active":"2023-10-06T05:06:57.611226+0000","last_became_peered":"2023-10-06T05:06:57.611226+0000","last_unstale":"2023-10-06T05:07:01.745354+0000","last_undegraded":"2023-10-06T05:07:01.745354+0000","last_fullsized":"2023-10-06T05:07:01.745354+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:26:29.348498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.761343+0000","last_change":"2023-10-06T05:07:01.760161+0000","last_active":"2023-10-06T05:07:01.761343+0000","last_peered":"2023-10-06T05:07:01.761343+0000","last_clean":"2023-10-06T05:07:01.761343+0000","last_became_active":"2023-10-06T05:07:01.759742+0000","last_became_peered":"2023-10-06T05:07:01.759742+0000","last_unstale":"2023-10-06T05:07:01.761343+0000","last_undegraded":"2023-10-06T05:07:01.761343+0000","last_fullsized":"2023-10-06T05:07:01.761343+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:10:16.106747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1,5],"acting":[2,0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"24'2","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.751770+0000","last_change":"2023-10-06T05:06:56.722004+0000","last_active":"2023-10-06T05:07:01.751770+0000","last_peered":"2023-10-06T05:07:01.751770+0000","last_clean":"2023-10-06T05:07:01.751770+0000","last_became_active":"2023-10-06T05:06:56.721717+0000","last_became_peered":"2023-10-06T05:06:56.721717+0000","last_unstale":"2023-10-06T05:07:01.751770+0000","last_undegraded":"2023-10-06T05:07:01.751770+0000","last_fullsized":"2023-10-06T05:07:01.751770+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:50:35.786032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.761552+0000","last_change":"2023-10-06T05:06:57.609279+0000","last_active":"2023-10-06T05:07:01.761552+0000","last_peered":"2023-10-06T05:07:01.761552+0000","last_clean":"2023-10-06T05:07:01.761552+0000","last_became_active":"2023-10-06T05:06:57.608982+0000","last_became_peered":"2023-10-06T05:06:57.608982+0000","last_unstale":"2023-10-06T05:07:01.761552+0000","last_undegraded":"2023-10-06T05:07:01.761552+0000","last_fullsized":"2023-10-06T05:07:01.761552+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:38:16.871694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.765310+0000","last_change":"2023-10-06T05:07:01.762960+0000","last_active":"2023-10-06T05:07:01.765310+0000","last_peered":"2023-10-06T05:07:01.765310+0000","last_clean":"2023-10-06T05:07:01.765310+0000","last_became_active":"2023-10-06T05:07:01.762308+0000","last_became_peered":"2023-10-06T05:07:01.762308+0000","last_unstale":"2023-10-06T05:07:01.765310+0000","last_undegraded":"2023-10-06T05:07:01.765310+0000","last_fullsized":"2023-10-06T05:07:01.765310+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:49:52.354792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5,6],"acting":[2,0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"24'1","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.754352+0000","last_change":"2023-10-06T05:06:56.710907+0000","last_active":"2023-10-06T05:07:01.754352+0000","last_peered":"2023-10-06T05:07:01.754352+0000","last_clean":"2023-10-06T05:07:01.754352+0000","last_became_active":"2023-10-06T05:06:56.710595+0000","last_became_peered":"2023-10-06T05:06:56.710595+0000","last_unstale":"2023-10-06T05:07:01.754352+0000","last_undegraded":"2023-10-06T05:07:01.754352+0000","last_fullsized":"2023-10-06T05:07:01.754352+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:17:31.213115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.751968+0000","last_change":"2023-10-06T05:06:57.611226+0000","last_active":"2023-10-06T05:07:01.751968+0000","last_peered":"2023-10-06T05:07:01.751968+0000","last_clean":"2023-10-06T05:07:01.751968+0000","last_became_active":"2023-10-06T05:06:57.610966+0000","last_became_peered":"2023-10-06T05:06:57.610966+0000","last_unstale":"2023-10-06T05:07:01.751968+0000","last_undegraded":"2023-10-06T05:07:01.751968+0000","last_fullsized":"2023-10-06T05:07:01.751968+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:15:48.746279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.768101+0000","last_change":"2023-10-06T05:07:01.764757+0000","last_active":"2023-10-06T05:07:01.768101+0000","last_peered":"2023-10-06T05:07:01.768101+0000","last_clean":"2023-10-06T05:07:01.768101+0000","last_became_active":"2023-10-06T05:07:01.762819+0000","last_became_peered":"2023-10-06T05:07:01.762819+0000","last_unstale":"2023-10-06T05:07:01.768101+0000","last_undegraded":"2023-10-06T05:07:01.768101+0000","last_fullsized":"2023-10-06T05:07:01.768101+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:21:17.237790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3,6],"acting":[2,4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.745471+0000","last_change":"2023-10-06T05:06:56.704818+0000","last_active":"2023-10-06T05:07:01.745471+0000","last_peered":"2023-10-06T05:07:01.745471+0000","last_clean":"2023-10-06T05:07:01.745471+0000","last_became_active":"2023-10-06T05:06:56.704596+0000","last_became_peered":"2023-10-06T05:06:56.704596+0000","last_unstale":"2023-10-06T05:07:01.745471+0000","last_undegraded":"2023-10-06T05:07:01.745471+0000","last_fullsized":"2023-10-06T05:07:01.745471+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:23:02.300780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750614+0000","last_change":"2023-10-06T05:06:57.615733+0000","last_active":"2023-10-06T05:07:01.750614+0000","last_peered":"2023-10-06T05:07:01.750614+0000","last_clean":"2023-10-06T05:07:01.750614+0000","last_became_active":"2023-10-06T05:06:57.615061+0000","last_became_peered":"2023-10-06T05:06:57.615061+0000","last_unstale":"2023-10-06T05:07:01.750614+0000","last_undegraded":"2023-10-06T05:07:01.750614+0000","last_fullsized":"2023-10-06T05:07:01.750614+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:45:23.272110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.768073+0000","last_change":"2023-10-06T05:07:01.763754+0000","last_active":"2023-10-06T05:07:01.768073+0000","last_peered":"2023-10-06T05:07:01.768073+0000","last_clean":"2023-10-06T05:07:01.768073+0000","last_became_active":"2023-10-06T05:07:01.763392+0000","last_became_peered":"2023-10-06T05:07:01.763392+0000","last_unstale":"2023-10-06T05:07:01.768073+0000","last_undegraded":"2023-10-06T05:07:01.768073+0000","last_fullsized":"2023-10-06T05:07:01.768073+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:38:20.917328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5,6],"acting":[2,7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"24'2","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750935+0000","last_change":"2023-10-06T05:06:56.717936+0000","last_active":"2023-10-06T05:07:01.750935+0000","last_peered":"2023-10-06T05:07:01.750935+0000","last_clean":"2023-10-06T05:07:01.750935+0000","last_became_active":"2023-10-06T05:06:56.717683+0000","last_became_peered":"2023-10-06T05:06:56.717683+0000","last_unstale":"2023-10-06T05:07:01.750935+0000","last_undegraded":"2023-10-06T05:07:01.750935+0000","last_fullsized":"2023-10-06T05:07:01.750935+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:23:02.300780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.760808+0000","last_change":"2023-10-06T05:06:57.617318+0000","last_active":"2023-10-06T05:07:01.760808+0000","last_peered":"2023-10-06T05:07:01.760808+0000","last_clean":"2023-10-06T05:07:01.760808+0000","last_became_active":"2023-10-06T05:06:57.617044+0000","last_became_peered":"2023-10-06T05:06:57.617044+0000","last_unstale":"2023-10-06T05:07:01.760808+0000","last_undegraded":"2023-10-06T05:07:01.760808+0000","last_fullsized":"2023-10-06T05:07:01.760808+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:51:15.570020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.772045+0000","last_change":"2023-10-06T05:07:01.762386+0000","last_active":"2023-10-06T05:07:01.772045+0000","last_peered":"2023-10-06T05:07:01.772045+0000","last_clean":"2023-10-06T05:07:01.772045+0000","last_became_active":"2023-10-06T05:07:01.761040+0000","last_became_peered":"2023-10-06T05:07:01.761040+0000","last_unstale":"2023-10-06T05:07:01.772045+0000","last_undegraded":"2023-10-06T05:07:01.772045+0000","last_fullsized":"2023-10-06T05:07:01.772045+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:11:22.468821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0,6],"acting":[1,4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.752055+0000","last_change":"2023-10-06T05:06:56.723332+0000","last_active":"2023-10-06T05:07:01.752055+0000","last_peered":"2023-10-06T05:07:01.752055+0000","last_clean":"2023-10-06T05:07:01.752055+0000","last_became_active":"2023-10-06T05:06:56.723023+0000","last_became_peered":"2023-10-06T05:06:56.723023+0000","last_unstale":"2023-10-06T05:07:01.752055+0000","last_undegraded":"2023-10-06T05:07:01.752055+0000","last_fullsized":"2023-10-06T05:07:01.752055+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:36:39.323759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.761806+0000","last_change":"2023-10-06T05:06:57.609386+0000","last_active":"2023-10-06T05:07:01.761806+0000","last_peered":"2023-10-06T05:07:01.761806+0000","last_clean":"2023-10-06T05:07:01.761806+0000","last_became_active":"2023-10-06T05:06:57.609019+0000","last_became_peered":"2023-10-06T05:06:57.609019+0000","last_unstale":"2023-10-06T05:07:01.761806+0000","last_undegraded":"2023-10-06T05:07:01.761806+0000","last_fullsized":"2023-10-06T05:07:01.761806+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:20:18.722541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.767691+0000","last_change":"2023-10-06T05:07:01.763186+0000","last_active":"2023-10-06T05:07:01.767691+0000","last_peered":"2023-10-06T05:07:01.767691+0000","last_clean":"2023-10-06T05:07:01.767691+0000","last_became_active":"2023-10-06T05:07:01.762259+0000","last_became_peered":"2023-10-06T05:07:01.762259+0000","last_unstale":"2023-10-06T05:07:01.767691+0000","last_undegraded":"2023-10-06T05:07:01.767691+0000","last_fullsized":"2023-10-06T05:07:01.767691+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:46:13.640987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6,2],"acting":[5,4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.761404+0000","last_change":"2023-10-06T05:06:56.722507+0000","last_active":"2023-10-06T05:07:01.761404+0000","last_peered":"2023-10-06T05:07:01.761404+0000","last_clean":"2023-10-06T05:07:01.761404+0000","last_became_active":"2023-10-06T05:06:56.722243+0000","last_became_peered":"2023-10-06T05:06:56.722243+0000","last_unstale":"2023-10-06T05:07:01.761404+0000","last_undegraded":"2023-10-06T05:07:01.761404+0000","last_fullsized":"2023-10-06T05:07:01.761404+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:57:59.037083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.761170+0000","last_change":"2023-10-06T05:06:57.606979+0000","last_active":"2023-10-06T05:07:01.761170+0000","last_peered":"2023-10-06T05:07:01.761170+0000","last_clean":"2023-10-06T05:07:01.761170+0000","last_became_active":"2023-10-06T05:06:57.606633+0000","last_became_peered":"2023-10-06T05:06:57.606633+0000","last_unstale":"2023-10-06T05:07:01.761170+0000","last_undegraded":"2023-10-06T05:07:01.761170+0000","last_fullsized":"2023-10-06T05:07:01.761170+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:21:13.192156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.768126+0000","last_change":"2023-10-06T05:07:01.765104+0000","last_active":"2023-10-06T05:07:01.768126+0000","last_peered":"2023-10-06T05:07:01.768126+0000","last_clean":"2023-10-06T05:07:01.768126+0000","last_became_active":"2023-10-06T05:07:01.764692+0000","last_became_peered":"2023-10-06T05:07:01.764692+0000","last_unstale":"2023-10-06T05:07:01.768126+0000","last_undegraded":"2023-10-06T05:07:01.768126+0000","last_fullsized":"2023-10-06T05:07:01.768126+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:40:47.847563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0,2],"acting":[4,5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.749439+0000","last_change":"2023-10-06T05:06:56.708403+0000","last_active":"2023-10-06T05:07:01.749439+0000","last_peered":"2023-10-06T05:07:01.749439+0000","last_clean":"2023-10-06T05:07:01.749439+0000","last_became_active":"2023-10-06T05:06:56.708144+0000","last_became_peered":"2023-10-06T05:06:56.708144+0000","last_unstale":"2023-10-06T05:07:01.749439+0000","last_undegraded":"2023-10-06T05:07:01.749439+0000","last_fullsized":"2023-10-06T05:07:01.749439+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:17:02.349663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750522+0000","last_change":"2023-10-06T05:06:57.620557+0000","last_active":"2023-10-06T05:07:01.750522+0000","last_peered":"2023-10-06T05:07:01.750522+0000","last_clean":"2023-10-06T05:07:01.750522+0000","last_became_active":"2023-10-06T05:06:57.620286+0000","last_became_peered":"2023-10-06T05:06:57.620286+0000","last_unstale":"2023-10-06T05:07:01.750522+0000","last_undegraded":"2023-10-06T05:07:01.750522+0000","last_fullsized":"2023-10-06T05:07:01.750522+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T16:14:57.399087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.765266+0000","last_change":"2023-10-06T05:07:01.762346+0000","last_active":"2023-10-06T05:07:01.765266+0000","last_peered":"2023-10-06T05:07:01.765266+0000","last_clean":"2023-10-06T05:07:01.765266+0000","last_became_active":"2023-10-06T05:07:01.761861+0000","last_became_peered":"2023-10-06T05:07:01.761861+0000","last_unstale":"2023-10-06T05:07:01.765266+0000","last_undegraded":"2023-10-06T05:07:01.765266+0000","last_fullsized":"2023-10-06T05:07:01.765266+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:41:09.047631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6,5],"acting":[0,1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.748453+0000","last_change":"2023-10-06T05:06:52.627232+0000","last_active":"2023-10-06T05:07:01.748453+0000","last_peered":"2023-10-06T05:07:01.748453+0000","last_clean":"2023-10-06T05:07:01.748453+0000","last_became_active":"2023-10-06T05:06:52.626936+0000","last_became_peered":"2023-10-06T05:06:52.626936+0000","last_unstale":"2023-10-06T05:07:01.748453+0000","last_undegraded":"2023-10-06T05:07:01.748453+0000","last_fullsized":"2023-10-06T05:07:01.748453+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:20:13.925170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.744621+0000","last_change":"2023-10-06T05:06:56.712027+0000","last_active":"2023-10-06T05:07:01.744621+0000","last_peered":"2023-10-06T05:07:01.744621+0000","last_clean":"2023-10-06T05:07:01.744621+0000","last_became_active":"2023-10-06T05:06:56.711779+0000","last_became_peered":"2023-10-06T05:06:56.711779+0000","last_unstale":"2023-10-06T05:07:01.744621+0000","last_undegraded":"2023-10-06T05:07:01.744621+0000","last_fullsized":"2023-10-06T05:07:01.744621+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:49:27.022333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.747306+0000","last_change":"2023-10-06T05:06:57.613847+0000","last_active":"2023-10-06T05:07:01.747306+0000","last_peered":"2023-10-06T05:07:01.747306+0000","last_clean":"2023-10-06T05:07:01.747306+0000","last_became_active":"2023-10-06T05:06:57.613560+0000","last_became_peered":"2023-10-06T05:06:57.613560+0000","last_unstale":"2023-10-06T05:07:01.747306+0000","last_undegraded":"2023-10-06T05:07:01.747306+0000","last_fullsized":"2023-10-06T05:07:01.747306+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:50:36.787873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.766219+0000","last_change":"2023-10-06T05:07:01.764522+0000","last_active":"2023-10-06T05:07:01.766219+0000","last_peered":"2023-10-06T05:07:01.766219+0000","last_clean":"2023-10-06T05:07:01.766219+0000","last_became_active":"2023-10-06T05:07:01.764199+0000","last_became_peered":"2023-10-06T05:07:01.764199+0000","last_unstale":"2023-10-06T05:07:01.766219+0000","last_undegraded":"2023-10-06T05:07:01.766219+0000","last_fullsized":"2023-10-06T05:07:01.766219+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:13:19.947679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,6],"acting":[0,1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.744573+0000","last_change":"2023-10-06T05:06:52.587795+0000","last_active":"2023-10-06T05:07:01.744573+0000","last_peered":"2023-10-06T05:07:01.744573+0000","last_clean":"2023-10-06T05:07:01.744573+0000","last_became_active":"2023-10-06T05:06:52.587407+0000","last_became_peered":"2023-10-06T05:06:52.587407+0000","last_unstale":"2023-10-06T05:07:01.744573+0000","last_undegraded":"2023-10-06T05:07:01.744573+0000","last_fullsized":"2023-10-06T05:07:01.744573+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:36:31.526532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.761857+0000","last_change":"2023-10-06T05:06:56.721976+0000","last_active":"2023-10-06T05:07:01.761857+0000","last_peered":"2023-10-06T05:07:01.761857+0000","last_clean":"2023-10-06T05:07:01.761857+0000","last_became_active":"2023-10-06T05:06:56.721679+0000","last_became_peered":"2023-10-06T05:06:56.721679+0000","last_unstale":"2023-10-06T05:07:01.761857+0000","last_undegraded":"2023-10-06T05:07:01.761857+0000","last_fullsized":"2023-10-06T05:07:01.761857+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:38:50.877673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.748383+0000","last_change":"2023-10-06T05:06:57.612621+0000","last_active":"2023-10-06T05:07:01.748383+0000","last_peered":"2023-10-06T05:07:01.748383+0000","last_clean":"2023-10-06T05:07:01.748383+0000","last_became_active":"2023-10-06T05:06:57.612305+0000","last_became_peered":"2023-10-06T05:06:57.612305+0000","last_unstale":"2023-10-06T05:07:01.748383+0000","last_undegraded":"2023-10-06T05:07:01.748383+0000","last_fullsized":"2023-10-06T05:07:01.748383+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:35:18.828024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.768737+0000","last_change":"2023-10-06T05:07:01.764514+0000","last_active":"2023-10-06T05:07:01.768737+0000","last_peered":"2023-10-06T05:07:01.768737+0000","last_clean":"2023-10-06T05:07:01.768737+0000","last_became_active":"2023-10-06T05:07:01.764176+0000","last_became_peered":"2023-10-06T05:07:01.764176+0000","last_unstale":"2023-10-06T05:07:01.768737+0000","last_undegraded":"2023-10-06T05:07:01.768737+0000","last_fullsized":"2023-10-06T05:07:01.768737+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:20:08.719247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1,7],"acting":[2,5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.746279+0000","last_change":"2023-10-06T05:06:52.675238+0000","last_active":"2023-10-06T05:07:01.746279+0000","last_peered":"2023-10-06T05:07:01.746279+0000","last_clean":"2023-10-06T05:07:01.746279+0000","last_became_active":"2023-10-06T05:06:52.674952+0000","last_became_peered":"2023-10-06T05:06:52.674952+0000","last_unstale":"2023-10-06T05:07:01.746279+0000","last_undegraded":"2023-10-06T05:07:01.746279+0000","last_fullsized":"2023-10-06T05:07:01.746279+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:27:51.141184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.746657+0000","last_change":"2023-10-06T05:06:56.711280+0000","last_active":"2023-10-06T05:07:01.746657+0000","last_peered":"2023-10-06T05:07:01.746657+0000","last_clean":"2023-10-06T05:07:01.746657+0000","last_became_active":"2023-10-06T05:06:56.711033+0000","last_became_peered":"2023-10-06T05:06:56.711033+0000","last_unstale":"2023-10-06T05:07:01.746657+0000","last_undegraded":"2023-10-06T05:07:01.746657+0000","last_fullsized":"2023-10-06T05:07:01.746657+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:38:12.815936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750448+0000","last_change":"2023-10-06T05:06:57.614460+0000","last_active":"2023-10-06T05:07:01.750448+0000","last_peered":"2023-10-06T05:07:01.750448+0000","last_clean":"2023-10-06T05:07:01.750448+0000","last_became_active":"2023-10-06T05:06:57.614245+0000","last_became_peered":"2023-10-06T05:06:57.614245+0000","last_unstale":"2023-10-06T05:07:01.750448+0000","last_undegraded":"2023-10-06T05:07:01.750448+0000","last_fullsized":"2023-10-06T05:07:01.750448+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:51:31.873476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.775792+0000","last_change":"2023-10-06T05:07:01.766672+0000","last_active":"2023-10-06T05:07:01.775792+0000","last_peered":"2023-10-06T05:07:01.775792+0000","last_clean":"2023-10-06T05:07:01.775792+0000","last_became_active":"2023-10-06T05:07:01.766118+0000","last_became_peered":"2023-10-06T05:07:01.766118+0000","last_unstale":"2023-10-06T05:07:01.775792+0000","last_undegraded":"2023-10-06T05:07:01.775792+0000","last_fullsized":"2023-10-06T05:07:01.775792+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:55:02.566598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5,3],"acting":[7,2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.745050+0000","last_change":"2023-10-06T05:06:52.587778+0000","last_active":"2023-10-06T05:07:01.745050+0000","last_peered":"2023-10-06T05:07:01.745050+0000","last_clean":"2023-10-06T05:07:01.745050+0000","last_became_active":"2023-10-06T05:06:52.587388+0000","last_became_peered":"2023-10-06T05:06:52.587388+0000","last_unstale":"2023-10-06T05:07:01.745050+0000","last_undegraded":"2023-10-06T05:07:01.745050+0000","last_fullsized":"2023-10-06T05:07:01.745050+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:45:52.502298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":169,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.748339+0000","last_change":"2023-10-06T05:06:46.307169+0000","last_active":"2023-10-06T05:07:01.748339+0000","last_peered":"2023-10-06T05:07:01.748339+0000","last_clean":"2023-10-06T05:07:01.748339+0000","last_became_active":"2023-10-06T05:06:46.306107+0000","last_became_peered":"2023-10-06T05:06:46.306107+0000","last_unstale":"2023-10-06T05:07:01.748339+0000","last_undegraded":"2023-10-06T05:07:01.748339+0000","last_fullsized":"2023-10-06T05:07:01.748339+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:45.018006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:45.018006+0000","last_clean_scrub_stamp":"2023-10-06T05:06:45.018006+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:38:17.064131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750303+0000","last_change":"2023-10-06T05:06:56.706870+0000","last_active":"2023-10-06T05:07:01.750303+0000","last_peered":"2023-10-06T05:07:01.750303+0000","last_clean":"2023-10-06T05:07:01.750303+0000","last_became_active":"2023-10-06T05:06:56.706636+0000","last_became_peered":"2023-10-06T05:06:56.706636+0000","last_unstale":"2023-10-06T05:07:01.750303+0000","last_undegraded":"2023-10-06T05:07:01.750303+0000","last_fullsized":"2023-10-06T05:07:01.750303+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:00:29.033929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.753036+0000","last_change":"2023-10-06T05:06:57.604630+0000","last_active":"2023-10-06T05:07:01.753036+0000","last_peered":"2023-10-06T05:07:01.753036+0000","last_clean":"2023-10-06T05:07:01.753036+0000","last_became_active":"2023-10-06T05:06:57.604341+0000","last_became_peered":"2023-10-06T05:06:57.604341+0000","last_unstale":"2023-10-06T05:07:01.753036+0000","last_undegraded":"2023-10-06T05:07:01.753036+0000","last_fullsized":"2023-10-06T05:07:01.753036+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:57:12.615864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.767715+0000","last_change":"2023-10-06T05:07:01.763448+0000","last_active":"2023-10-06T05:07:01.767715+0000","last_peered":"2023-10-06T05:07:01.767715+0000","last_clean":"2023-10-06T05:07:01.767715+0000","last_became_active":"2023-10-06T05:07:01.762832+0000","last_became_peered":"2023-10-06T05:07:01.762832+0000","last_unstale":"2023-10-06T05:07:01.767715+0000","last_undegraded":"2023-10-06T05:07:01.767715+0000","last_fullsized":"2023-10-06T05:07:01.767715+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:49:53.682865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0,4],"acting":[5,1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.748590+0000","last_change":"2023-10-06T05:06:52.588475+0000","last_active":"2023-10-06T05:07:01.748590+0000","last_peered":"2023-10-06T05:07:01.748590+0000","last_clean":"2023-10-06T05:07:01.748590+0000","last_became_active":"2023-10-06T05:06:52.588134+0000","last_became_peered":"2023-10-06T05:06:52.588134+0000","last_unstale":"2023-10-06T05:07:01.748590+0000","last_undegraded":"2023-10-06T05:07:01.748590+0000","last_fullsized":"2023-10-06T05:07:01.748590+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:35:05.269778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.749699+0000","last_change":"2023-10-06T05:06:56.723240+0000","last_active":"2023-10-06T05:07:01.749699+0000","last_peered":"2023-10-06T05:07:01.749699+0000","last_clean":"2023-10-06T05:07:01.749699+0000","last_became_active":"2023-10-06T05:06:56.722965+0000","last_became_peered":"2023-10-06T05:06:56.722965+0000","last_unstale":"2023-10-06T05:07:01.749699+0000","last_undegraded":"2023-10-06T05:07:01.749699+0000","last_fullsized":"2023-10-06T05:07:01.749699+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:33:23.298771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.749910+0000","last_change":"2023-10-06T05:06:57.616494+0000","last_active":"2023-10-06T05:07:01.749910+0000","last_peered":"2023-10-06T05:07:01.749910+0000","last_clean":"2023-10-06T05:07:01.749910+0000","last_became_active":"2023-10-06T05:06:57.616234+0000","last_became_peered":"2023-10-06T05:06:57.616234+0000","last_unstale":"2023-10-06T05:07:01.749910+0000","last_undegraded":"2023-10-06T05:07:01.749910+0000","last_fullsized":"2023-10-06T05:07:01.749910+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:34:55.987076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.773039+0000","last_change":"2023-10-06T05:07:01.768822+0000","last_active":"2023-10-06T05:07:01.773039+0000","last_peered":"2023-10-06T05:07:01.773039+0000","last_clean":"2023-10-06T05:07:01.773039+0000","last_became_active":"2023-10-06T05:07:01.768425+0000","last_became_peered":"2023-10-06T05:07:01.768425+0000","last_unstale":"2023-10-06T05:07:01.773039+0000","last_undegraded":"2023-10-06T05:07:01.773039+0000","last_fullsized":"2023-10-06T05:07:01.773039+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:52:18.818435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5,2],"acting":[6,0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.754250+0000","last_change":"2023-10-06T05:06:52.627289+0000","last_active":"2023-10-06T05:07:01.754250+0000","last_peered":"2023-10-06T05:07:01.754250+0000","last_clean":"2023-10-06T05:07:01.754250+0000","last_became_active":"2023-10-06T05:06:52.626937+0000","last_became_peered":"2023-10-06T05:06:52.626937+0000","last_unstale":"2023-10-06T05:07:01.754250+0000","last_undegraded":"2023-10-06T05:07:01.754250+0000","last_fullsized":"2023-10-06T05:07:01.754250+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:46:59.070674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.748611+0000","last_change":"2023-10-06T05:06:56.720424+0000","last_active":"2023-10-06T05:07:01.748611+0000","last_peered":"2023-10-06T05:07:01.748611+0000","last_clean":"2023-10-06T05:07:01.748611+0000","last_became_active":"2023-10-06T05:06:56.720133+0000","last_became_peered":"2023-10-06T05:06:56.720133+0000","last_unstale":"2023-10-06T05:07:01.748611+0000","last_undegraded":"2023-10-06T05:07:01.748611+0000","last_fullsized":"2023-10-06T05:07:01.748611+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T16:11:31.404755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750404+0000","last_change":"2023-10-06T05:06:57.619048+0000","last_active":"2023-10-06T05:07:01.750404+0000","last_peered":"2023-10-06T05:07:01.750404+0000","last_clean":"2023-10-06T05:07:01.750404+0000","last_became_active":"2023-10-06T05:06:57.618823+0000","last_became_peered":"2023-10-06T05:06:57.618823+0000","last_unstale":"2023-10-06T05:07:01.750404+0000","last_undegraded":"2023-10-06T05:07:01.750404+0000","last_fullsized":"2023-10-06T05:07:01.750404+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:25:56.905027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.768167+0000","last_change":"2023-10-06T05:07:01.765125+0000","last_active":"2023-10-06T05:07:01.768167+0000","last_peered":"2023-10-06T05:07:01.768167+0000","last_clean":"2023-10-06T05:07:01.768167+0000","last_became_active":"2023-10-06T05:07:01.764733+0000","last_became_peered":"2023-10-06T05:07:01.764733+0000","last_unstale":"2023-10-06T05:07:01.768167+0000","last_undegraded":"2023-10-06T05:07:01.768167+0000","last_fullsized":"2023-10-06T05:07:01.768167+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T16:29:47.226299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7,0],"acting":[4,3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"24'2","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750393+0000","last_change":"2023-10-06T05:06:56.710924+0000","last_active":"2023-10-06T05:07:01.750393+0000","last_peered":"2023-10-06T05:07:01.750393+0000","last_clean":"2023-10-06T05:07:01.750393+0000","last_became_active":"2023-10-06T05:06:56.710518+0000","last_became_peered":"2023-10-06T05:06:56.710518+0000","last_unstale":"2023-10-06T05:07:01.750393+0000","last_undegraded":"2023-10-06T05:07:01.750393+0000","last_fullsized":"2023-10-06T05:07:01.750393+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:49:27.022333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.748508+0000","last_change":"2023-10-06T05:06:52.588853+0000","last_active":"2023-10-06T05:07:01.748508+0000","last_peered":"2023-10-06T05:07:01.748508+0000","last_clean":"2023-10-06T05:07:01.748508+0000","last_became_active":"2023-10-06T05:06:52.588561+0000","last_became_peered":"2023-10-06T05:06:52.588561+0000","last_unstale":"2023-10-06T05:07:01.748508+0000","last_undegraded":"2023-10-06T05:07:01.748508+0000","last_fullsized":"2023-10-06T05:07:01.748508+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:06:28.729493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.749357+0000","last_change":"2023-10-06T05:06:57.619145+0000","last_active":"2023-10-06T05:07:01.749357+0000","last_peered":"2023-10-06T05:07:01.749357+0000","last_clean":"2023-10-06T05:07:01.749357+0000","last_became_active":"2023-10-06T05:06:57.618666+0000","last_became_peered":"2023-10-06T05:06:57.618666+0000","last_unstale":"2023-10-06T05:07:01.749357+0000","last_undegraded":"2023-10-06T05:07:01.749357+0000","last_fullsized":"2023-10-06T05:07:01.749357+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:29:19.951056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.765168+0000","last_change":"2023-10-06T05:07:01.762410+0000","last_active":"2023-10-06T05:07:01.765168+0000","last_peered":"2023-10-06T05:07:01.765168+0000","last_clean":"2023-10-06T05:07:01.765168+0000","last_became_active":"2023-10-06T05:07:01.762060+0000","last_became_peered":"2023-10-06T05:07:01.762060+0000","last_unstale":"2023-10-06T05:07:01.765168+0000","last_undegraded":"2023-10-06T05:07:01.765168+0000","last_fullsized":"2023-10-06T05:07:01.765168+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:49:53.682865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,0],"acting":[3,1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.751181+0000","last_change":"2023-10-06T05:06:52.589519+0000","last_active":"2023-10-06T05:07:01.751181+0000","last_peered":"2023-10-06T05:07:01.751181+0000","last_clean":"2023-10-06T05:07:01.751181+0000","last_became_active":"2023-10-06T05:06:52.589246+0000","last_became_peered":"2023-10-06T05:06:52.589246+0000","last_unstale":"2023-10-06T05:07:01.751181+0000","last_undegraded":"2023-10-06T05:07:01.751181+0000","last_fullsized":"2023-10-06T05:07:01.751181+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:54:23.352764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"28'1","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:02.685849+0000","last_change":"2023-10-06T05:06:56.714058+0000","last_active":"2023-10-06T05:07:02.685849+0000","last_peered":"2023-10-06T05:07:02.685849+0000","last_clean":"2023-10-06T05:07:02.685849+0000","last_became_active":"2023-10-06T05:06:56.713772+0000","last_became_peered":"2023-10-06T05:06:56.713772+0000","last_unstale":"2023-10-06T05:07:02.685849+0000","last_undegraded":"2023-10-06T05:07:02.685849+0000","last_fullsized":"2023-10-06T05:07:02.685849+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:53:47.748350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.761483+0000","last_change":"2023-10-06T05:06:57.608832+0000","last_active":"2023-10-06T05:07:01.761483+0000","last_peered":"2023-10-06T05:07:01.761483+0000","last_clean":"2023-10-06T05:07:01.761483+0000","last_became_active":"2023-10-06T05:06:57.608499+0000","last_became_peered":"2023-10-06T05:06:57.608499+0000","last_unstale":"2023-10-06T05:07:01.761483+0000","last_undegraded":"2023-10-06T05:07:01.761483+0000","last_fullsized":"2023-10-06T05:07:01.761483+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:29:19.951056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.772123+0000","last_change":"2023-10-06T05:07:01.765615+0000","last_active":"2023-10-06T05:07:01.772123+0000","last_peered":"2023-10-06T05:07:01.772123+0000","last_clean":"2023-10-06T05:07:01.772123+0000","last_became_active":"2023-10-06T05:07:01.762552+0000","last_became_peered":"2023-10-06T05:07:01.762552+0000","last_unstale":"2023-10-06T05:07:01.772123+0000","last_undegraded":"2023-10-06T05:07:01.772123+0000","last_fullsized":"2023-10-06T05:07:01.772123+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:30:14.995001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,0],"acting":[1,3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"24'1","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750483+0000","last_change":"2023-10-06T05:06:56.723224+0000","last_active":"2023-10-06T05:07:01.750483+0000","last_peered":"2023-10-06T05:07:01.750483+0000","last_clean":"2023-10-06T05:07:01.750483+0000","last_became_active":"2023-10-06T05:06:56.722971+0000","last_became_peered":"2023-10-06T05:06:56.722971+0000","last_unstale":"2023-10-06T05:07:01.750483+0000","last_undegraded":"2023-10-06T05:07:01.750483+0000","last_fullsized":"2023-10-06T05:07:01.750483+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:10:32.670079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.746067+0000","last_change":"2023-10-06T05:06:57.610613+0000","last_active":"2023-10-06T05:07:01.746067+0000","last_peered":"2023-10-06T05:07:01.746067+0000","last_clean":"2023-10-06T05:07:01.746067+0000","last_became_active":"2023-10-06T05:06:57.610363+0000","last_became_peered":"2023-10-06T05:06:57.610363+0000","last_unstale":"2023-10-06T05:07:01.746067+0000","last_undegraded":"2023-10-06T05:07:01.746067+0000","last_fullsized":"2023-10-06T05:07:01.746067+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:55:09.125970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.775891+0000","last_change":"2023-10-06T05:07:01.771319+0000","last_active":"2023-10-06T05:07:01.775891+0000","last_peered":"2023-10-06T05:07:01.775891+0000","last_clean":"2023-10-06T05:07:01.775891+0000","last_became_active":"2023-10-06T05:07:01.770953+0000","last_became_peered":"2023-10-06T05:07:01.770953+0000","last_unstale":"2023-10-06T05:07:01.775891+0000","last_undegraded":"2023-10-06T05:07:01.775891+0000","last_fullsized":"2023-10-06T05:07:01.775891+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:13:19.947679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1,6],"acting":[7,4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"24'2","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.745566+0000","last_change":"2023-10-06T05:06:56.723033+0000","last_active":"2023-10-06T05:07:01.745566+0000","last_peered":"2023-10-06T05:07:01.745566+0000","last_clean":"2023-10-06T05:07:01.745566+0000","last_became_active":"2023-10-06T05:06:56.722747+0000","last_became_peered":"2023-10-06T05:06:56.722747+0000","last_unstale":"2023-10-06T05:07:01.745566+0000","last_undegraded":"2023-10-06T05:07:01.745566+0000","last_fullsized":"2023-10-06T05:07:01.745566+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:17:02.349663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.761541+0000","last_change":"2023-10-06T05:06:57.621769+0000","last_active":"2023-10-06T05:07:01.761541+0000","last_peered":"2023-10-06T05:07:01.761541+0000","last_clean":"2023-10-06T05:07:01.761541+0000","last_became_active":"2023-10-06T05:06:57.621489+0000","last_became_peered":"2023-10-06T05:06:57.621489+0000","last_unstale":"2023-10-06T05:07:01.761541+0000","last_undegraded":"2023-10-06T05:07:01.761541+0000","last_fullsized":"2023-10-06T05:07:01.761541+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:59:39.994965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.773062+0000","last_change":"2023-10-06T05:07:01.769584+0000","last_active":"2023-10-06T05:07:01.773062+0000","last_peered":"2023-10-06T05:07:01.773062+0000","last_clean":"2023-10-06T05:07:01.773062+0000","last_became_active":"2023-10-06T05:07:01.769168+0000","last_became_peered":"2023-10-06T05:07:01.769168+0000","last_unstale":"2023-10-06T05:07:01.773062+0000","last_undegraded":"2023-10-06T05:07:01.773062+0000","last_fullsized":"2023-10-06T05:07:01.773062+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:25:41.281713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7,4],"acting":[5,0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"24'2","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.744323+0000","last_change":"2023-10-06T05:06:56.721940+0000","last_active":"2023-10-06T05:07:01.744323+0000","last_peered":"2023-10-06T05:07:01.744323+0000","last_clean":"2023-10-06T05:07:01.744323+0000","last_became_active":"2023-10-06T05:06:56.721668+0000","last_became_peered":"2023-10-06T05:06:56.721668+0000","last_unstale":"2023-10-06T05:07:01.744323+0000","last_undegraded":"2023-10-06T05:07:01.744323+0000","last_fullsized":"2023-10-06T05:07:01.744323+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:55:08.124129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.760815+0000","last_change":"2023-10-06T05:06:57.614557+0000","last_active":"2023-10-06T05:07:01.760815+0000","last_peered":"2023-10-06T05:07:01.760815+0000","last_clean":"2023-10-06T05:07:01.760815+0000","last_became_active":"2023-10-06T05:06:57.614113+0000","last_became_peered":"2023-10-06T05:06:57.614113+0000","last_unstale":"2023-10-06T05:07:01.760815+0000","last_undegraded":"2023-10-06T05:07:01.760815+0000","last_fullsized":"2023-10-06T05:07:01.760815+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:57:12.615864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.772225+0000","last_change":"2023-10-06T05:07:01.766934+0000","last_active":"2023-10-06T05:07:01.772225+0000","last_peered":"2023-10-06T05:07:01.772225+0000","last_clean":"2023-10-06T05:07:01.772225+0000","last_became_active":"2023-10-06T05:07:01.766522+0000","last_became_peered":"2023-10-06T05:07:01.766522+0000","last_unstale":"2023-10-06T05:07:01.772225+0000","last_undegraded":"2023-10-06T05:07:01.772225+0000","last_fullsized":"2023-10-06T05:07:01.772225+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:15:30.724218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,2],"acting":[4,6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"24'1","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.753608+0000","last_change":"2023-10-06T05:06:56.717832+0000","last_active":"2023-10-06T05:07:01.753608+0000","last_peered":"2023-10-06T05:07:01.753608+0000","last_clean":"2023-10-06T05:07:01.753608+0000","last_became_active":"2023-10-06T05:06:56.717568+0000","last_became_peered":"2023-10-06T05:06:56.717568+0000","last_unstale":"2023-10-06T05:07:01.753608+0000","last_undegraded":"2023-10-06T05:07:01.753608+0000","last_fullsized":"2023-10-06T05:07:01.753608+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:30:18.507322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750496+0000","last_change":"2023-10-06T05:06:57.620391+0000","last_active":"2023-10-06T05:07:01.750496+0000","last_peered":"2023-10-06T05:07:01.750496+0000","last_clean":"2023-10-06T05:07:01.750496+0000","last_became_active":"2023-10-06T05:06:57.619822+0000","last_became_peered":"2023-10-06T05:06:57.619822+0000","last_unstale":"2023-10-06T05:07:01.750496+0000","last_undegraded":"2023-10-06T05:07:01.750496+0000","last_fullsized":"2023-10-06T05:07:01.750496+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:59:39.994965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.772089+0000","last_change":"2023-10-06T05:07:01.762139+0000","last_active":"2023-10-06T05:07:01.772089+0000","last_peered":"2023-10-06T05:07:01.772089+0000","last_clean":"2023-10-06T05:07:01.772089+0000","last_became_active":"2023-10-06T05:07:01.761672+0000","last_became_peered":"2023-10-06T05:07:01.761672+0000","last_unstale":"2023-10-06T05:07:01.772089+0000","last_undegraded":"2023-10-06T05:07:01.772089+0000","last_fullsized":"2023-10-06T05:07:01.772089+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:07:51.743165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7,2],"acting":[1,5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"24'4","reported_seq":18,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.751360+0000","last_change":"2023-10-06T05:06:56.714871+0000","last_active":"2023-10-06T05:07:01.751360+0000","last_peered":"2023-10-06T05:07:01.751360+0000","last_clean":"2023-10-06T05:07:01.751360+0000","last_became_active":"2023-10-06T05:06:56.714616+0000","last_became_peered":"2023-10-06T05:06:56.714616+0000","last_unstale":"2023-10-06T05:07:01.751360+0000","last_undegraded":"2023-10-06T05:07:01.751360+0000","last_fullsized":"2023-10-06T05:07:01.751360+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:35:42.402195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.758731+0000","last_change":"2023-10-06T05:06:57.605198+0000","last_active":"2023-10-06T05:07:01.758731+0000","last_peered":"2023-10-06T05:07:01.758731+0000","last_clean":"2023-10-06T05:07:01.758731+0000","last_became_active":"2023-10-06T05:06:57.604924+0000","last_became_peered":"2023-10-06T05:06:57.604924+0000","last_unstale":"2023-10-06T05:07:01.758731+0000","last_undegraded":"2023-10-06T05:07:01.758731+0000","last_fullsized":"2023-10-06T05:07:01.758731+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:01:11.983432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750721+0000","last_change":"2023-10-06T05:06:57.608286+0000","last_active":"2023-10-06T05:07:01.750721+0000","last_peered":"2023-10-06T05:07:01.750721+0000","last_clean":"2023-10-06T05:07:01.750721+0000","last_became_active":"2023-10-06T05:06:57.607986+0000","last_became_peered":"2023-10-06T05:06:57.607986+0000","last_unstale":"2023-10-06T05:07:01.750721+0000","last_undegraded":"2023-10-06T05:07:01.750721+0000","last_fullsized":"2023-10-06T05:07:01.750721+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:51:31.370159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750626+0000","last_change":"2023-10-06T05:06:57.623937+0000","last_active":"2023-10-06T05:07:01.750626+0000","last_peered":"2023-10-06T05:07:01.750626+0000","last_clean":"2023-10-06T05:07:01.750626+0000","last_became_active":"2023-10-06T05:06:57.623705+0000","last_became_peered":"2023-10-06T05:06:57.623705+0000","last_unstale":"2023-10-06T05:07:01.750626+0000","last_undegraded":"2023-10-06T05:07:01.750626+0000","last_fullsized":"2023-10-06T05:07:01.750626+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:20:04.673613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.761577+0000","last_change":"2023-10-06T05:06:57.611066+0000","last_active":"2023-10-06T05:07:01.761577+0000","last_peered":"2023-10-06T05:07:01.761577+0000","last_clean":"2023-10-06T05:07:01.761577+0000","last_became_active":"2023-10-06T05:06:57.610827+0000","last_became_peered":"2023-10-06T05:06:57.610827+0000","last_unstale":"2023-10-06T05:07:01.761577+0000","last_undegraded":"2023-10-06T05:07:01.761577+0000","last_fullsized":"2023-10-06T05:07:01.761577+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:43:18.121629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750840+0000","last_change":"2023-10-06T05:06:57.615191+0000","last_active":"2023-10-06T05:07:01.750840+0000","last_peered":"2023-10-06T05:07:01.750840+0000","last_clean":"2023-10-06T05:07:01.750840+0000","last_became_active":"2023-10-06T05:06:57.614632+0000","last_became_peered":"2023-10-06T05:06:57.614632+0000","last_unstale":"2023-10-06T05:07:01.750840+0000","last_undegraded":"2023-10-06T05:07:01.750840+0000","last_fullsized":"2023-10-06T05:07:01.750840+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:32:33.731449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.748531+0000","last_change":"2023-10-06T05:06:57.616743+0000","last_active":"2023-10-06T05:07:01.748531+0000","last_peered":"2023-10-06T05:07:01.748531+0000","last_clean":"2023-10-06T05:07:01.748531+0000","last_became_active":"2023-10-06T05:06:57.616487+0000","last_became_peered":"2023-10-06T05:06:57.616487+0000","last_unstale":"2023-10-06T05:07:01.748531+0000","last_undegraded":"2023-10-06T05:07:01.748531+0000","last_fullsized":"2023-10-06T05:07:01.748531+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:15:55.618648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":4484,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":8968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509448,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297588,"kb_used_data":300,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074252,"statfs":{"total":96636764160,"available":96332034048,"internally_reserved":0,"allocated":307200,"data_stored":126099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.376}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.498}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42799999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64500000000000002}]}]},{"osd":5,"up_from":15,"seq":64424509448,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297596,"kb_used_data":308,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074244,"statfs":{"total":96636764160,"available":96332025856,"internally_reserved":0,"allocated":315392,"data_stored":126723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56799999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48799999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55500000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66800000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63200000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542152,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298172,"kb_used_data":884,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073668,"statfs":{"total":96636764160,"available":96331436032,"internally_reserved":0,"allocated":905216,"data_stored":717021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33700000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]}]},{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298196,"kb_used_data":908,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073644,"statfs":{"total":96636764160,"available":96331411456,"internally_reserved":0,"allocated":929792,"data_stored":718672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56599999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57599999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53300000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64300000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69599999999999995}]}]},{"osd":1,"up_from":14,"seq":60129542151,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297588,"kb_used_data":300,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074252,"statfs":{"total":96636764160,"available":96332034048,"internally_reserved":0,"allocated":307200,"data_stored":126099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29999999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33200000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54900000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72399999999999998}]}]},{"osd":2,"up_from":15,"seq":64424509447,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297588,"kb_used_data":300,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074252,"statfs":{"total":96636764160,"available":96332034048,"internally_reserved":0,"allocated":307200,"data_stored":126099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26400000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30599999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28000000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39400000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42299999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59799999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.504}]}]},{"osd":3,"up_from":15,"seq":64424509447,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297592,"kb_used_data":304,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074248,"statfs":{"total":96636764160,"available":96332029952,"internally_reserved":0,"allocated":311296,"data_stored":126189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26700000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28599999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32700000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66900000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68000000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.625}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56299999999999994}]}]},{"osd":4,"up_from":15,"seq":64424509448,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297608,"kb_used_data":320,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074232,"statfs":{"total":96636764160,"available":96332013568,"internally_reserved":0,"allocated":327680,"data_stored":129869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40100000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.379}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":3770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":3930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-06T05:07:11.201 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-06T05:07:11.201 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-10-06T05:07:11.202 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-10-06T05:07:11.202 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-10-06T05:07:11.202 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-10-06T05:07:11.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.346+0000 7f194945f700 1 Processor -- start 2023-10-06T05:07:11.349 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.346+0000 7f194945f700 1 -- start start 2023-10-06T05:07:11.351 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.347+0000 7f194945f700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f1944159560 0x7f19440aa530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:11.351 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.347+0000 7f194945f700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f19440b49a0 0x7f19440aaa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:11.351 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.347+0000 7f194945f700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f19440b0760 0x7f19440b3890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:11.351 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.347+0000 7f194945f700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19440b0c60 con 0x7f19440b49a0 2023-10-06T05:07:11.351 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.347+0000 7f194945f700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19440b0f40 con 0x7f1944159560 2023-10-06T05:07:11.351 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.347+0000 7f194945f700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19440b1220 con 0x7f19440b0760 2023-10-06T05:07:11.351 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.347+0000 7f193a7fc700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f19440b49a0 0x7f19440aaa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:11.352 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.347+0000 7f193a7fc700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f19440b49a0 0x7f19440aaa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45656/0 (socket says 172.21.15.98:45656) 2023-10-06T05:07:11.352 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.347+0000 7f193a7fc700 1 -- 172.21.15.98:0/1643282471 learned_addr learned my addr 172.21.15.98:0/1643282471 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:11.352 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.347+0000 7f19437fe700 1 --2- 172.21.15.98:0/1643282471 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f19440b0760 0x7f19440b3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:11.352 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.348+0000 7f193a7fc700 1 -- 172.21.15.98:0/1643282471 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f19440b0760 msgr2=0x7f19440b3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:11.352 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.348+0000 7f1942ffd700 1 --2- 172.21.15.98:0/1643282471 >> v2:172.21.15.174:3300/0 conn(0x7f1944159560 0x7f19440aa530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:11.352 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.348+0000 7f193a7fc700 1 --2- 172.21.15.98:0/1643282471 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f19440b0760 0x7f19440b3890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:11.352 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.348+0000 7f193a7fc700 1 -- 172.21.15.98:0/1643282471 >> v2:172.21.15.174:3300/0 conn(0x7f1944159560 msgr2=0x7f19440aa530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:11.352 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.348+0000 7f193a7fc700 1 --2- 172.21.15.98:0/1643282471 >> v2:172.21.15.174:3300/0 conn(0x7f1944159560 0x7f19440aa530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:11.352 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.348+0000 7f193a7fc700 1 -- 172.21.15.98:0/1643282471 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19440b1500 con 0x7f19440b49a0 2023-10-06T05:07:11.352 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.348+0000 7f193a7fc700 1 --2- 172.21.15.98:0/1643282471 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19440b49a0 0x7f19440aaa70 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f192c0099b0 tx=0x7f192c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5382ebbf227c3e78 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:11.352 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.348+0000 7f19427fc700 1 -- 172.21.15.98:0/1643282471 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f192c00b820 con 0x7f19440b49a0 2023-10-06T05:07:11.352 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.348+0000 7f19427fc700 1 -- 172.21.15.98:0/1643282471 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f192c00b980 con 0x7f19440b49a0 2023-10-06T05:07:11.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.348+0000 7f19427fc700 1 -- 172.21.15.98:0/1643282471 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f192c00bd20 con 0x7f19440b49a0 2023-10-06T05:07:11.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.348+0000 7f194945f700 1 -- 172.21.15.98:0/1643282471 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19440b49a0 msgr2=0x7f19440aaa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:11.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.348+0000 7f194945f700 1 --2- 172.21.15.98:0/1643282471 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19440b49a0 0x7f19440aaa70 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f192c0099b0 tx=0x7f192c009250 comp rx=0 tx=0).stop 2023-10-06T05:07:11.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.348+0000 7f194945f700 1 -- 172.21.15.98:0/1643282471 shutdown_connections 2023-10-06T05:07:11.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.348+0000 7f194945f700 1 --2- 172.21.15.98:0/1643282471 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f19440b0760 0x7f19440b3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:11.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.348+0000 7f194945f700 1 --2- 172.21.15.98:0/1643282471 >> v2:172.21.15.174:3300/0 conn(0x7f1944159560 0x7f19440aa530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:11.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.348+0000 7f194945f700 1 --2- 172.21.15.98:0/1643282471 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19440b49a0 0x7f19440aaa70 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:11.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.348+0000 7f194945f700 1 -- 172.21.15.98:0/1643282471 >> 172.21.15.98:0/1643282471 conn(0x7f19440997f0 msgr2=0x7f194406f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:11.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.348+0000 7f194945f700 1 -- 172.21.15.98:0/1643282471 shutdown_connections 2023-10-06T05:07:11.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.348+0000 7f194945f700 1 -- 172.21.15.98:0/1643282471 wait complete. 2023-10-06T05:07:11.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.350+0000 7f194945f700 1 Processor -- start 2023-10-06T05:07:11.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.350+0000 7f194945f700 1 -- start start 2023-10-06T05:07:11.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.350+0000 7f194945f700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f19440b0760 0x7f194409acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:11.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.350+0000 7f194945f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19440b49a0 0x7f194409bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:11.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.350+0000 7f194945f700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1944159560 0x7f19440aec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:11.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.350+0000 7f194945f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f194416aec0 con 0x7f19440b49a0 2023-10-06T05:07:11.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.350+0000 7f194945f700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f194409a840 con 0x7f19440b0760 2023-10-06T05:07:11.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.350+0000 7f194945f700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f194416abe0 con 0x7f1944159560 2023-10-06T05:07:11.357 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.350+0000 7f193a7fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19440b49a0 0x7f194409bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:11.357 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.351+0000 7f193a7fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19440b49a0 0x7f194409bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45666/0 (socket says 172.21.15.98:45666) 2023-10-06T05:07:11.357 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.351+0000 7f193a7fc700 1 -- 172.21.15.98:0/3009380779 learned_addr learned my addr 172.21.15.98:0/3009380779 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:11.357 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.351+0000 7f1942ffd700 1 --2- 172.21.15.98:0/3009380779 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f19440b0760 0x7f194409acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:11.357 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.351+0000 7f19437fe700 1 --2- 172.21.15.98:0/3009380779 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1944159560 0x7f19440aec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:11.357 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.351+0000 7f193a7fc700 1 -- 172.21.15.98:0/3009380779 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1944159560 msgr2=0x7f19440aec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:11.357 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.351+0000 7f193a7fc700 1 --2- 172.21.15.98:0/3009380779 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1944159560 0x7f19440aec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:11.357 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.351+0000 7f193a7fc700 1 -- 172.21.15.98:0/3009380779 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f19440b0760 msgr2=0x7f194409acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:11.357 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.351+0000 7f193a7fc700 1 --2- 172.21.15.98:0/3009380779 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f19440b0760 0x7f194409acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:11.357 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.351+0000 7f193a7fc700 1 -- 172.21.15.98:0/3009380779 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f192c0094f0 con 0x7f19440b49a0 2023-10-06T05:07:11.357 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.351+0000 7f193a7fc700 1 --2- 172.21.15.98:0/3009380779 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19440b49a0 0x7f194409bdb0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f192c009290 tx=0x7f192c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:11.358 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.351+0000 7f193b7fe700 1 -- 172.21.15.98:0/3009380779 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f192c011dc0 con 0x7f19440b49a0 2023-10-06T05:07:11.358 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.351+0000 7f193b7fe700 1 -- 172.21.15.98:0/3009380779 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f192c01a420 con 0x7f19440b49a0 2023-10-06T05:07:11.358 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.351+0000 7f194945f700 1 -- 172.21.15.98:0/3009380779 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f194409a1f0 con 0x7f19440b49a0 2023-10-06T05:07:11.358 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.351+0000 7f193b7fe700 1 -- 172.21.15.98:0/3009380779 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f192c01a7e0 con 0x7f19440b49a0 2023-10-06T05:07:11.358 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.351+0000 7f194945f700 1 -- 172.21.15.98:0/3009380779 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f194409b210 con 0x7f19440b49a0 2023-10-06T05:07:11.358 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.352+0000 7f194945f700 1 -- 172.21.15.98:0/3009380779 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f194409a050 con 0x7f19440b49a0 2023-10-06T05:07:11.358 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.353+0000 7f193b7fe700 1 -- 172.21.15.98:0/3009380779 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f192c01ac10 con 0x7f19440b49a0 2023-10-06T05:07:11.358 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.353+0000 7f193b7fe700 1 --2- 172.21.15.98:0/3009380779 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f1924066f60 0x7f1924069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:11.358 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.353+0000 7f193b7fe700 1 -- 172.21.15.98:0/3009380779 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f192c086280 con 0x7f19440b49a0 2023-10-06T05:07:11.358 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.353+0000 7f1942ffd700 1 --2- 172.21.15.98:0/3009380779 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f1924066f60 0x7f1924069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:11.358 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.353+0000 7f1942ffd700 1 --2- 172.21.15.98:0/3009380779 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f1924066f60 0x7f1924069410 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f19440b2920 tx=0x7f1934008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:11.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.357+0000 7f193b7fe700 1 -- 172.21.15.98:0/3009380779 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f194409a050 con 0x7f19440b49a0 2023-10-06T05:07:11.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.594+0000 7f194945f700 1 -- 172.21.15.98:0/3009380779 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f194409a050 con 0x7f19440b49a0 2023-10-06T05:07:11.600 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.597+0000 7f193b7fe700 1 -- 172.21.15.98:0/3009380779 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+163698 (secure 0 0 0) 0x7f194409a050 con 0x7f19440b49a0 2023-10-06T05:07:11.602 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:07:11.605 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.600+0000 7f194945f700 1 -- 172.21.15.98:0/3009380779 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f1924066f60 msgr2=0x7f1924069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:11.605 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.600+0000 7f194945f700 1 --2- 172.21.15.98:0/3009380779 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f1924066f60 0x7f1924069410 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f19440b2920 tx=0x7f1934008040 comp rx=0 tx=0).stop 2023-10-06T05:07:11.605 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.600+0000 7f194945f700 1 -- 172.21.15.98:0/3009380779 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19440b49a0 msgr2=0x7f194409bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:11.605 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.600+0000 7f194945f700 1 --2- 172.21.15.98:0/3009380779 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19440b49a0 0x7f194409bdb0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f192c009290 tx=0x7f192c017000 comp rx=0 tx=0).stop 2023-10-06T05:07:11.605 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.600+0000 7f194945f700 1 -- 172.21.15.98:0/3009380779 shutdown_connections 2023-10-06T05:07:11.605 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.600+0000 7f194945f700 1 --2- 172.21.15.98:0/3009380779 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1944159560 0x7f19440aec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:11.605 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.600+0000 7f194945f700 1 --2- 172.21.15.98:0/3009380779 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f19440b0760 0x7f194409acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:11.606 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.600+0000 7f194945f700 1 --2- 172.21.15.98:0/3009380779 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f1924066f60 0x7f1924069410 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:11.606 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.600+0000 7f194945f700 1 --2- 172.21.15.98:0/3009380779 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f19440b49a0 0x7f194409bdb0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:11.606 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.600+0000 7f194945f700 1 -- 172.21.15.98:0/3009380779 >> 172.21.15.98:0/3009380779 conn(0x7f19440997f0 msgr2=0x7f1944069a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:11.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.600+0000 7f194945f700 1 -- 172.21.15.98:0/3009380779 shutdown_connections 2023-10-06T05:07:11.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.600+0000 7f194945f700 1 -- 172.21.15.98:0/3009380779 wait complete. 2023-10-06T05:07:11.616 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":5,"active_gid":4104,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":90397},{"type":"v1","addr":"172.21.15.98:6833","nonce":90397}]},"active_addr":"172.21.15.98:6833/90397","active_change":"2023-10-06T05:06:38.511256+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":4101,"name":"x","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.98:0","nonce":1229857629}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.98:0","nonce":3786165554}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.98:0","nonce":1019129426}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.98:0","nonce":525771720}]}]} 2023-10-06T05:07:11.620 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-10-06T05:07:11.621 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-10-06T05:07:11.622 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T05:07:11.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.736+0000 7fb0db59c700 1 Processor -- start 2023-10-06T05:07:11.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.736+0000 7fb0db59c700 1 -- start start 2023-10-06T05:07:11.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.737+0000 7fb0db59c700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb0d4159440 0x7fb0d40b3090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:11.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.737+0000 7fb0db59c700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fb0d40b35d0 0x7fb0d40cbc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:11.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.737+0000 7fb0db59c700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0d40cb7a0 con 0x7fb0d40cc170 2023-10-06T05:07:11.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.737+0000 7fb0db59c700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0d40b0860 con 0x7fb0d40b35d0 2023-10-06T05:07:11.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.737+0000 7fb0db59c700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d40cb150 con 0x7fb0d4159440 2023-10-06T05:07:11.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.737+0000 7fb0dad9b700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fb0d40cc170 0x7fb0d40b2100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:41136/0 (socket says 172.21.15.98:41136) 2023-10-06T05:07:11.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.737+0000 7fb0dad9b700 1 -- 172.21.15.98:0/4057676322 learned_addr learned my addr 172.21.15.98:0/4057676322 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:07:11.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.737+0000 7fb0d9d99700 1 --2- 172.21.15.98:0/4057676322 >> v2:172.21.15.174:3300/0 conn(0x7fb0d40b35d0 0x7fb0d40cbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:11.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.737+0000 7fb0d9598700 1 -- 172.21.15.98:0/4057676322 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1822318131 0 0) 0x7fb0d40cb7a0 con 0x7fb0d40cc170 2023-10-06T05:07:11.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.737+0000 7fb0da59a700 1 --2- 172.21.15.98:0/4057676322 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb0d4159440 0x7fb0d40b3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:11.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.737+0000 7fb0d9598700 1 -- 172.21.15.98:0/4057676322 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0c0003570 con 0x7fb0d40cc170 2023-10-06T05:07:11.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.738+0000 7fb0d9598700 1 -- 172.21.15.98:0/4057676322 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 73701680 0 0) 0x7fb0c0003570 con 0x7fb0d40cc170 2023-10-06T05:07:11.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.738+0000 7fb0d9598700 1 -- 172.21.15.98:0/4057676322 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb0d4159440 msgr2=0x7fb0d40b3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:11.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.738+0000 7fb0d9598700 1 --2- 172.21.15.98:0/4057676322 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb0d4159440 0x7fb0d40b3090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:11.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.738+0000 7fb0d9598700 1 -- 172.21.15.98:0/4057676322 >> v2:172.21.15.174:3300/0 conn(0x7fb0d40b35d0 msgr2=0x7fb0d40cbc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:11.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.738+0000 7fb0d9598700 1 --2- 172.21.15.98:0/4057676322 >> v2:172.21.15.174:3300/0 conn(0x7fb0d40b35d0 0x7fb0d40cbc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:11.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.738+0000 7fb0d9598700 1 -- 172.21.15.98:0/4057676322 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0d40cb430 con 0x7fb0d40cc170 2023-10-06T05:07:11.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.738+0000 7fb0d9598700 1 -- 172.21.15.98:0/4057676322 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1543991876 0 0) 0x7fb0cc0017f0 con 0x7fb0d40cc170 2023-10-06T05:07:11.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.738+0000 7fb0d9598700 1 -- 172.21.15.98:0/4057676322 >> v1:172.21.15.98:6789/0 conn(0x7fb0d40cc170 legacy=0x7fb0d40b2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:11.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.738+0000 7fb0d9598700 1 --2- 172.21.15.98:0/4057676322 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb0c0003a60 0x7fb0c0003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:11.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.738+0000 7fb0d9598700 1 --2- 172.21.15.98:0/4057676322 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb0d4159440 0x7fb0c0004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:11.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.738+0000 7fb0d9598700 1 --2- 172.21.15.98:0/4057676322 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0c0004d50 0x7fb0c00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:11.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.738+0000 7fb0d9598700 1 -- 172.21.15.98:0/4057676322 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d40b0860 con 0x7fb0c0004d50 2023-10-06T05:07:11.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.738+0000 7fb0d9598700 1 -- 172.21.15.98:0/4057676322 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d40cb150 con 0x7fb0c0003a60 2023-10-06T05:07:11.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.738+0000 7fb0d9598700 1 -- 172.21.15.98:0/4057676322 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c000ba60 con 0x7fb0d4159440 2023-10-06T05:07:11.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.738+0000 7fb0dad9b700 1 --2- 172.21.15.98:0/4057676322 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0c0004d50 0x7fb0c00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:11.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.738+0000 7fb0da59a700 1 --2- 172.21.15.98:0/4057676322 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb0c0003a60 0x7fb0c0003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:11.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.738+0000 7fb0d9d99700 1 --2- 172.21.15.98:0/4057676322 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb0d4159440 0x7fb0c0004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:11.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.738+0000 7fb0dad9b700 1 -- 172.21.15.98:0/4057676322 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb0d4159440 msgr2=0x7fb0c0004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:11.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.738+0000 7fb0dad9b700 1 --2- 172.21.15.98:0/4057676322 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb0d4159440 0x7fb0c0004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:11.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.738+0000 7fb0dad9b700 1 -- 172.21.15.98:0/4057676322 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb0c0003a60 msgr2=0x7fb0c0003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:11.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.738+0000 7fb0dad9b700 1 --2- 172.21.15.98:0/4057676322 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb0c0003a60 0x7fb0c0003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:11.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.738+0000 7fb0dad9b700 1 -- 172.21.15.98:0/4057676322 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0c0003570 con 0x7fb0c0004d50 2023-10-06T05:07:11.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.738+0000 7fb0dad9b700 1 --2- 172.21.15.98:0/4057676322 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0c0004d50 0x7fb0c00051c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc00a970 tx=0x7fb0cc002210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef3cafe835925898 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:11.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.738+0000 7fb0d9598700 1 -- 172.21.15.98:0/4057676322 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0cc012440 con 0x7fb0c0004d50 2023-10-06T05:07:11.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.738+0000 7fb0d9598700 1 -- 172.21.15.98:0/4057676322 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0cc0125c0 con 0x7fb0c0004d50 2023-10-06T05:07:11.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.738+0000 7fb0d9598700 1 -- 172.21.15.98:0/4057676322 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0cc012980 con 0x7fb0c0004d50 2023-10-06T05:07:11.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.739+0000 7fb0db59c700 1 -- 172.21.15.98:0/4057676322 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0c0004d50 msgr2=0x7fb0c00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:11.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.739+0000 7fb0db59c700 1 --2- 172.21.15.98:0/4057676322 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0c0004d50 0x7fb0c00051c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc00a970 tx=0x7fb0cc002210 comp rx=0 tx=0).stop 2023-10-06T05:07:11.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.739+0000 7fb0dad9b700 1 -- 172.21.15.98:0/4057676322 reap_dead start 2023-10-06T05:07:11.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.739+0000 7fb0db59c700 1 -- 172.21.15.98:0/4057676322 shutdown_connections 2023-10-06T05:07:11.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.739+0000 7fb0db59c700 1 -- 172.21.15.98:0/4057676322 >> 172.21.15.98:0/4057676322 conn(0x7fb0d406ece0 msgr2=0x7fb0d406f0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:11.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.739+0000 7fb0db59c700 1 -- 172.21.15.98:0/4057676322 shutdown_connections 2023-10-06T05:07:11.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.739+0000 7fb0db59c700 1 -- 172.21.15.98:0/4057676322 wait complete. 2023-10-06T05:07:11.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.740+0000 7fb0db59c700 1 Processor -- start 2023-10-06T05:07:11.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.740+0000 7fb0db59c700 1 -- start start 2023-10-06T05:07:11.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.741+0000 7fb0db59c700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb0c0004d50 0x7fb0d415f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:11.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.741+0000 7fb0db59c700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb0d41547a0 0x7fb0d40cf230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:11.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.741+0000 7fb0db59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0d40b5220 0x7fb0d40b56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:11.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.741+0000 7fb0db59c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d40b5be0 con 0x7fb0d40b5220 2023-10-06T05:07:11.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.741+0000 7fb0db59c700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d40d0620 con 0x7fb0c0004d50 2023-10-06T05:07:11.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.741+0000 7fb0db59c700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0d40cf770 con 0x7fb0d41547a0 2023-10-06T05:07:11.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.741+0000 7fb0dad9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0d40b5220 0x7fb0d40b56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:11.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.741+0000 7fb0dad9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0d40b5220 0x7fb0d40b56a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45676/0 (socket says 172.21.15.98:45676) 2023-10-06T05:07:11.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.741+0000 7fb0dad9b700 1 -- 172.21.15.98:0/1295200686 learned_addr learned my addr 172.21.15.98:0/1295200686 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:11.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.741+0000 7fb0da59a700 1 --2- 172.21.15.98:0/1295200686 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb0c0004d50 0x7fb0d415f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:11.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.741+0000 7fb0d9d99700 1 --2- 172.21.15.98:0/1295200686 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb0d41547a0 0x7fb0d40cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:11.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.741+0000 7fb0dad9b700 1 -- 172.21.15.98:0/1295200686 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb0d41547a0 msgr2=0x7fb0d40cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:11.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.741+0000 7fb0dad9b700 1 --2- 172.21.15.98:0/1295200686 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb0d41547a0 0x7fb0d40cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:11.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.741+0000 7fb0dad9b700 1 -- 172.21.15.98:0/1295200686 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb0c0004d50 msgr2=0x7fb0d415f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:11.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.741+0000 7fb0dad9b700 1 --2- 172.21.15.98:0/1295200686 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb0c0004d50 0x7fb0d415f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:11.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.741+0000 7fb0dad9b700 1 -- 172.21.15.98:0/1295200686 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0d40cfa10 con 0x7fb0d40b5220 2023-10-06T05:07:11.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.741+0000 7fb0dad9b700 1 --2- 172.21.15.98:0/1295200686 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0d40b5220 0x7fb0d40b56a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb0c0004730 tx=0x7fb0cc015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:11.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.741+0000 7fb0ca7fc700 1 -- 172.21.15.98:0/1295200686 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0cc001360 con 0x7fb0d40b5220 2023-10-06T05:07:11.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.741+0000 7fb0ca7fc700 1 -- 172.21.15.98:0/1295200686 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0cc012440 con 0x7fb0d40b5220 2023-10-06T05:07:11.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.742+0000 7fb0db59c700 1 -- 172.21.15.98:0/1295200686 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0d40cfcf0 con 0x7fb0d40b5220 2023-10-06T05:07:11.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.742+0000 7fb0ca7fc700 1 -- 172.21.15.98:0/1295200686 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0cc0017b0 con 0x7fb0d40b5220 2023-10-06T05:07:11.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.742+0000 7fb0db59c700 1 -- 172.21.15.98:0/1295200686 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0d40d02b0 con 0x7fb0d40b5220 2023-10-06T05:07:11.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.743+0000 7fb0db59c700 1 -- 172.21.15.98:0/1295200686 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0d40c3980 con 0x7fb0d40b5220 2023-10-06T05:07:11.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.743+0000 7fb0ca7fc700 1 -- 172.21.15.98:0/1295200686 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fb0cc019070 con 0x7fb0d40b5220 2023-10-06T05:07:11.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.743+0000 7fb0ca7fc700 1 --2- 172.21.15.98:0/1295200686 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fb0b8066f30 0x7fb0b80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:11.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.743+0000 7fb0ca7fc700 1 -- 172.21.15.98:0/1295200686 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fb0cc086390 con 0x7fb0d40b5220 2023-10-06T05:07:11.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.743+0000 7fb0da59a700 1 --2- 172.21.15.98:0/1295200686 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fb0b8066f30 0x7fb0b80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:11.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.744+0000 7fb0da59a700 1 --2- 172.21.15.98:0/1295200686 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fb0b8066f30 0x7fb0b80693e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb0d0006f00 tx=0x7fb0d00091a0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:11.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.747+0000 7fb0ca7fc700 1 -- 172.21.15.98:0/1295200686 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb0cc053260 con 0x7fb0d40b5220 2023-10-06T05:07:11.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.933+0000 7fb0db59c700 1 -- 172.21.15.98:0/1295200686 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb0d40d07b0 con 0x7fb0d40b5220 2023-10-06T05:07:11.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.933+0000 7fb0ca7fc700 1 -- 172.21.15.98:0/1295200686 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+18707 (secure 0 0 0) 0x7fb0cc012940 con 0x7fb0d40b5220 2023-10-06T05:07:11.936 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:07:11.937 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":29,"fsid":"f50c5df5-cad5-47a4-9bb1-8c791d4311ec","created":"2023-10-06T05:06:32.971175+0000","modified":"2023-10-06T05:07:02.131672+0000","last_up_change":"2023-10-06T05:06:44.016489+0000","last_in_change":"2023-10-06T05:06:35.544511+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-06T05:06:44.559671+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-06T05:06:51.432161+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-10-06T05:06:55.589057+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-10-06T05:06:56.041687+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-10-06T05:06:59.754099+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d9495d95-da75-4c3b-8927-e9bae6a8b093","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":90627},{"type":"v1","addr":"172.21.15.98:6801","nonce":90627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":90627},{"type":"v1","addr":"172.21.15.98:6803","nonce":90627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":90627},{"type":"v1","addr":"172.21.15.98:6807","nonce":90627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":90627},{"type":"v1","addr":"172.21.15.98:6805","nonce":90627}]},"public_addr":"172.21.15.98:6801/90627","cluster_addr":"172.21.15.98:6803/90627","heartbeat_back_addr":"172.21.15.98:6807/90627","heartbeat_front_addr":"172.21.15.98:6805/90627","state":["exists","up"]},{"osd":1,"uuid":"f25ba5be-d65e-4707-a134-154092deb754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":90628},{"type":"v1","addr":"172.21.15.98:6809","nonce":90628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":90628},{"type":"v1","addr":"172.21.15.98:6811","nonce":90628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":90628},{"type":"v1","addr":"172.21.15.98:6815","nonce":90628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":90628},{"type":"v1","addr":"172.21.15.98:6813","nonce":90628}]},"public_addr":"172.21.15.98:6809/90628","cluster_addr":"172.21.15.98:6811/90628","heartbeat_back_addr":"172.21.15.98:6815/90628","heartbeat_front_addr":"172.21.15.98:6813/90628","state":["exists","up"]},{"osd":2,"uuid":"5ab9e028-945c-43e2-8c95-3b0b4cf38e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":90662},{"type":"v1","addr":"172.21.15.98:6818","nonce":90662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":90662},{"type":"v1","addr":"172.21.15.98:6822","nonce":90662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":90662},{"type":"v1","addr":"172.21.15.98:6830","nonce":90662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":90662},{"type":"v1","addr":"172.21.15.98:6826","nonce":90662}]},"public_addr":"172.21.15.98:6818/90662","cluster_addr":"172.21.15.98:6822/90662","heartbeat_back_addr":"172.21.15.98:6830/90662","heartbeat_front_addr":"172.21.15.98:6826/90662","state":["exists","up"]},{"osd":3,"uuid":"3bb803ff-1738-46bf-be62-2267bf378822","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6817","nonce":90663},{"type":"v1","addr":"172.21.15.98:6819","nonce":90663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6821","nonce":90663},{"type":"v1","addr":"172.21.15.98:6823","nonce":90663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6829","nonce":90663},{"type":"v1","addr":"172.21.15.98:6831","nonce":90663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6825","nonce":90663},{"type":"v1","addr":"172.21.15.98:6827","nonce":90663}]},"public_addr":"172.21.15.98:6819/90663","cluster_addr":"172.21.15.98:6823/90663","heartbeat_back_addr":"172.21.15.98:6831/90663","heartbeat_front_addr":"172.21.15.98:6827/90663","state":["exists","up"]},{"osd":4,"uuid":"7d57a4c9-6462-43f6-b64a-7f9e9b4353d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":90847},{"type":"v1","addr":"172.21.15.174:6801","nonce":90847}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":90847},{"type":"v1","addr":"172.21.15.174:6803","nonce":90847}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":90847},{"type":"v1","addr":"172.21.15.174:6807","nonce":90847}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":90847},{"type":"v1","addr":"172.21.15.174:6805","nonce":90847}]},"public_addr":"172.21.15.174:6801/90847","cluster_addr":"172.21.15.174:6803/90847","heartbeat_back_addr":"172.21.15.174:6807/90847","heartbeat_front_addr":"172.21.15.174:6805/90847","state":["exists","up"]},{"osd":5,"uuid":"83287ea4-e44d-4af9-bc77-36d47c60d4a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":90848},{"type":"v1","addr":"172.21.15.174:6809","nonce":90848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":90848},{"type":"v1","addr":"172.21.15.174:6811","nonce":90848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":90848},{"type":"v1","addr":"172.21.15.174:6815","nonce":90848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":90848},{"type":"v1","addr":"172.21.15.174:6813","nonce":90848}]},"public_addr":"172.21.15.174:6809/90848","cluster_addr":"172.21.15.174:6811/90848","heartbeat_back_addr":"172.21.15.174:6815/90848","heartbeat_front_addr":"172.21.15.174:6813/90848","state":["exists","up"]},{"osd":6,"uuid":"496a6272-892e-43c4-a145-206623c22f99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":90877},{"type":"v1","addr":"172.21.15.174:6817","nonce":90877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":90877},{"type":"v1","addr":"172.21.15.174:6819","nonce":90877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":90877},{"type":"v1","addr":"172.21.15.174:6823","nonce":90877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":90877},{"type":"v1","addr":"172.21.15.174:6821","nonce":90877}]},"public_addr":"172.21.15.174:6817/90877","cluster_addr":"172.21.15.174:6819/90877","heartbeat_back_addr":"172.21.15.174:6823/90877","heartbeat_front_addr":"172.21.15.174:6821/90877","state":["exists","up"]},{"osd":7,"uuid":"49967e34-8b07-4bab-a13e-0c16d6a0f3a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":90878},{"type":"v1","addr":"172.21.15.174:6825","nonce":90878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":90878},{"type":"v1","addr":"172.21.15.174:6827","nonce":90878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":90878},{"type":"v1","addr":"172.21.15.174:6831","nonce":90878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":90878},{"type":"v1","addr":"172.21.15.174:6829","nonce":90878}]},"public_addr":"172.21.15.174:6825/90878","cluster_addr":"172.21.15.174:6827/90878","heartbeat_back_addr":"172.21.15.174:6831/90878","heartbeat_front_addr":"172.21.15.174:6829/90878","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.446479+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.514266+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.794108+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.810894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.731919+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.717756+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.737435+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.508682+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T05:07:11.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.936+0000 7fb0db59c700 1 -- 172.21.15.98:0/1295200686 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fb0b8066f30 msgr2=0x7fb0b80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:11.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.936+0000 7fb0db59c700 1 --2- 172.21.15.98:0/1295200686 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fb0b8066f30 0x7fb0b80693e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb0d0006f00 tx=0x7fb0d00091a0 comp rx=0 tx=0).stop 2023-10-06T05:07:11.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.936+0000 7fb0db59c700 1 -- 172.21.15.98:0/1295200686 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0d40b5220 msgr2=0x7fb0d40b56a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:11.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.936+0000 7fb0db59c700 1 --2- 172.21.15.98:0/1295200686 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0d40b5220 0x7fb0d40b56a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb0c0004730 tx=0x7fb0cc015000 comp rx=0 tx=0).stop 2023-10-06T05:07:11.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.936+0000 7fb0db59c700 1 -- 172.21.15.98:0/1295200686 shutdown_connections 2023-10-06T05:07:11.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.936+0000 7fb0db59c700 1 --2- 172.21.15.98:0/1295200686 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fb0b8066f30 0x7fb0b80693e0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:11.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.936+0000 7fb0db59c700 1 --2- 172.21.15.98:0/1295200686 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0d40b5220 0x7fb0d40b56a0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:11.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.936+0000 7fb0db59c700 1 --2- 172.21.15.98:0/1295200686 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb0c0004d50 0x7fb0d415f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:11.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.936+0000 7fb0db59c700 1 --2- 172.21.15.98:0/1295200686 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb0d41547a0 0x7fb0d40cf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:11.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.936+0000 7fb0db59c700 1 -- 172.21.15.98:0/1295200686 >> 172.21.15.98:0/1295200686 conn(0x7fb0d406ece0 msgr2=0x7fb0d41585f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:11.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.936+0000 7fb0db59c700 1 -- 172.21.15.98:0/1295200686 shutdown_connections 2023-10-06T05:07:11.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:11.936+0000 7fb0db59c700 1 -- 172.21.15.98:0/1295200686 wait complete. 2023-10-06T05:07:11.951 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-10-06T05:07:11.951 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T05:07:12.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.064+0000 7f8d4ead3700 1 Processor -- start 2023-10-06T05:07:12.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.064+0000 7f8d4ead3700 1 -- start start 2023-10-06T05:07:12.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.065+0000 7f8d4ead3700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f8d48159560 0x7f8d480aa530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.065+0000 7f8d4ead3700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8d480b0760 0x7f8d480b1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.065+0000 7f8d4ead3700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d480b0c60 con 0x7f8d480b49a0 2023-10-06T05:07:12.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.065+0000 7f8d4ead3700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d480b0f40 con 0x7f8d48159560 2023-10-06T05:07:12.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.065+0000 7f8d4ead3700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d480b1220 con 0x7f8d480b0760 2023-10-06T05:07:12.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.065+0000 7f8d47fff700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f8d480b49a0 0x7f8d480aaa70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:41146/0 (socket says 172.21.15.98:41146) 2023-10-06T05:07:12.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.065+0000 7f8d47fff700 1 -- 172.21.15.98:0/2577609260 learned_addr learned my addr 172.21.15.98:0/2577609260 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:07:12.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.065+0000 7f8d4c86f700 1 --2- 172.21.15.98:0/2577609260 >> v2:172.21.15.174:3300/0 conn(0x7f8d48159560 0x7f8d480aa530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.065+0000 7f8d4d070700 1 --2- 172.21.15.98:0/2577609260 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8d480b0760 0x7f8d480b1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.065+0000 7f8d477fe700 1 -- 172.21.15.98:0/2577609260 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1739665026 0 0) 0x7f8d480b0c60 con 0x7f8d480b49a0 2023-10-06T05:07:12.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.065+0000 7f8d477fe700 1 -- 172.21.15.98:0/2577609260 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d34003570 con 0x7f8d480b49a0 2023-10-06T05:07:12.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.065+0000 7f8d4d070700 1 -- 172.21.15.98:0/2577609260 >> v2:172.21.15.174:3300/0 conn(0x7f8d48159560 msgr2=0x7f8d480aa530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:12.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.065+0000 7f8d4d070700 1 --2- 172.21.15.98:0/2577609260 >> v2:172.21.15.174:3300/0 conn(0x7f8d48159560 0x7f8d480aa530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.065+0000 7f8d4d070700 1 -- 172.21.15.98:0/2577609260 >> v1:172.21.15.98:6789/0 conn(0x7f8d480b49a0 legacy=0x7f8d480aaa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.065+0000 7f8d4d070700 1 -- 172.21.15.98:0/2577609260 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d480b1500 con 0x7f8d480b0760 2023-10-06T05:07:12.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.065+0000 7f8d4c86f700 1 --2- 172.21.15.98:0/2577609260 >> v2:172.21.15.174:3300/0 conn(0x7f8d48159560 0x7f8d480aa530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T05:07:12.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.065+0000 7f8d4d070700 1 --2- 172.21.15.98:0/2577609260 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8d480b0760 0x7f8d480b1bc0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f8d30009910 tx=0x7f8d300091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec11dbd960ad9da8 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:12.072 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.066+0000 7f8d477fe700 1 -- 172.21.15.98:0/2577609260 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d30011630 con 0x7f8d480b0760 2023-10-06T05:07:12.072 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.066+0000 7f8d477fe700 1 -- 172.21.15.98:0/2577609260 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d30011790 con 0x7f8d480b0760 2023-10-06T05:07:12.073 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.066+0000 7f8d477fe700 1 -- 172.21.15.98:0/2577609260 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d30011b30 con 0x7f8d480b0760 2023-10-06T05:07:12.073 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.066+0000 7f8d4ead3700 1 -- 172.21.15.98:0/2577609260 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8d480b0760 msgr2=0x7f8d480b1bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.073 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.066+0000 7f8d4ead3700 1 --2- 172.21.15.98:0/2577609260 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8d480b0760 0x7f8d480b1bc0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f8d30009910 tx=0x7f8d300091a0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.073 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.066+0000 7f8d4ead3700 1 -- 172.21.15.98:0/2577609260 shutdown_connections 2023-10-06T05:07:12.073 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.066+0000 7f8d4ead3700 1 --2- 172.21.15.98:0/2577609260 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8d480b0760 0x7f8d480b1bc0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.066+0000 7f8d4ead3700 1 --2- 172.21.15.98:0/2577609260 >> v2:172.21.15.174:3300/0 conn(0x7f8d48159560 0x7f8d480aa530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.066+0000 7f8d4ead3700 1 -- 172.21.15.98:0/2577609260 >> 172.21.15.98:0/2577609260 conn(0x7f8d480997f0 msgr2=0x7f8d4806f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:12.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.066+0000 7f8d4ead3700 1 -- 172.21.15.98:0/2577609260 shutdown_connections 2023-10-06T05:07:12.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.066+0000 7f8d4ead3700 1 -- 172.21.15.98:0/2577609260 wait complete. 2023-10-06T05:07:12.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.067+0000 7f8d4ead3700 1 Processor -- start 2023-10-06T05:07:12.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.068+0000 7f8d4ead3700 1 -- start start 2023-10-06T05:07:12.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.068+0000 7f8d4ead3700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8d480b0760 0x7f8d48099f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.068+0000 7f8d4ead3700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8d480b49a0 0x7f8d4814fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.068+0000 7f8d4ead3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d48159560 0x7f8d480ab200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.068+0000 7f8d4ead3700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d4809a850 con 0x7f8d48159560 2023-10-06T05:07:12.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.068+0000 7f8d4ead3700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d4809ab30 con 0x7f8d480b49a0 2023-10-06T05:07:12.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.068+0000 7f8d4ead3700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d4809ae10 con 0x7f8d480b0760 2023-10-06T05:07:12.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.068+0000 7f8d4d070700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d48159560 0x7f8d480ab200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.068+0000 7f8d4d070700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d48159560 0x7f8d480ab200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45684/0 (socket says 172.21.15.98:45684) 2023-10-06T05:07:12.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.068+0000 7f8d4d070700 1 -- 172.21.15.98:0/1753907164 learned_addr learned my addr 172.21.15.98:0/1753907164 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:12.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.069+0000 7f8d4c86f700 1 --2- 172.21.15.98:0/1753907164 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8d480b0760 0x7f8d48099f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.069+0000 7f8d47fff700 1 --2- 172.21.15.98:0/1753907164 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8d480b49a0 0x7f8d4814fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.069+0000 7f8d4d070700 1 -- 172.21.15.98:0/1753907164 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8d480b0760 msgr2=0x7f8d48099f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.069+0000 7f8d4d070700 1 --2- 172.21.15.98:0/1753907164 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8d480b0760 0x7f8d48099f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.069+0000 7f8d4d070700 1 -- 172.21.15.98:0/1753907164 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8d480b49a0 msgr2=0x7f8d4814fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.069+0000 7f8d4d070700 1 --2- 172.21.15.98:0/1753907164 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8d480b49a0 0x7f8d4814fc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.069+0000 7f8d4d070700 1 -- 172.21.15.98:0/1753907164 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d40009a20 con 0x7f8d48159560 2023-10-06T05:07:12.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.069+0000 7f8d4d070700 1 --2- 172.21.15.98:0/1753907164 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d48159560 0x7f8d480ab200 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f8d30013000 tx=0x7f8d30017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:12.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.069+0000 7f8d44ff9700 1 -- 172.21.15.98:0/1753907164 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d30025400 con 0x7f8d48159560 2023-10-06T05:07:12.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.069+0000 7f8d44ff9700 1 -- 172.21.15.98:0/1753907164 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d30025560 con 0x7f8d48159560 2023-10-06T05:07:12.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.069+0000 7f8d4ead3700 1 -- 172.21.15.98:0/1753907164 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d30009420 con 0x7f8d48159560 2023-10-06T05:07:12.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.069+0000 7f8d44ff9700 1 -- 172.21.15.98:0/1753907164 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d30025900 con 0x7f8d48159560 2023-10-06T05:07:12.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.069+0000 7f8d4ead3700 1 -- 172.21.15.98:0/1753907164 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d4809b3d0 con 0x7f8d48159560 2023-10-06T05:07:12.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.070+0000 7f8d4ead3700 1 -- 172.21.15.98:0/1753907164 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d48089c60 con 0x7f8d48159560 2023-10-06T05:07:12.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.070+0000 7f8d44ff9700 1 -- 172.21.15.98:0/1753907164 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f8d300193b0 con 0x7f8d48159560 2023-10-06T05:07:12.080 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.071+0000 7f8d44ff9700 1 --2- 172.21.15.98:0/1753907164 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f8d28066f30 0x7f8d280693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.080 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.071+0000 7f8d44ff9700 1 -- 172.21.15.98:0/1753907164 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f8d30086910 con 0x7f8d48159560 2023-10-06T05:07:12.080 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.071+0000 7f8d4c86f700 1 --2- 172.21.15.98:0/1753907164 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f8d28066f30 0x7f8d280693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.080 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.071+0000 7f8d4c86f700 1 --2- 172.21.15.98:0/1753907164 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f8d28066f30 0x7f8d280693e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8d400099f0 tx=0x7f8d4000b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:12.080 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.075+0000 7f8d44ff9700 1 -- 172.21.15.98:0/1753907164 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8d30053730 con 0x7f8d48159560 2023-10-06T05:07:12.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.258+0000 7f8d4ead3700 1 -- 172.21.15.98:0/1753907164 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8d48064a90 con 0x7f8d48159560 2023-10-06T05:07:12.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.259+0000 7f8d44ff9700 1 -- 172.21.15.98:0/1753907164 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+18707 (secure 0 0 0) 0x7f8d30056d50 con 0x7f8d48159560 2023-10-06T05:07:12.261 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:07:12.261 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":29,"fsid":"f50c5df5-cad5-47a4-9bb1-8c791d4311ec","created":"2023-10-06T05:06:32.971175+0000","modified":"2023-10-06T05:07:02.131672+0000","last_up_change":"2023-10-06T05:06:44.016489+0000","last_in_change":"2023-10-06T05:06:35.544511+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-06T05:06:44.559671+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-06T05:06:51.432161+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-10-06T05:06:55.589057+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-10-06T05:06:56.041687+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-10-06T05:06:59.754099+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d9495d95-da75-4c3b-8927-e9bae6a8b093","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":90627},{"type":"v1","addr":"172.21.15.98:6801","nonce":90627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":90627},{"type":"v1","addr":"172.21.15.98:6803","nonce":90627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":90627},{"type":"v1","addr":"172.21.15.98:6807","nonce":90627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":90627},{"type":"v1","addr":"172.21.15.98:6805","nonce":90627}]},"public_addr":"172.21.15.98:6801/90627","cluster_addr":"172.21.15.98:6803/90627","heartbeat_back_addr":"172.21.15.98:6807/90627","heartbeat_front_addr":"172.21.15.98:6805/90627","state":["exists","up"]},{"osd":1,"uuid":"f25ba5be-d65e-4707-a134-154092deb754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":90628},{"type":"v1","addr":"172.21.15.98:6809","nonce":90628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":90628},{"type":"v1","addr":"172.21.15.98:6811","nonce":90628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":90628},{"type":"v1","addr":"172.21.15.98:6815","nonce":90628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":90628},{"type":"v1","addr":"172.21.15.98:6813","nonce":90628}]},"public_addr":"172.21.15.98:6809/90628","cluster_addr":"172.21.15.98:6811/90628","heartbeat_back_addr":"172.21.15.98:6815/90628","heartbeat_front_addr":"172.21.15.98:6813/90628","state":["exists","up"]},{"osd":2,"uuid":"5ab9e028-945c-43e2-8c95-3b0b4cf38e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":90662},{"type":"v1","addr":"172.21.15.98:6818","nonce":90662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":90662},{"type":"v1","addr":"172.21.15.98:6822","nonce":90662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":90662},{"type":"v1","addr":"172.21.15.98:6830","nonce":90662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":90662},{"type":"v1","addr":"172.21.15.98:6826","nonce":90662}]},"public_addr":"172.21.15.98:6818/90662","cluster_addr":"172.21.15.98:6822/90662","heartbeat_back_addr":"172.21.15.98:6830/90662","heartbeat_front_addr":"172.21.15.98:6826/90662","state":["exists","up"]},{"osd":3,"uuid":"3bb803ff-1738-46bf-be62-2267bf378822","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6817","nonce":90663},{"type":"v1","addr":"172.21.15.98:6819","nonce":90663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6821","nonce":90663},{"type":"v1","addr":"172.21.15.98:6823","nonce":90663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6829","nonce":90663},{"type":"v1","addr":"172.21.15.98:6831","nonce":90663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6825","nonce":90663},{"type":"v1","addr":"172.21.15.98:6827","nonce":90663}]},"public_addr":"172.21.15.98:6819/90663","cluster_addr":"172.21.15.98:6823/90663","heartbeat_back_addr":"172.21.15.98:6831/90663","heartbeat_front_addr":"172.21.15.98:6827/90663","state":["exists","up"]},{"osd":4,"uuid":"7d57a4c9-6462-43f6-b64a-7f9e9b4353d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":90847},{"type":"v1","addr":"172.21.15.174:6801","nonce":90847}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":90847},{"type":"v1","addr":"172.21.15.174:6803","nonce":90847}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":90847},{"type":"v1","addr":"172.21.15.174:6807","nonce":90847}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":90847},{"type":"v1","addr":"172.21.15.174:6805","nonce":90847}]},"public_addr":"172.21.15.174:6801/90847","cluster_addr":"172.21.15.174:6803/90847","heartbeat_back_addr":"172.21.15.174:6807/90847","heartbeat_front_addr":"172.21.15.174:6805/90847","state":["exists","up"]},{"osd":5,"uuid":"83287ea4-e44d-4af9-bc77-36d47c60d4a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":90848},{"type":"v1","addr":"172.21.15.174:6809","nonce":90848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":90848},{"type":"v1","addr":"172.21.15.174:6811","nonce":90848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":90848},{"type":"v1","addr":"172.21.15.174:6815","nonce":90848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":90848},{"type":"v1","addr":"172.21.15.174:6813","nonce":90848}]},"public_addr":"172.21.15.174:6809/90848","cluster_addr":"172.21.15.174:6811/90848","heartbeat_back_addr":"172.21.15.174:6815/90848","heartbeat_front_addr":"172.21.15.174:6813/90848","state":["exists","up"]},{"osd":6,"uuid":"496a6272-892e-43c4-a145-206623c22f99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":90877},{"type":"v1","addr":"172.21.15.174:6817","nonce":90877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":90877},{"type":"v1","addr":"172.21.15.174:6819","nonce":90877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":90877},{"type":"v1","addr":"172.21.15.174:6823","nonce":90877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":90877},{"type":"v1","addr":"172.21.15.174:6821","nonce":90877}]},"public_addr":"172.21.15.174:6817/90877","cluster_addr":"172.21.15.174:6819/90877","heartbeat_back_addr":"172.21.15.174:6823/90877","heartbeat_front_addr":"172.21.15.174:6821/90877","state":["exists","up"]},{"osd":7,"uuid":"49967e34-8b07-4bab-a13e-0c16d6a0f3a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":90878},{"type":"v1","addr":"172.21.15.174:6825","nonce":90878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":90878},{"type":"v1","addr":"172.21.15.174:6827","nonce":90878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":90878},{"type":"v1","addr":"172.21.15.174:6831","nonce":90878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":90878},{"type":"v1","addr":"172.21.15.174:6829","nonce":90878}]},"public_addr":"172.21.15.174:6825/90878","cluster_addr":"172.21.15.174:6827/90878","heartbeat_back_addr":"172.21.15.174:6831/90878","heartbeat_front_addr":"172.21.15.174:6829/90878","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.446479+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.514266+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.794108+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.810894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.731919+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.717756+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.737435+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.508682+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T05:07:12.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.261+0000 7f8d4ead3700 1 -- 172.21.15.98:0/1753907164 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f8d28066f30 msgr2=0x7f8d280693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.261+0000 7f8d4ead3700 1 --2- 172.21.15.98:0/1753907164 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f8d28066f30 0x7f8d280693e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8d400099f0 tx=0x7f8d4000b040 comp rx=0 tx=0).stop 2023-10-06T05:07:12.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.261+0000 7f8d4ead3700 1 -- 172.21.15.98:0/1753907164 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d48159560 msgr2=0x7f8d480ab200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.261+0000 7f8d4ead3700 1 --2- 172.21.15.98:0/1753907164 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d48159560 0x7f8d480ab200 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f8d30013000 tx=0x7f8d30017000 comp rx=0 tx=0).stop 2023-10-06T05:07:12.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.261+0000 7f8d4ead3700 1 -- 172.21.15.98:0/1753907164 shutdown_connections 2023-10-06T05:07:12.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.261+0000 7f8d4ead3700 1 --2- 172.21.15.98:0/1753907164 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f8d28066f30 0x7f8d280693e0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.261+0000 7f8d4ead3700 1 --2- 172.21.15.98:0/1753907164 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d48159560 0x7f8d480ab200 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.261+0000 7f8d4ead3700 1 --2- 172.21.15.98:0/1753907164 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8d480b0760 0x7f8d48099f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.261+0000 7f8d4ead3700 1 --2- 172.21.15.98:0/1753907164 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8d480b49a0 0x7f8d4814fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.261+0000 7f8d4ead3700 1 -- 172.21.15.98:0/1753907164 >> 172.21.15.98:0/1753907164 conn(0x7f8d480997f0 msgr2=0x7f8d480b3230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:12.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.261+0000 7f8d4ead3700 1 -- 172.21.15.98:0/1753907164 shutdown_connections 2023-10-06T05:07:12.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.261+0000 7f8d4ead3700 1 -- 172.21.15.98:0/1753907164 wait complete. 2023-10-06T05:07:12.276 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-10-06T05:07:12.277 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-10-06T05:07:12.278 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-10-06T05:07:12.278 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-10-06T05:07:12.279 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-10-06T05:07:12.279 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-10-06T05:07:12.280 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-10-06T05:07:12.281 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-10-06T05:07:12.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.476+0000 7f745b16a700 1 Processor -- start 2023-10-06T05:07:12.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.476+0000 7f745b16a700 1 -- start start 2023-10-06T05:07:12.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.477+0000 7f9594a5e700 1 Processor -- start 2023-10-06T05:07:12.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.477+0000 7f9594a5e700 1 -- start start 2023-10-06T05:07:12.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.477+0000 7f745b16a700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7454160df0 0x7f74541611c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.477+0000 7f745b16a700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f74540c8750 0x7f74540aeeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.477+0000 7f745b16a700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74540c7e40 con 0x7f74540c82d0 2023-10-06T05:07:12.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.477+0000 7f745b16a700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74540c7230 con 0x7f7454160df0 2023-10-06T05:07:12.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.477+0000 7f745b16a700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74540c7510 con 0x7f74540c8750 2023-10-06T05:07:12.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.478+0000 7f5f55126700 1 Processor -- start 2023-10-06T05:07:12.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.478+0000 7f5f55126700 1 -- start start 2023-10-06T05:07:12.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.478+0000 7f7458f06700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7454160df0 0x7f74541611c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.478+0000 7f7458f06700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7454160df0 0x7f74541611c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45688/0 (socket says 172.21.15.98:45688) 2023-10-06T05:07:12.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.478+0000 7f7458f06700 1 -- 172.21.15.98:0/2371448408 learned_addr learned my addr 172.21.15.98:0/2371448408 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:12.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.478+0000 7f7458f06700 1 -- 172.21.15.98:0/2371448408 >> v2:172.21.15.174:3300/0 conn(0x7f74540c8750 msgr2=0x7f74540aeeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:12.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.478+0000 7f7459707700 1 --2- 172.21.15.98:0/2371448408 >> v2:172.21.15.174:3300/0 conn(0x7f74540c8750 0x7f74540aeeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.478+0000 7f7458f06700 1 --2- 172.21.15.98:0/2371448408 >> v2:172.21.15.174:3300/0 conn(0x7f74540c8750 0x7f74540aeeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.478+0000 7f7458f06700 1 -- 172.21.15.98:0/2371448408 >> v1:172.21.15.174:6789/0 conn(0x7f74540c82d0 legacy=0x7f7454069520 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-06T05:07:12.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.478+0000 7f7458f06700 1 -- 172.21.15.98:0/2371448408 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74540c77f0 con 0x7f7454160df0 2023-10-06T05:07:12.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.478+0000 7f7458f06700 1 --2- 172.21.15.98:0/2371448408 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7454160df0 0x7f74541611c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f745406e6f0 tx=0x7f7440009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f351b2df24299d2e server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:12.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.479+0000 7f9594a5e700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9590157470 0x7f95900883c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.479+0000 7f9594a5e700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f95900ca5b0 0x7f9590157840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.479+0000 7f9594a5e700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95900c9cb0 con 0x7f95900ca140 2023-10-06T05:07:12.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.479+0000 7f9594a5e700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95900cf050 con 0x7f9590157470 2023-10-06T05:07:12.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.479+0000 7f9594a5e700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95900ccee0 con 0x7f95900ca5b0 2023-10-06T05:07:12.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.479+0000 7f5f55126700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5f500ccc40 0x7f5f50069520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.485 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.479+0000 7f5f55126700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f500c7330 con 0x7f5f500cc7f0 2023-10-06T05:07:12.485 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.479+0000 7f5f55126700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f500c6720 con 0x7f5f500ac000 2023-10-06T05:07:12.485 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.479+0000 7f5f55126700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f50156110 con 0x7f5f500ccc40 2023-10-06T05:07:12.485 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.479+0000 7f74537fe700 1 -- 172.21.15.98:0/2371448408 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f744000b820 con 0x7f7454160df0 2023-10-06T05:07:12.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.479+0000 7f74537fe700 1 -- 172.21.15.98:0/2371448408 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f744000b980 con 0x7f7454160df0 2023-10-06T05:07:12.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.479+0000 7f74537fe700 1 -- 172.21.15.98:0/2371448408 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f744000bd20 con 0x7f7454160df0 2023-10-06T05:07:12.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.479+0000 7f958e59c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9590157470 0x7f95900883c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.479+0000 7f958e59c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9590157470 0x7f95900883c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45692/0 (socket says 172.21.15.98:45692) 2023-10-06T05:07:12.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.479+0000 7f958e59c700 1 -- 172.21.15.98:0/3343199314 learned_addr learned my addr 172.21.15.98:0/3343199314 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:12.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.479+0000 7f1009769700 1 Processor -- start 2023-10-06T05:07:12.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.479+0000 7f1009769700 1 -- start start 2023-10-06T05:07:12.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.480+0000 7f958e59c700 1 -- 172.21.15.98:0/3343199314 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f95900ca5b0 msgr2=0x7f9590157840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:12.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.480+0000 7f958e59c700 1 --2- 172.21.15.98:0/3343199314 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f95900ca5b0 0x7f9590157840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.480+0000 7f958e59c700 1 -- 172.21.15.98:0/3343199314 >> v1:172.21.15.174:6789/0 conn(0x7f95900ca140 legacy=0x7f95900c3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:12.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.480+0000 7f0b0e35f700 1 Processor -- start 2023-10-06T05:07:12.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.479+0000 7f5f4f59e700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5f500ccc40 0x7f5f50069520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.479+0000 7f5f4f59e700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5f500ccc40 0x7f5f50069520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45708/0 (socket says 172.21.15.98:45708) 2023-10-06T05:07:12.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.479+0000 7f5f4f59e700 1 -- 172.21.15.98:0/3571581363 learned_addr learned my addr 172.21.15.98:0/3571581363 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:12.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.479+0000 7f5f4dd9b700 1 -- 172.21.15.98:0/3571581363 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4118833181 0 0) 0x7f5f500c7330 con 0x7f5f500cc7f0 2023-10-06T05:07:12.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.479+0000 7f5f4dd9b700 1 -- 172.21.15.98:0/3571581363 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f38003570 con 0x7f5f500cc7f0 2023-10-06T05:07:12.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.480+0000 7f5f4f59e700 1 -- 172.21.15.98:0/3571581363 >> v1:172.21.15.174:6789/0 conn(0x7f5f500ac000 legacy=0x7f5f500adf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:12.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.480+0000 7f5f4f59e700 1 -- 172.21.15.98:0/3571581363 >> v1:172.21.15.98:6789/0 conn(0x7f5f500cc7f0 legacy=0x7f5f5006e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.480+0000 7f745b16a700 1 -- 172.21.15.98:0/2371448408 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7454160df0 msgr2=0x7f74541611c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.480+0000 7f745b16a700 1 --2- 172.21.15.98:0/2371448408 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7454160df0 0x7f74541611c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f745406e6f0 tx=0x7f7440009250 comp rx=0 tx=0).stop 2023-10-06T05:07:12.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.480+0000 7f5f4f59e700 1 -- 172.21.15.98:0/3571581363 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f500c6a00 con 0x7f5f500ccc40 2023-10-06T05:07:12.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.480+0000 7f958e59c700 1 -- 172.21.15.98:0/3343199314 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95900cd180 con 0x7f9590157470 2023-10-06T05:07:12.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.480+0000 7f745b16a700 1 -- 172.21.15.98:0/2371448408 shutdown_connections 2023-10-06T05:07:12.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.480+0000 7f745b16a700 1 --2- 172.21.15.98:0/2371448408 >> v2:172.21.15.174:3300/0 conn(0x7f74540c8750 0x7f74540aeeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.480+0000 7f745b16a700 1 --2- 172.21.15.98:0/2371448408 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7454160df0 0x7f74541611c0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.480+0000 7f745b16a700 1 -- 172.21.15.98:0/2371448408 >> 172.21.15.98:0/2371448408 conn(0x7f7454068080 msgr2=0x7f74540cc0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:12.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.480+0000 7f5f4f59e700 1 --2- 172.21.15.98:0/3571581363 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5f500ccc40 0x7f5f50069520 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f5f48009f20 tx=0x7f5f48009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5dfdec0a1b6b37c server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:12.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.480+0000 7f958e59c700 1 --2- 172.21.15.98:0/3343199314 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9590157470 0x7f95900883c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f95800099b0 tx=0x7f9580009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcca711aaa6851a6 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:12.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.480+0000 7f745b16a700 1 -- 172.21.15.98:0/2371448408 shutdown_connections 2023-10-06T05:07:12.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.480+0000 7f745b16a700 1 -- 172.21.15.98:0/2371448408 wait complete. 2023-10-06T05:07:12.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.480+0000 7f958d59a700 1 -- 172.21.15.98:0/3343199314 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f958000b860 con 0x7f9590157470 2023-10-06T05:07:12.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.480+0000 7f958d59a700 1 -- 172.21.15.98:0/3343199314 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f958000b9c0 con 0x7f9590157470 2023-10-06T05:07:12.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.480+0000 7f958d59a700 1 -- 172.21.15.98:0/3343199314 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f958000bd60 con 0x7f9590157470 2023-10-06T05:07:12.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.480+0000 7f5f4dd9b700 1 -- 172.21.15.98:0/3571581363 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f4800e070 con 0x7f5f500ccc40 2023-10-06T05:07:12.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.480+0000 7f5f4dd9b700 1 -- 172.21.15.98:0/3571581363 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f4800b9d0 con 0x7f5f500ccc40 2023-10-06T05:07:12.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.480+0000 7f5f4dd9b700 1 -- 172.21.15.98:0/3571581363 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f480092e0 con 0x7f5f500ccc40 2023-10-06T05:07:12.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.480+0000 7f1009769700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f10040696e0 0x7f1004069ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.480+0000 7f1009769700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f10040ada60 0x7f100406d350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.480+0000 7f1009769700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f100406d890 con 0x7f10040aeeb0 2023-10-06T05:07:12.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.480+0000 7f1009769700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10040cd000 con 0x7f10040ada60 2023-10-06T05:07:12.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.480+0000 7f1009769700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f100406e6d0 con 0x7f10040696e0 2023-10-06T05:07:12.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.480+0000 7f0b0e35f700 1 -- start start 2023-10-06T05:07:12.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.481+0000 7f1002ffd700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f10040696e0 0x7f1004069ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.481+0000 7f1002ffd700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f10040696e0 0x7f1004069ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.98:51210/0 (socket says 172.21.15.98:51210) 2023-10-06T05:07:12.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.481+0000 7f1002ffd700 1 -- 172.21.15.98:0/1309146975 learned_addr learned my addr 172.21.15.98:0/1309146975 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:12.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.481+0000 7f9594a5e700 1 -- 172.21.15.98:0/3343199314 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9590157470 msgr2=0x7f95900883c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.481+0000 7f9594a5e700 1 --2- 172.21.15.98:0/3343199314 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9590157470 0x7f95900883c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f95800099b0 tx=0x7f9580009250 comp rx=0 tx=0).stop 2023-10-06T05:07:12.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.481+0000 7f5f55126700 1 -- 172.21.15.98:0/3571581363 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5f500ccc40 msgr2=0x7f5f50069520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.481+0000 7f9594a5e700 1 -- 172.21.15.98:0/3343199314 shutdown_connections 2023-10-06T05:07:12.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.481+0000 7f9594a5e700 1 --2- 172.21.15.98:0/3343199314 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f95900ca5b0 0x7f9590157840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.481+0000 7f9594a5e700 1 --2- 172.21.15.98:0/3343199314 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9590157470 0x7f95900883c0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.497 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.481+0000 7f5f55126700 1 --2- 172.21.15.98:0/3571581363 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5f500ccc40 0x7f5f50069520 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f5f48009f20 tx=0x7f5f48009840 comp rx=0 tx=0).stop 2023-10-06T05:07:12.497 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.481+0000 7f9594a5e700 1 -- 172.21.15.98:0/3343199314 >> 172.21.15.98:0/3343199314 conn(0x7f9590087fc0 msgr2=0x7f95900ce260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:12.497 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.481+0000 7f9594a5e700 1 -- 172.21.15.98:0/3343199314 shutdown_connections 2023-10-06T05:07:12.497 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.481+0000 7f9594a5e700 1 -- 172.21.15.98:0/3343199314 wait complete. 2023-10-06T05:07:12.498 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.481+0000 7f5f55126700 1 -- 172.21.15.98:0/3571581363 shutdown_connections 2023-10-06T05:07:12.498 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.481+0000 7f5f55126700 1 --2- 172.21.15.98:0/3571581363 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5f500ccc40 0x7f5f50069520 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.498 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.481+0000 7f5f55126700 1 -- 172.21.15.98:0/3571581363 >> 172.21.15.98:0/3571581363 conn(0x7f5f50068080 msgr2=0x7f5f500cc0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:12.498 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.482+0000 7f5f55126700 1 -- 172.21.15.98:0/3571581363 shutdown_connections 2023-10-06T05:07:12.498 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.482+0000 7f5f55126700 1 -- 172.21.15.98:0/3571581363 wait complete. 2023-10-06T05:07:12.498 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.482+0000 7f745b16a700 1 Processor -- start 2023-10-06T05:07:12.499 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.482+0000 7f0b0e35f700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0b08161b70 0x7f0b08168890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.499 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.482+0000 7f0b0e35f700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b0806bfa0 con 0x7f0b08067800 2023-10-06T05:07:12.499 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.482+0000 7f0b0e35f700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b08162040 con 0x7f0b080ca9f0 2023-10-06T05:07:12.499 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.482+0000 7f0b0e35f700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b08162320 con 0x7f0b08161b70 2023-10-06T05:07:12.499 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.482+0000 7f0b07fff700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f0b08067800 0x7f0b080c7a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:41160/0 (socket says 172.21.15.98:41160) 2023-10-06T05:07:12.499 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.482+0000 7f0b07fff700 1 -- 172.21.15.98:0/1851221450 learned_addr learned my addr 172.21.15.98:0/1851221450 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:07:12.500 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.483+0000 7f9594a5e700 1 Processor -- start 2023-10-06T05:07:12.500 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.482+0000 7f745b16a700 1 -- start start 2023-10-06T05:07:12.500 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.482+0000 7f0b0c8fc700 1 --2- 172.21.15.98:0/1851221450 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0b08161b70 0x7f0b08168890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.500 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.482+0000 7f0b06ffd700 1 -- 172.21.15.98:0/1851221450 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2367593967 0 0) 0x7f0b0806bfa0 con 0x7f0b08067800 2023-10-06T05:07:12.500 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.483+0000 7f0b06ffd700 1 -- 172.21.15.98:0/1851221450 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0af4003570 con 0x7f0b08067800 2023-10-06T05:07:12.500 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.483+0000 7f0b0c8fc700 1 -- 172.21.15.98:0/1851221450 >> v1:172.21.15.174:6789/0 conn(0x7f0b080ca9f0 legacy=0x7f0b0806c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.483+0000 7f0b0c8fc700 1 -- 172.21.15.98:0/1851221450 >> v1:172.21.15.98:6789/0 conn(0x7f0b08067800 legacy=0x7f0b080c7a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.483+0000 7f0b0c8fc700 1 -- 172.21.15.98:0/1851221450 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b08162600 con 0x7f0b08161b70 2023-10-06T05:07:12.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.483+0000 7f0b0c8fc700 1 --2- 172.21.15.98:0/1851221450 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0b08161b70 0x7f0b08168890 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f0af0009910 tx=0x7f0af00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d0d2c14835e910f server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:12.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.483+0000 7f0b06ffd700 1 -- 172.21.15.98:0/1851221450 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0af000b680 con 0x7f0b08161b70 2023-10-06T05:07:12.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.483+0000 7f0b06ffd700 1 -- 172.21.15.98:0/1851221450 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0af000b7e0 con 0x7f0b08161b70 2023-10-06T05:07:12.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.483+0000 7f0b06ffd700 1 -- 172.21.15.98:0/1851221450 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0af000bba0 con 0x7f0b08161b70 2023-10-06T05:07:12.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.483+0000 7f61cbde3700 1 Processor -- start 2023-10-06T05:07:12.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.481+0000 7f1002ffd700 1 -- 172.21.15.98:0/1309146975 >> v2:172.21.15.98:3300/0 conn(0x7f10040ada60 msgr2=0x7f100406d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:12.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.481+0000 7f1002ffd700 1 --2- 172.21.15.98:0/1309146975 >> v2:172.21.15.98:3300/0 conn(0x7f10040ada60 0x7f100406d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.481+0000 7f1002ffd700 1 -- 172.21.15.98:0/1309146975 >> v1:172.21.15.98:6789/0 conn(0x7f10040aeeb0 legacy=0x7f100406eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:12.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.481+0000 7f1002ffd700 1 -- 172.21.15.98:0/1309146975 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f100406dac0 con 0x7f10040696e0 2023-10-06T05:07:12.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.481+0000 7f1002ffd700 1 --2- 172.21.15.98:0/1309146975 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f10040696e0 0x7f1004069ab0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1004070150 tx=0x7f0ff4009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=39ae1051beab6c2b server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:12.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.482+0000 7f1001ffb700 1 -- 172.21.15.98:0/1309146975 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ff400b820 con 0x7f10040696e0 2023-10-06T05:07:12.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.482+0000 7f1001ffb700 1 -- 172.21.15.98:0/1309146975 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ff400b980 con 0x7f10040696e0 2023-10-06T05:07:12.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.482+0000 7f1001ffb700 1 -- 172.21.15.98:0/1309146975 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ff400bd20 con 0x7f10040696e0 2023-10-06T05:07:12.503 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.483+0000 7f9594a5e700 1 -- start start 2023-10-06T05:07:12.503 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.483+0000 7f745b16a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74540c82d0 0x7f74540c2730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.503 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.483+0000 7f745b16a700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f74540c8750 0x7f74540c0530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.503 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.483+0000 7f745b16a700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7454160df0 0x7f74541591a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.504 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.483+0000 7f745b16a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74540c0a70 con 0x7f74540c82d0 2023-10-06T05:07:12.504 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.483+0000 7f745b16a700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74540c0cd0 con 0x7f7454160df0 2023-10-06T05:07:12.504 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.483+0000 7f745b16a700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f745415dda0 con 0x7f74540c8750 2023-10-06T05:07:12.504 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.483+0000 7f1009769700 1 -- 172.21.15.98:0/1309146975 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f10040696e0 msgr2=0x7f1004069ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.504 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.483+0000 7f1009769700 1 --2- 172.21.15.98:0/1309146975 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f10040696e0 0x7f1004069ab0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1004070150 tx=0x7f0ff4009250 comp rx=0 tx=0).stop 2023-10-06T05:07:12.504 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.483+0000 7f1009769700 1 -- 172.21.15.98:0/1309146975 shutdown_connections 2023-10-06T05:07:12.504 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.483+0000 7f1009769700 1 --2- 172.21.15.98:0/1309146975 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f10040696e0 0x7f1004069ab0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.505 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.483+0000 7f1009769700 1 --2- 172.21.15.98:0/1309146975 >> v2:172.21.15.98:3300/0 conn(0x7f10040ada60 0x7f100406d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.505 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.483+0000 7f1009769700 1 -- 172.21.15.98:0/1309146975 >> 172.21.15.98:0/1309146975 conn(0x7f1004068080 msgr2=0x7f10040cc0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:12.505 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.484+0000 7ff3e24c1700 1 Processor -- start 2023-10-06T05:07:12.505 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.484+0000 7ff3e24c1700 1 -- start start 2023-10-06T05:07:12.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.483+0000 7f5f55126700 1 Processor -- start 2023-10-06T05:07:12.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.484+0000 7f7459707700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7454160df0 0x7f74541591a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.484+0000 7f7459707700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7454160df0 0x7f74541591a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.98:51224/0 (socket says 172.21.15.98:51224) 2023-10-06T05:07:12.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.484+0000 7f7459707700 1 -- 172.21.15.98:0/2328044221 learned_addr learned my addr 172.21.15.98:0/2328044221 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:12.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.484+0000 7f0b0e35f700 1 -- 172.21.15.98:0/1851221450 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0b08161b70 msgr2=0x7f0b08168890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.484+0000 7f0b0e35f700 1 --2- 172.21.15.98:0/1851221450 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0b08161b70 0x7f0b08168890 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f0af0009910 tx=0x7f0af00091a0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.484+0000 7f9594a5e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95900ca140 0x7f95900b1f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.484+0000 7f9594a5e700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f95900ca5b0 0x7f959016aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.484+0000 7f9594a5e700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9590157470 0x7f959014adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.484+0000 7f9594a5e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95900b24a0 con 0x7f95900ca140 2023-10-06T05:07:12.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.484+0000 7f9594a5e700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95900c6b40 con 0x7f95900ca5b0 2023-10-06T05:07:12.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.484+0000 7f9594a5e700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f959016b530 con 0x7f9590157470 2023-10-06T05:07:12.508 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.484+0000 7f958dd9b700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f95900ca5b0 0x7f959016aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.508 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.484+0000 7f958dd9b700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f95900ca5b0 0x7f959016aff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.98:51232/0 (socket says 172.21.15.98:51232) 2023-10-06T05:07:12.508 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.484+0000 7f958dd9b700 1 -- 172.21.15.98:0/4288607882 learned_addr learned my addr 172.21.15.98:0/4288607882 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:12.508 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.484+0000 7f7459707700 1 -- 172.21.15.98:0/2328044221 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f74540c8750 msgr2=0x7f74540c0530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.508 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.484+0000 7f7459707700 1 --2- 172.21.15.98:0/2328044221 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f74540c8750 0x7f74540c0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.508 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.484+0000 7f7459707700 1 -- 172.21.15.98:0/2328044221 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74540c82d0 msgr2=0x7f74540c2730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.508 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.484+0000 7f7459707700 1 --2- 172.21.15.98:0/2328044221 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f74540c82d0 0x7f74540c2730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.508 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.484+0000 7f7459707700 1 -- 172.21.15.98:0/2328044221 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74400094f0 con 0x7f7454160df0 2023-10-06T05:07:12.509 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.484+0000 7f0b0e35f700 1 -- 172.21.15.98:0/1851221450 shutdown_connections 2023-10-06T05:07:12.509 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.484+0000 7f0b0e35f700 1 --2- 172.21.15.98:0/1851221450 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0b08161b70 0x7f0b08168890 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.509 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.484+0000 7f0b0e35f700 1 -- 172.21.15.98:0/1851221450 >> 172.21.15.98:0/1851221450 conn(0x7f0b080cdeb0 msgr2=0x7f0b080c9e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:12.509 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.484+0000 7f7459707700 1 --2- 172.21.15.98:0/2328044221 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7454160df0 0x7f74541591a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f744c00ceb0 tx=0x7f744c007870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:12.509 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.483+0000 7f61cbde3700 1 -- start start 2023-10-06T05:07:12.510 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.484+0000 7f61cbde3700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f61c4069c30 0x7f61c40c8450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.510 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.484+0000 7f61cbde3700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f61c40bf6f0 0x7f61c40c6680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.510 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.484+0000 7f61cbde3700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61c40c7fc0 con 0x7f61c40bf320 2023-10-06T05:07:12.510 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.484+0000 7f61cbde3700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61c415e190 con 0x7f61c40bf6f0 2023-10-06T05:07:12.510 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.484+0000 7f61cbde3700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61c40c73b0 con 0x7f61c4069c30 2023-10-06T05:07:12.510 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.484+0000 7f958e59c700 1 --2- 172.21.15.98:0/4288607882 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95900ca140 0x7f95900b1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.510 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.485+0000 7f958ed9d700 1 --2- 172.21.15.98:0/4288607882 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9590157470 0x7f959014adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.511 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.485+0000 7f61ca380700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f61c40bf6f0 0x7f61c40c6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.511 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.485+0000 7f61c937e700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f61c4069c30 0x7f61c40c8450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.511 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.485+0000 7f61ca380700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f61c40bf6f0 0x7f61c40c6680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.98:51234/0 (socket says 172.21.15.98:51234) 2023-10-06T05:07:12.511 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.485+0000 7f61ca380700 1 -- 172.21.15.98:0/550547902 learned_addr learned my addr 172.21.15.98:0/550547902 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:12.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.485+0000 7f958dd9b700 1 -- 172.21.15.98:0/4288607882 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9590157470 msgr2=0x7f959014adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.485+0000 7f958dd9b700 1 --2- 172.21.15.98:0/4288607882 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9590157470 0x7f959014adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.484+0000 7f5f55126700 1 -- start start 2023-10-06T05:07:12.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.485+0000 7f5f55126700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5f500ac000 0x7f5f500c1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.485+0000 7f5f55126700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5f500cc7f0 0x7f5f500c2530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.485+0000 7f5f55126700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5f500ccc40 0x7f5f5016dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.485+0000 7f5f55126700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f500c5c40 con 0x7f5f500ac000 2023-10-06T05:07:12.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.485+0000 7f5f55126700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f500c0470 con 0x7f5f500cc7f0 2023-10-06T05:07:12.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.485+0000 7f5f55126700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f500c0750 con 0x7f5f500ccc40 2023-10-06T05:07:12.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.485+0000 7ff3e24c1700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ff3dc157730 0x7ff3dc06ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.485+0000 7ff3e24c1700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3dc15b5e0 con 0x7ff3dc157360 2023-10-06T05:07:12.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.485+0000 7ff3e24c1700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3dc15b8c0 con 0x7ff3dc15b150 2023-10-06T05:07:12.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.485+0000 7ff3e24c1700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3dc0b1c10 con 0x7ff3dc157730 2023-10-06T05:07:12.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.485+0000 7f958dd9b700 1 -- 172.21.15.98:0/4288607882 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95900ca140 msgr2=0x7f95900b1f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.485+0000 7f958dd9b700 1 --2- 172.21.15.98:0/4288607882 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95900ca140 0x7f95900b1f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.485+0000 7f958dd9b700 1 -- 172.21.15.98:0/4288607882 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95800094f0 con 0x7f95900ca5b0 2023-10-06T05:07:12.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.485+0000 7f958e59c700 1 --2- 172.21.15.98:0/4288607882 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95900ca140 0x7f95900b1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T05:07:12.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.485+0000 7f5f4ed9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5f500ac000 0x7f5f500c1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.485+0000 7f5f4e59c700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5f500cc7f0 0x7f5f500c2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.485+0000 7f5f4ed9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5f500ac000 0x7f5f500c1ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45742/0 (socket says 172.21.15.98:45742) 2023-10-06T05:07:12.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.485+0000 7f5f4ed9d700 1 -- 172.21.15.98:0/3777412 learned_addr learned my addr 172.21.15.98:0/3777412 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:12.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.485+0000 7f5f4e59c700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5f500cc7f0 0x7f5f500c2530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.98:51244/0 (socket says 172.21.15.98:51244) 2023-10-06T05:07:12.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.485+0000 7f7450ff9700 1 -- 172.21.15.98:0/2328044221 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f744c00ae40 con 0x7f7454160df0 2023-10-06T05:07:12.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.485+0000 7f745b16a700 1 -- 172.21.15.98:0/2328044221 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f745415e080 con 0x7f7454160df0 2023-10-06T05:07:12.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.485+0000 7f745b16a700 1 -- 172.21.15.98:0/2328044221 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74541596e0 con 0x7f7454160df0 2023-10-06T05:07:12.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.485+0000 7f0b0e35f700 1 -- 172.21.15.98:0/1851221450 shutdown_connections 2023-10-06T05:07:12.516 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.485+0000 7f61ca380700 1 -- 172.21.15.98:0/550547902 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f61c4069c30 msgr2=0x7f61c40c8450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:12.516 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.485+0000 7f61ca380700 1 --2- 172.21.15.98:0/550547902 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f61c4069c30 0x7f61c40c8450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.516 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.485+0000 7f61ca380700 1 -- 172.21.15.98:0/550547902 >> v1:172.21.15.174:6789/0 conn(0x7f61c40bf320 legacy=0x7f61c4069520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:12.516 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.485+0000 7f61ca380700 1 -- 172.21.15.98:0/550547902 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61c40c7690 con 0x7f61c40bf6f0 2023-10-06T05:07:12.516 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.485+0000 7ff3db7fe700 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7ff3dc15b150 0x7ff3dc0af8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.98:55496/0 (socket says 172.21.15.98:55496) 2023-10-06T05:07:12.516 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.485+0000 7ff3db7fe700 1 -- 172.21.15.98:0/189655836 learned_addr learned my addr 172.21.15.98:0/189655836 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:07:12.516 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.485+0000 7f5f4f59e700 1 --2- 172.21.15.98:0/3777412 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5f500ccc40 0x7f5f5016dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.517 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.485+0000 7f958dd9b700 1 --2- 172.21.15.98:0/4288607882 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f95900ca5b0 0x7f959016aff0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f958800cb40 tx=0x7f9588001770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:12.517 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.485+0000 7f7450ff9700 1 -- 172.21.15.98:0/2328044221 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f744c016430 con 0x7f7454160df0 2023-10-06T05:07:12.517 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.485+0000 7f7450ff9700 1 -- 172.21.15.98:0/2328044221 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f744c0167b0 con 0x7f7454160df0 2023-10-06T05:07:12.517 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.485+0000 7f5f4e59c700 1 -- 172.21.15.98:0/3777412 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5f500ccc40 msgr2=0x7f5f5016dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.485+0000 7f5f4e59c700 1 --2- 172.21.15.98:0/3777412 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5f500ccc40 0x7f5f5016dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.485+0000 7f5f4e59c700 1 -- 172.21.15.98:0/3777412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5f500ac000 msgr2=0x7f5f500c1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.485+0000 7f5f4e59c700 1 --2- 172.21.15.98:0/3777412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5f500ac000 0x7f5f500c1ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.485+0000 7f5f4e59c700 1 -- 172.21.15.98:0/3777412 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f48009ae0 con 0x7f5f500cc7f0 2023-10-06T05:07:12.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.485+0000 7ff3e0a5e700 1 --2- 172.21.15.98:0/189655836 >> v2:172.21.15.174:3300/0 conn(0x7ff3dc157730 0x7ff3dc06ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.486+0000 7f61ca380700 1 --2- 172.21.15.98:0/550547902 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61c40bf6f0 0x7f61c40c6680 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f61c00099b0 tx=0x7f61c0009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=516762f76f889c24 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:12.519 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.486+0000 7f957e7fc700 1 -- 172.21.15.98:0/4288607882 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f958800a7e0 con 0x7f95900ca5b0 2023-10-06T05:07:12.519 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.486+0000 7f957e7fc700 1 -- 172.21.15.98:0/4288607882 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f958800a940 con 0x7f95900ca5b0 2023-10-06T05:07:12.519 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.486+0000 7f957e7fc700 1 -- 172.21.15.98:0/4288607882 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f958800ad00 con 0x7f95900ca5b0 2023-10-06T05:07:12.519 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.486+0000 7f9594a5e700 1 -- 172.21.15.98:0/4288607882 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95900c67f0 con 0x7f95900ca5b0 2023-10-06T05:07:12.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.486+0000 7f5f4e59c700 1 --2- 172.21.15.98:0/3777412 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5f500cc7f0 0x7f5f500c2530 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f5f4400c590 tx=0x7f5f44007d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:12.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.486+0000 7f9594a5e700 1 -- 172.21.15.98:0/4288607882 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f959016bab0 con 0x7f95900ca5b0 2023-10-06T05:07:12.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.486+0000 7ff3e0a5e700 1 -- 172.21.15.98:0/189655836 >> v1:172.21.15.174:6789/0 conn(0x7ff3dc15b150 legacy=0x7ff3dc0af8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.486+0000 7ff3e0a5e700 1 -- 172.21.15.98:0/189655836 >> v1:172.21.15.98:6789/0 conn(0x7ff3dc157360 legacy=0x7ff3dc0b1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.486+0000 7ff3e0a5e700 1 -- 172.21.15.98:0/189655836 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3dc15bba0 con 0x7ff3dc157730 2023-10-06T05:07:12.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.486+0000 7ff3e0a5e700 1 --2- 172.21.15.98:0/189655836 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff3dc157730 0x7ff3dc06ad00 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff3d4009e60 tx=0x7ff3d4009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=73beaa4b064359a6 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:12.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.487+0000 7ff3daffd700 1 -- 172.21.15.98:0/189655836 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3d400e070 con 0x7ff3dc157730 2023-10-06T05:07:12.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.487+0000 7f1009769700 1 -- 172.21.15.98:0/1309146975 shutdown_connections 2023-10-06T05:07:12.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.487+0000 7f9594a5e700 1 -- 172.21.15.98:0/4288607882 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f959006d490 con 0x7f95900ca5b0 2023-10-06T05:07:12.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.487+0000 7f61c8b7d700 1 -- 172.21.15.98:0/550547902 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61c0009ea0 con 0x7f61c40bf6f0 2023-10-06T05:07:12.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.487+0000 7f61c8b7d700 1 -- 172.21.15.98:0/550547902 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61c000b820 con 0x7f61c40bf6f0 2023-10-06T05:07:12.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.487+0000 7f61c8b7d700 1 -- 172.21.15.98:0/550547902 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61c000bba0 con 0x7f61c40bf6f0 2023-10-06T05:07:12.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.487+0000 7f5f36ffd700 1 -- 172.21.15.98:0/3777412 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f44011590 con 0x7f5f500cc7f0 2023-10-06T05:07:12.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.487+0000 7f5f36ffd700 1 -- 172.21.15.98:0/3777412 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f440116f0 con 0x7f5f500cc7f0 2023-10-06T05:07:12.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.487+0000 7f5f36ffd700 1 -- 172.21.15.98:0/3777412 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f44011a90 con 0x7f5f500cc7f0 2023-10-06T05:07:12.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.487+0000 7f5f55126700 1 -- 172.21.15.98:0/3777412 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f500c0a30 con 0x7f5f500cc7f0 2023-10-06T05:07:12.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.488+0000 7f5f55126700 1 -- 172.21.15.98:0/3777412 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f500c0ff0 con 0x7f5f500cc7f0 2023-10-06T05:07:12.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.488+0000 7f61cbde3700 1 -- 172.21.15.98:0/550547902 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61c40bf6f0 msgr2=0x7f61c40c6680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.488+0000 7f61cbde3700 1 --2- 172.21.15.98:0/550547902 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61c40bf6f0 0x7f61c40c6680 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f61c00099b0 tx=0x7f61c0009250 comp rx=0 tx=0).stop 2023-10-06T05:07:12.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.488+0000 7f61cbde3700 1 -- 172.21.15.98:0/550547902 shutdown_connections 2023-10-06T05:07:12.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.488+0000 7f61cbde3700 1 --2- 172.21.15.98:0/550547902 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61c40bf6f0 0x7f61c40c6680 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.488+0000 7f61cbde3700 1 --2- 172.21.15.98:0/550547902 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f61c4069c30 0x7f61c40c8450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.488+0000 7f61cbde3700 1 -- 172.21.15.98:0/550547902 >> 172.21.15.98:0/550547902 conn(0x7f61c4068080 msgr2=0x7f61c40cc0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:12.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.488+0000 7f957e7fc700 1 -- 172.21.15.98:0/4288607882 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f9588003030 con 0x7f95900ca5b0 2023-10-06T05:07:12.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.488+0000 7f957e7fc700 1 --2- 172.21.15.98:0/4288607882 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f9570066f30 0x7f95700693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.488+0000 7f7450ff9700 1 -- 172.21.15.98:0/2328044221 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f744c009030 con 0x7f7454160df0 2023-10-06T05:07:12.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.488+0000 7f7450ff9700 1 --2- 172.21.15.98:0/2328044221 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f7438066fb0 0x7f7438069460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.488+0000 7f7450ff9700 1 -- 172.21.15.98:0/2328044221 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f744c012070 con 0x7f7454160df0 2023-10-06T05:07:12.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.487+0000 7f1009769700 1 -- 172.21.15.98:0/1309146975 wait complete. 2023-10-06T05:07:12.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.489+0000 7f1009769700 1 Processor -- start 2023-10-06T05:07:12.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.489+0000 7f957e7fc700 1 -- 172.21.15.98:0/4288607882 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f9588085010 con 0x7f95900ca5b0 2023-10-06T05:07:12.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.489+0000 7f958e59c700 1 --2- 172.21.15.98:0/4288607882 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f9570066f30 0x7f95700693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.489+0000 7f7458f06700 1 --2- 172.21.15.98:0/2328044221 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f7438066fb0 0x7f7438069460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.489+0000 7f957e7fc700 1 --2- 172.21.15.98:0/4288607882 >> [v2:172.21.15.98:6800/90627,v1:172.21.15.98:6801/90627] conn(0x7f9570070c00 0x7f9570073030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.489+0000 7f957e7fc700 1 -- 172.21.15.98:0/4288607882 --> [v2:172.21.15.98:6800/90627,v1:172.21.15.98:6801/90627] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9570073700 con 0x7f9570070c00 2023-10-06T05:07:12.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.489+0000 7f957e7fc700 1 -- 172.21.15.98:0/4288607882 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f95880853e0 con 0x7f95900ca5b0 2023-10-06T05:07:12.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.486+0000 7f0b0e35f700 1 -- 172.21.15.98:0/1851221450 wait complete. 2023-10-06T05:07:12.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.488+0000 7ff3daffd700 1 -- 172.21.15.98:0/189655836 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3d400bc30 con 0x7ff3dc157730 2023-10-06T05:07:12.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.488+0000 7ff3daffd700 1 -- 172.21.15.98:0/189655836 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3d4012430 con 0x7ff3dc157730 2023-10-06T05:07:12.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.489+0000 7f958ed9d700 1 --2- 172.21.15.98:0/4288607882 >> [v2:172.21.15.98:6800/90627,v1:172.21.15.98:6801/90627] conn(0x7f9570070c00 0x7f9570073030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.489+0000 7ff3e24c1700 1 -- 172.21.15.98:0/189655836 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff3dc157730 msgr2=0x7ff3dc06ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.489+0000 7ff3e24c1700 1 --2- 172.21.15.98:0/189655836 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff3dc157730 0x7ff3dc06ad00 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff3d4009e60 tx=0x7ff3d4009780 comp rx=0 tx=0).stop 2023-10-06T05:07:12.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.487+0000 7f0b0e35f700 1 Processor -- start 2023-10-06T05:07:12.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.488+0000 7f0b0e35f700 1 -- start start 2023-10-06T05:07:12.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.489+0000 7f1009769700 1 -- start start 2023-10-06T05:07:12.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.489+0000 7ff3e24c1700 1 -- 172.21.15.98:0/189655836 shutdown_connections 2023-10-06T05:07:12.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.489+0000 7ff3e24c1700 1 --2- 172.21.15.98:0/189655836 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff3dc157730 0x7ff3dc06ad00 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.489+0000 7ff3e24c1700 1 -- 172.21.15.98:0/189655836 >> 172.21.15.98:0/189655836 conn(0x7ff3dc0cc6f0 msgr2=0x7ff3dc0ccaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:12.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.489+0000 7f61cbde3700 1 -- 172.21.15.98:0/550547902 shutdown_connections 2023-10-06T05:07:12.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.489+0000 7ff3e24c1700 1 -- 172.21.15.98:0/189655836 shutdown_connections 2023-10-06T05:07:12.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.490+0000 7ff3e24c1700 1 -- 172.21.15.98:0/189655836 wait complete. 2023-10-06T05:07:12.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.490+0000 7f7458f06700 1 --2- 172.21.15.98:0/2328044221 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f7438066fb0 0x7f7438069460 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f745406e6f0 tx=0x7f7440016000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:12.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.489+0000 7f0b0e35f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b08067800 0x7f0b081436d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.489+0000 7f0b0e35f700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0b08161b70 0x7f0b08157030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.489+0000 7f0b0e35f700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0b08157890 0x7f0b080c6c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.489+0000 7f0b0e35f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b08158980 con 0x7f0b08067800 2023-10-06T05:07:12.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.489+0000 7f0b0e35f700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b08157d70 con 0x7f0b08157890 2023-10-06T05:07:12.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.489+0000 7f0b0e35f700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b08158050 con 0x7f0b08161b70 2023-10-06T05:07:12.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.490+0000 7f0b077fe700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0b08161b70 0x7f0b08157030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.490+0000 7f0b0c8fc700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0b08157890 0x7f0b080c6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.490+0000 7f0b077fe700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0b08161b70 0x7f0b08157030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.98:55878/0 (socket says 172.21.15.98:55878) 2023-10-06T05:07:12.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.490+0000 7f0b077fe700 1 -- 172.21.15.98:0/1439664353 learned_addr learned my addr 172.21.15.98:0/1439664353 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:12.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.490+0000 7f0b07fff700 1 --2- 172.21.15.98:0/1439664353 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b08067800 0x7f0b081436d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.490+0000 7f0b077fe700 1 -- 172.21.15.98:0/1439664353 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0b08157890 msgr2=0x7f0b080c6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.533 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.490+0000 7f0b077fe700 1 --2- 172.21.15.98:0/1439664353 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0b08157890 0x7f0b080c6c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.533 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.490+0000 7f0b077fe700 1 -- 172.21.15.98:0/1439664353 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b08067800 msgr2=0x7f0b081436d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.533 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.490+0000 7f0b077fe700 1 --2- 172.21.15.98:0/1439664353 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b08067800 0x7f0b081436d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.533 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.490+0000 7f0b077fe700 1 -- 172.21.15.98:0/1439664353 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0af0009420 con 0x7f0b08161b70 2023-10-06T05:07:12.533 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.490+0000 7f0b07fff700 1 --2- 172.21.15.98:0/1439664353 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0b08067800 0x7f0b081436d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:07:12.533 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.490+0000 7f5f36ffd700 1 -- 172.21.15.98:0/3777412 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f5f4401c460 con 0x7f5f500cc7f0 2023-10-06T05:07:12.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.489+0000 7f958e59c700 1 --2- 172.21.15.98:0/4288607882 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f9570066f30 0x7f95700693e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f95800099b0 tx=0x7f9580017000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:12.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.489+0000 7f958ed9d700 1 --2- 172.21.15.98:0/4288607882 >> [v2:172.21.15.98:6800/90627,v1:172.21.15.98:6801/90627] conn(0x7f9570070c00 0x7f9570073030 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:12.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.490+0000 7f0b077fe700 1 --2- 172.21.15.98:0/1439664353 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0b08161b70 0x7f0b08157030 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0b0000cb10 tx=0x7f0b00001750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:12.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.490+0000 7f1009769700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f10040696e0 0x7f1004154670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.490+0000 7f1009769700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10040ada60 0x7f10040c1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.490+0000 7f1009769700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f10040aeeb0 0x7f10040c2300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.490+0000 7f1009769700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f100416dad0 con 0x7f10040ada60 2023-10-06T05:07:12.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.490+0000 7f1009769700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1004176f00 con 0x7f10040696e0 2023-10-06T05:07:12.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.490+0000 7f1009769700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10041771e0 con 0x7f10040aeeb0 2023-10-06T05:07:12.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.490+0000 7f1002ffd700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f10040696e0 0x7f1004154670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.491+0000 7f1002ffd700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f10040696e0 0x7f1004154670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.98:51276/0 (socket says 172.21.15.98:51276) 2023-10-06T05:07:12.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.491+0000 7f1002ffd700 1 -- 172.21.15.98:0/96809018 learned_addr learned my addr 172.21.15.98:0/96809018 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:12.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.491+0000 7f0afffff700 1 -- 172.21.15.98:0/1439664353 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b00002060 con 0x7f0b08161b70 2023-10-06T05:07:12.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.491+0000 7f0afffff700 1 -- 172.21.15.98:0/1439664353 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b0000a7e0 con 0x7f0b08161b70 2023-10-06T05:07:12.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.491+0000 7f0afffff700 1 -- 172.21.15.98:0/1439664353 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b0000aba0 con 0x7f0b08161b70 2023-10-06T05:07:12.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.491+0000 7f5f36ffd700 1 --2- 172.21.15.98:0/3777412 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f5f3c067000 0x7f5f3c0694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.491+0000 7f5f36ffd700 1 -- 172.21.15.98:0/3777412 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f5f440845b0 con 0x7f5f500cc7f0 2023-10-06T05:07:12.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.491+0000 7f957e7fc700 1 -- 172.21.15.98:0/4288607882 <== osd.0 v2:172.21.15.98:6800/90627 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f9570073700 con 0x7f9570070c00 2023-10-06T05:07:12.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.491+0000 7f0b0e35f700 1 -- 172.21.15.98:0/1439664353 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b08158330 con 0x7f0b08161b70 2023-10-06T05:07:12.538 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.491+0000 7ff3e24c1700 1 Processor -- start 2023-10-06T05:07:12.538 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.491+0000 7f1002ffd700 1 -- 172.21.15.98:0/96809018 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f10040aeeb0 msgr2=0x7f10040c2300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-06T05:07:12.538 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.491+0000 7f1002ffd700 1 --2- 172.21.15.98:0/96809018 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f10040aeeb0 0x7f10040c2300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.538 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.491+0000 7f1002ffd700 1 -- 172.21.15.98:0/96809018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10040ada60 msgr2=0x7f10040c1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.538 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.491+0000 7f1002ffd700 1 --2- 172.21.15.98:0/96809018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f10040ada60 0x7f10040c1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.538 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.491+0000 7f1002ffd700 1 -- 172.21.15.98:0/96809018 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ff40094f0 con 0x7f10040696e0 2023-10-06T05:07:12.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.491+0000 7f5f4ed9d700 1 --2- 172.21.15.98:0/3777412 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f5f3c067000 0x7f5f3c0694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.491+0000 7f0b0e35f700 1 -- 172.21.15.98:0/1439664353 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b08157570 con 0x7f0b08161b70 2023-10-06T05:07:12.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.491+0000 7f1002ffd700 1 --2- 172.21.15.98:0/96809018 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f10040696e0 0x7f1004154670 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0ff4003080 tx=0x7f0ff4016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:12.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.491+0000 7f745b16a700 1 --2- 172.21.15.98:0/2328044221 >> [v2:172.21.15.98:6817/90663,v1:172.21.15.98:6819/90663] conn(0x7f74340015f0 0x7f7434003aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.491+0000 7f745b16a700 1 -- 172.21.15.98:0/2328044221 --> [v2:172.21.15.98:6817/90663,v1:172.21.15.98:6819/90663] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7434006b90 con 0x7f74340015f0 2023-10-06T05:07:12.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.492+0000 7f0feaffd700 1 -- 172.21.15.98:0/96809018 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ff4011e60 con 0x7f10040696e0 2023-10-06T05:07:12.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.492+0000 7f1009769700 1 -- 172.21.15.98:0/96809018 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10041774c0 con 0x7f10040696e0 2023-10-06T05:07:12.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.490+0000 7f61cbde3700 1 -- 172.21.15.98:0/550547902 wait complete. 2023-10-06T05:07:12.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.492+0000 7f61cbde3700 1 Processor -- start 2023-10-06T05:07:12.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.492+0000 7f1009769700 1 -- 172.21.15.98:0/96809018 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1004176b60 con 0x7f10040696e0 2023-10-06T05:07:12.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.492+0000 7f0feaffd700 1 -- 172.21.15.98:0/96809018 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ff401a670 con 0x7f10040696e0 2023-10-06T05:07:12.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.492+0000 7f0feaffd700 1 -- 172.21.15.98:0/96809018 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ff401aa10 con 0x7f10040696e0 2023-10-06T05:07:12.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.492+0000 7f5f55126700 1 --2- 172.21.15.98:0/3777412 >> [v2:172.21.15.98:6808/90628,v1:172.21.15.98:6809/90628] conn(0x7f5f50156250 0x7f5f5016e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.492+0000 7f5f55126700 1 -- 172.21.15.98:0/3777412 --> [v2:172.21.15.98:6808/90628,v1:172.21.15.98:6809/90628] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5f500c6ce0 con 0x7f5f50156250 2023-10-06T05:07:12.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.492+0000 7f7453fff700 1 --2- 172.21.15.98:0/2328044221 >> [v2:172.21.15.98:6817/90663,v1:172.21.15.98:6819/90663] conn(0x7f74340015f0 0x7f7434003aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.542 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.493+0000 7f5f4f59e700 1 --2- 172.21.15.98:0/3777412 >> [v2:172.21.15.98:6808/90628,v1:172.21.15.98:6809/90628] conn(0x7f5f50156250 0x7f5f5016e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.542 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.492+0000 7f1009769700 1 -- 172.21.15.98:0/96809018 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0fe4000fd0 con 0x7f10040696e0 2023-10-06T05:07:12.542 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.493+0000 7f5f4ed9d700 1 --2- 172.21.15.98:0/3777412 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f5f3c067000 0x7f5f3c0694b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5f400078a0 tx=0x7f5f40007830 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:12.543 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.491+0000 7ff3e24c1700 1 -- start start 2023-10-06T05:07:12.543 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.492+0000 7ff3e24c1700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff3dc157360 0x7ff3dc0c3f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.543 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.492+0000 7ff3e24c1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3dc157730 0x7ff3dc149040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.543 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.492+0000 7ff3e24c1700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff3dc15b150 0x7ff3dc149580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.543 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.492+0000 7ff3e24c1700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3dc14a960 con 0x7ff3dc157730 2023-10-06T05:07:12.543 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.492+0000 7ff3e24c1700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3dc149ac0 con 0x7ff3dc157360 2023-10-06T05:07:12.543 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.492+0000 7ff3e24c1700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3dc149d50 con 0x7ff3dc15b150 2023-10-06T05:07:12.544 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.492+0000 7f61cbde3700 1 -- start start 2023-10-06T05:07:12.544 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.493+0000 7ff3db7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3dc157730 0x7ff3dc149040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.544 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.493+0000 7ff3db7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3dc157730 0x7ff3dc149040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45784/0 (socket says 172.21.15.98:45784) 2023-10-06T05:07:12.544 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.493+0000 7ff3db7fe700 1 -- 172.21.15.98:0/3292772014 learned_addr learned my addr 172.21.15.98:0/3292772014 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:12.545 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.493+0000 7f61cbde3700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61c4069c30 0x7f61c40c4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.545 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.493+0000 7f61cbde3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61c40bf320 0x7f61c415ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.545 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.493+0000 7f61cbde3700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f61c40bf6f0 0x7f61c415cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.545 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.493+0000 7f61cbde3700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61c40c5100 con 0x7f61c40bf320 2023-10-06T05:07:12.545 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.493+0000 7f61cbde3700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61c40c5bc0 con 0x7f61c4069c30 2023-10-06T05:07:12.545 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.493+0000 7f61cbde3700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61c415b170 con 0x7f61c40bf6f0 2023-10-06T05:07:12.546 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.493+0000 7f61ca380700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f61c40bf6f0 0x7f61c415cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.546 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.493+0000 7f61c9b7f700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61c4069c30 0x7f61c40c4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.546 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.493+0000 7f61ca380700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f61c40bf6f0 0x7f61c415cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.98:55884/0 (socket says 172.21.15.98:55884) 2023-10-06T05:07:12.546 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.493+0000 7f61ca380700 1 -- 172.21.15.98:0/110896907 learned_addr learned my addr 172.21.15.98:0/110896907 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:12.546 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.493+0000 7f0b0e35f700 1 -- 172.21.15.98:0/1439664353 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0b0806f740 con 0x7f0b08161b70 2023-10-06T05:07:12.546 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.493+0000 7f61c937e700 1 --2- 172.21.15.98:0/110896907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61c40bf320 0x7f61c415ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.493+0000 7ff3e0a5e700 1 --2- 172.21.15.98:0/3292772014 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff3dc15b150 0x7ff3dc149580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.493+0000 7ff3db7fe700 1 -- 172.21.15.98:0/3292772014 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff3dc15b150 msgr2=0x7ff3dc149580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.493+0000 7ff3db7fe700 1 --2- 172.21.15.98:0/3292772014 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff3dc15b150 0x7ff3dc149580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.493+0000 7ff3db7fe700 1 -- 172.21.15.98:0/3292772014 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff3dc157360 msgr2=0x7ff3dc0c3f50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-06T05:07:12.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.493+0000 7ff3db7fe700 1 --2- 172.21.15.98:0/3292772014 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff3dc157360 0x7ff3dc0c3f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.493+0000 7ff3db7fe700 1 -- 172.21.15.98:0/3292772014 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3d4009a20 con 0x7ff3dc157730 2023-10-06T05:07:12.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.493+0000 7f7453fff700 1 --2- 172.21.15.98:0/2328044221 >> [v2:172.21.15.98:6817/90663,v1:172.21.15.98:6819/90663] conn(0x7f74340015f0 0x7f7434003aa0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:12.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.493+0000 7f61ca380700 1 -- 172.21.15.98:0/110896907 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61c4069c30 msgr2=0x7f61c40c4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.493+0000 7f61ca380700 1 --2- 172.21.15.98:0/110896907 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61c4069c30 0x7f61c40c4bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.493+0000 7f61ca380700 1 -- 172.21.15.98:0/110896907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61c40bf320 msgr2=0x7f61c415ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.493+0000 7f61ca380700 1 --2- 172.21.15.98:0/110896907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61c40bf320 0x7f61c415ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.494+0000 7f61ca380700 1 -- 172.21.15.98:0/110896907 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61c00094f0 con 0x7f61c40bf6f0 2023-10-06T05:07:12.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.494+0000 7f61ca380700 1 --2- 172.21.15.98:0/110896907 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f61c40bf6f0 0x7f61c415cec0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f61c406fd70 tx=0x7f61c0016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:12.549 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.494+0000 7ff3db7fe700 1 --2- 172.21.15.98:0/3292772014 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3dc157730 0x7ff3dc149040 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff3cc00c7c0 tx=0x7ff3cc007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:12.549 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.494+0000 7f5f4f59e700 1 --2- 172.21.15.98:0/3777412 >> [v2:172.21.15.98:6808/90628,v1:172.21.15.98:6809/90628] conn(0x7f5f50156250 0x7f5f5016e130 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:12.549 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.494+0000 7f0afffff700 1 -- 172.21.15.98:0/1439664353 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f0b00003030 con 0x7f0b08161b70 2023-10-06T05:07:12.549 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.494+0000 7f0afffff700 1 --2- 172.21.15.98:0/1439664353 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f0ae8067080 0x7f0ae8069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.494+0000 7f0afffff700 1 -- 172.21.15.98:0/1439664353 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f0b00084970 con 0x7f0b08161b70 2023-10-06T05:07:12.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.494+0000 7f0b07fff700 1 --2- 172.21.15.98:0/1439664353 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f0ae8067080 0x7f0ae8069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.494+0000 7f0afffff700 1 --2- 172.21.15.98:0/1439664353 >> [v2:172.21.15.174:6800/90847,v1:172.21.15.174:6801/90847] conn(0x7f0ae8070d50 0x7f0ae8073180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.494+0000 7f0afffff700 1 -- 172.21.15.98:0/1439664353 --> [v2:172.21.15.174:6800/90847,v1:172.21.15.174:6801/90847] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0ae8073850 con 0x7f0ae8070d50 2023-10-06T05:07:12.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.494+0000 7f0afffff700 1 -- 172.21.15.98:0/1439664353 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0b0008a1d0 con 0x7f0b08161b70 2023-10-06T05:07:12.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.494+0000 7f0feaffd700 1 -- 172.21.15.98:0/96809018 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f0ff401b440 con 0x7f10040696e0 2023-10-06T05:07:12.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.494+0000 7f0feaffd700 1 --2- 172.21.15.98:0/96809018 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f0ff0067000 0x7f0ff00694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.551 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.495+0000 7f7450ff9700 1 -- 172.21.15.98:0/2328044221 <== osd.3 v2:172.21.15.98:6817/90663 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f7434006b90 con 0x7f74340015f0 2023-10-06T05:07:12.551 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.495+0000 7f0b0c8fc700 1 --2- 172.21.15.98:0/1439664353 >> [v2:172.21.15.174:6800/90847,v1:172.21.15.174:6801/90847] conn(0x7f0ae8070d50 0x7f0ae8073180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.551 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.494+0000 7f10027fc700 1 --2- 172.21.15.98:0/96809018 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f0ff0067000 0x7f0ff00694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.552 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.495+0000 7f0feaffd700 1 -- 172.21.15.98:0/96809018 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f0ff4086410 con 0x7f10040696e0 2023-10-06T05:07:12.552 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.495+0000 7f0feaffd700 1 --2- 172.21.15.98:0/96809018 >> [v2:172.21.15.174:6824/90878,v1:172.21.15.174:6825/90878] conn(0x7f0ff0070c50 0x7f0ff0073080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.552 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.495+0000 7f0feaffd700 1 -- 172.21.15.98:0/96809018 --> [v2:172.21.15.174:6824/90878,v1:172.21.15.174:6825/90878] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0ff0073750 con 0x7f0ff0070c50 2023-10-06T05:07:12.552 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.495+0000 7f0b07fff700 1 --2- 172.21.15.98:0/1439664353 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f0ae8067080 0x7f0ae8069530 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0b080c9b90 tx=0x7f0af8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:12.552 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.495+0000 7f10027fc700 1 --2- 172.21.15.98:0/96809018 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f0ff0067000 0x7f0ff00694b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f100406a670 tx=0x7f0ffc009370 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:12.553 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.495+0000 7f0b0c8fc700 1 --2- 172.21.15.98:0/1439664353 >> [v2:172.21.15.174:6800/90847,v1:172.21.15.174:6801/90847] conn(0x7f0ae8070d50 0x7f0ae8073180 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:12.553 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.495+0000 7f5f36ffd700 1 -- 172.21.15.98:0/3777412 <== osd.1 v2:172.21.15.98:6808/90628 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f5f500c6ce0 con 0x7f5f50156250 2023-10-06T05:07:12.553 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.494+0000 7ff3c3fff700 1 -- 172.21.15.98:0/3292772014 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3cc005b30 con 0x7ff3dc157730 2023-10-06T05:07:12.554 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.494+0000 7ff3c3fff700 1 -- 172.21.15.98:0/3292772014 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3cc005cb0 con 0x7ff3dc157730 2023-10-06T05:07:12.554 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.494+0000 7ff3c3fff700 1 -- 172.21.15.98:0/3292772014 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3cc004a80 con 0x7ff3dc157730 2023-10-06T05:07:12.554 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.494+0000 7ff3e24c1700 1 -- 172.21.15.98:0/3292772014 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3dc14a030 con 0x7ff3dc157730 2023-10-06T05:07:12.554 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.494+0000 7ff3e24c1700 1 -- 172.21.15.98:0/3292772014 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3dc14a5f0 con 0x7ff3dc157730 2023-10-06T05:07:12.554 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.495+0000 7ff3e24c1700 1 -- 172.21.15.98:0/3292772014 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff3dc064a90 con 0x7ff3dc157730 2023-10-06T05:07:12.554 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.496+0000 7f10037fe700 1 --2- 172.21.15.98:0/96809018 >> [v2:172.21.15.174:6824/90878,v1:172.21.15.174:6825/90878] conn(0x7f0ff0070c50 0x7f0ff0073080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.555 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.496+0000 7f0feaffd700 1 -- 172.21.15.98:0/96809018 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0ff40532b0 con 0x7f10040696e0 2023-10-06T05:07:12.555 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.496+0000 7f0afffff700 1 -- 172.21.15.98:0/1439664353 <== osd.4 v2:172.21.15.174:6800/90847 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f0ae8073850 con 0x7f0ae8070d50 2023-10-06T05:07:12.555 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.496+0000 7f10037fe700 1 --2- 172.21.15.98:0/96809018 >> [v2:172.21.15.174:6824/90878,v1:172.21.15.174:6825/90878] conn(0x7f0ff0070c50 0x7f0ff0073080 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:12.556 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.496+0000 7f61b9ffb700 1 -- 172.21.15.98:0/110896907 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61c000bba0 con 0x7f61c40bf6f0 2023-10-06T05:07:12.556 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.496+0000 7f61b9ffb700 1 -- 172.21.15.98:0/110896907 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61c001c450 con 0x7f61c40bf6f0 2023-10-06T05:07:12.556 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.496+0000 7f61b9ffb700 1 -- 172.21.15.98:0/110896907 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61c001c7f0 con 0x7f61c40bf6f0 2023-10-06T05:07:12.556 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.496+0000 7f61cbde3700 1 -- 172.21.15.98:0/110896907 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61c415ef50 con 0x7f61c40bf6f0 2023-10-06T05:07:12.556 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.496+0000 7f61cbde3700 1 -- 172.21.15.98:0/110896907 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61c40c5600 con 0x7f61c40bf6f0 2023-10-06T05:07:12.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.497+0000 7ff3c3fff700 1 -- 172.21.15.98:0/3292772014 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7ff3cc0070f0 con 0x7ff3dc157730 2023-10-06T05:07:12.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.497+0000 7ff3c3fff700 1 --2- 172.21.15.98:0/3292772014 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7ff3c8067000 0x7ff3c80694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.497+0000 7f61cbde3700 1 -- 172.21.15.98:0/110896907 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f61a4000fd0 con 0x7f61c40bf6f0 2023-10-06T05:07:12.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.497+0000 7ff3c3fff700 1 -- 172.21.15.98:0/3292772014 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7ff3cc084500 con 0x7ff3dc157730 2023-10-06T05:07:12.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.498+0000 7ff3c3fff700 1 --2- 172.21.15.98:0/3292772014 >> [v2:172.21.15.174:6808/90848,v1:172.21.15.174:6809/90848] conn(0x7ff3c8070cd0 0x7ff3c8073100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.498+0000 7ff3c3fff700 1 -- 172.21.15.98:0/3292772014 --> [v2:172.21.15.174:6808/90848,v1:172.21.15.174:6809/90848] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff3c80737d0 con 0x7ff3c8070cd0 2023-10-06T05:07:12.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.498+0000 7ff3c3fff700 1 -- 172.21.15.98:0/3292772014 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff3cc084920 con 0x7ff3dc157730 2023-10-06T05:07:12.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.498+0000 7ff3dbfff700 1 --2- 172.21.15.98:0/3292772014 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7ff3c8067000 0x7ff3c80694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.498+0000 7ff3e0a5e700 1 --2- 172.21.15.98:0/3292772014 >> [v2:172.21.15.174:6808/90848,v1:172.21.15.174:6809/90848] conn(0x7ff3c8070cd0 0x7ff3c8073100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.498+0000 7ff3dbfff700 1 --2- 172.21.15.98:0/3292772014 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7ff3c8067000 0x7ff3c80694b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff3d0000f60 tx=0x7ff3d0007b40 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:12.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.498+0000 7f0feaffd700 1 -- 172.21.15.98:0/96809018 <== osd.7 v2:172.21.15.174:6824/90878 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f0ff0073750 con 0x7f0ff0070c50 2023-10-06T05:07:12.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.498+0000 7f61b9ffb700 1 -- 172.21.15.98:0/110896907 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f61c001cb60 con 0x7f61c40bf6f0 2023-10-06T05:07:12.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.499+0000 7ff3e0a5e700 1 --2- 172.21.15.98:0/3292772014 >> [v2:172.21.15.174:6808/90848,v1:172.21.15.174:6809/90848] conn(0x7ff3c8070cd0 0x7ff3c8073100 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:12.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.499+0000 7f61b9ffb700 1 --2- 172.21.15.98:0/110896907 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f61a8067080 0x7f61a8069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.499+0000 7f3784362700 1 Processor -- start 2023-10-06T05:07:12.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.499+0000 7f3784362700 1 -- start start 2023-10-06T05:07:12.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.499+0000 7f61b9ffb700 1 -- 172.21.15.98:0/110896907 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f61c0086430 con 0x7f61c40bf6f0 2023-10-06T05:07:12.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.499+0000 7f61b9ffb700 1 --2- 172.21.15.98:0/110896907 >> [v2:172.21.15.174:6816/90877,v1:172.21.15.174:6817/90877] conn(0x7f61a8070d50 0x7f61a8073180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.499+0000 7f61b9ffb700 1 -- 172.21.15.98:0/110896907 --> [v2:172.21.15.174:6816/90877,v1:172.21.15.174:6817/90877] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f61a8073850 con 0x7f61a8070d50 2023-10-06T05:07:12.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.499+0000 7f61b9ffb700 1 -- 172.21.15.98:0/110896907 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f61c00889d0 con 0x7f61c40bf6f0 2023-10-06T05:07:12.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.499+0000 7f61c9b7f700 1 --2- 172.21.15.98:0/110896907 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f61a8067080 0x7f61a8069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.500+0000 7ff3c3fff700 1 -- 172.21.15.98:0/3292772014 <== osd.5 v2:172.21.15.174:6808/90848 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7ff3c80737d0 con 0x7ff3c8070cd0 2023-10-06T05:07:12.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.499+0000 7f61c937e700 1 --2- 172.21.15.98:0/110896907 >> [v2:172.21.15.174:6816/90877,v1:172.21.15.174:6817/90877] conn(0x7f61a8070d50 0x7f61a8073180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.500+0000 7f3784362700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f377c067800 0x7f377c06d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.500+0000 7f3784362700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f377c067bd0 0x7f377c06b280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.500+0000 7f3784362700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f377c0bf4b0 0x7f377c0bf8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.501+0000 7f3784362700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f377c06b7c0 con 0x7f377c067bd0 2023-10-06T05:07:12.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.501+0000 7f3784362700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f377c0af4c0 con 0x7f377c067800 2023-10-06T05:07:12.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.501+0000 7f3784362700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f377c06dfa0 con 0x7f377c0bf4b0 2023-10-06T05:07:12.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.501+0000 7f37820fe700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f377c067800 0x7f377c06d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.501+0000 7f37820fe700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f377c067800 0x7f377c06d910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.98:51288/0 (socket says 172.21.15.98:51288) 2023-10-06T05:07:12.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.501+0000 7f37820fe700 1 -- 172.21.15.98:0/1313153064 learned_addr learned my addr 172.21.15.98:0/1313153064 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:12.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.501+0000 7f37820fe700 1 -- 172.21.15.98:0/1313153064 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f377c0bf4b0 msgr2=0x7f377c0bf8e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-06T05:07:12.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.501+0000 7f37820fe700 1 --2- 172.21.15.98:0/1313153064 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f377c0bf4b0 0x7f377c0bf8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.501+0000 7f37820fe700 1 -- 172.21.15.98:0/1313153064 >> v2:172.21.15.98:3300/0 conn(0x7f377c067bd0 msgr2=0x7f377c06b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:12.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.501+0000 7f37820fe700 1 --2- 172.21.15.98:0/1313153064 >> v2:172.21.15.98:3300/0 conn(0x7f377c067bd0 0x7f377c06b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.501+0000 7f37820fe700 1 -- 172.21.15.98:0/1313153064 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f377c06e280 con 0x7f377c067800 2023-10-06T05:07:12.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.501+0000 7f37820fe700 1 --2- 172.21.15.98:0/1313153064 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f377c067800 0x7f377c06d910 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f377c070370 tx=0x7f3774009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d2cedc97cab0f4aa server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:12.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.502+0000 7f37810fc700 1 -- 172.21.15.98:0/1313153064 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f377400e070 con 0x7f377c067800 2023-10-06T05:07:12.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.502+0000 7f37810fc700 1 -- 172.21.15.98:0/1313153064 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f377400ba50 con 0x7f377c067800 2023-10-06T05:07:12.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.502+0000 7f37810fc700 1 -- 172.21.15.98:0/1313153064 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37740092e0 con 0x7f377c067800 2023-10-06T05:07:12.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.503+0000 7f3784362700 1 -- 172.21.15.98:0/1313153064 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f377c067800 msgr2=0x7f377c06d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.503+0000 7f3784362700 1 --2- 172.21.15.98:0/1313153064 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f377c067800 0x7f377c06d910 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f377c070370 tx=0x7f3774009840 comp rx=0 tx=0).stop 2023-10-06T05:07:12.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.504+0000 7f3784362700 1 -- 172.21.15.98:0/1313153064 shutdown_connections 2023-10-06T05:07:12.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.504+0000 7f3784362700 1 --2- 172.21.15.98:0/1313153064 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f377c0bf4b0 0x7f377c0bf8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.504+0000 7f3784362700 1 --2- 172.21.15.98:0/1313153064 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f377c067800 0x7f377c06d910 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.504+0000 7f3784362700 1 --2- 172.21.15.98:0/1313153064 >> v2:172.21.15.98:3300/0 conn(0x7f377c067bd0 0x7f377c06b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.504+0000 7f3784362700 1 -- 172.21.15.98:0/1313153064 >> 172.21.15.98:0/1313153064 conn(0x7f377c0cc6f0 msgr2=0x7f377c0ccaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:12.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.504+0000 7f3784362700 1 -- 172.21.15.98:0/1313153064 shutdown_connections 2023-10-06T05:07:12.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.504+0000 7f3784362700 1 -- 172.21.15.98:0/1313153064 wait complete. 2023-10-06T05:07:12.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.506+0000 7f3784362700 1 Processor -- start 2023-10-06T05:07:12.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.506+0000 7f3784362700 1 -- start start 2023-10-06T05:07:12.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.507+0000 7f3784362700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f377c067800 0x7f377c160d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.507+0000 7f3784362700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f377c067bd0 0x7f377c0c0b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.507+0000 7f3784362700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f377c0bf4b0 0x7f377c0ca9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.507+0000 7f3784362700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f377c0c2e00 con 0x7f377c067800 2023-10-06T05:07:12.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.507+0000 7f3784362700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f377c161250 con 0x7f377c067bd0 2023-10-06T05:07:12.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.507+0000 7f3784362700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f377c0c47e0 con 0x7f377c0bf4b0 2023-10-06T05:07:12.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.507+0000 7f37820fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f377c067800 0x7f377c160d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.507+0000 7f37820fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f377c067800 0x7f377c160d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45806/0 (socket says 172.21.15.98:45806) 2023-10-06T05:07:12.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.507+0000 7f37820fe700 1 -- 172.21.15.98:0/1558810778 learned_addr learned my addr 172.21.15.98:0/1558810778 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:12.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.508+0000 7f37818fd700 1 --2- 172.21.15.98:0/1558810778 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f377c067bd0 0x7f377c0c0b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.508+0000 7f37828ff700 1 --2- 172.21.15.98:0/1558810778 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f377c0bf4b0 0x7f377c0ca9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.508+0000 7f37820fe700 1 -- 172.21.15.98:0/1558810778 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f377c0bf4b0 msgr2=0x7f377c0ca9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.508+0000 7f37820fe700 1 --2- 172.21.15.98:0/1558810778 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f377c0bf4b0 0x7f377c0ca9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.508+0000 7f37820fe700 1 -- 172.21.15.98:0/1558810778 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f377c067bd0 msgr2=0x7f377c0c0b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.508+0000 7f37820fe700 1 --2- 172.21.15.98:0/1558810778 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f377c067bd0 0x7f377c0c0b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.508+0000 7f37820fe700 1 -- 172.21.15.98:0/1558810778 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3774009ae0 con 0x7f377c067800 2023-10-06T05:07:12.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.508+0000 7f37820fe700 1 --2- 172.21.15.98:0/1558810778 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f377c067800 0x7f377c160d10 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f377c0af600 tx=0x7f3774018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:12.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.508+0000 7f37727fc700 1 -- 172.21.15.98:0/1558810778 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3774008b90 con 0x7f377c067800 2023-10-06T05:07:12.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.508+0000 7f37727fc700 1 -- 172.21.15.98:0/1558810778 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3774008d10 con 0x7f377c067800 2023-10-06T05:07:12.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.508+0000 7f37727fc700 1 -- 172.21.15.98:0/1558810778 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37740266b0 con 0x7f377c067800 2023-10-06T05:07:12.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.509+0000 7f3784362700 1 -- 172.21.15.98:0/1558810778 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f377c0c4ac0 con 0x7f377c067800 2023-10-06T05:07:12.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.509+0000 7f3784362700 1 -- 172.21.15.98:0/1558810778 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f377c0c5080 con 0x7f377c067800 2023-10-06T05:07:12.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.500+0000 7f61c937e700 1 --2- 172.21.15.98:0/110896907 >> [v2:172.21.15.174:6816/90877,v1:172.21.15.174:6817/90877] conn(0x7f61a8070d50 0x7f61a8073180 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:12.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.511+0000 7f61c9b7f700 1 --2- 172.21.15.98:0/110896907 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f61a8067080 0x7f61a8069530 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f61b4006fd0 tx=0x7f61b40090f0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:12.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.511+0000 7f61b9ffb700 1 -- 172.21.15.98:0/110896907 <== osd.6 v2:172.21.15.174:6816/90877 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f61a8073850 con 0x7f61a8070d50 2023-10-06T05:07:12.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.517+0000 7f1009769700 1 -- 172.21.15.98:0/96809018 --> [v2:172.21.15.174:6824/90878,v1:172.21.15.174:6825/90878] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0fe4002d30 con 0x7f0ff0070c50 2023-10-06T05:07:12.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.518+0000 7f0feaffd700 1 -- 172.21.15.98:0/96809018 <== osd.7 v2:172.21.15.174:6824/90878 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f0fe4002d30 con 0x7f0ff0070c50 2023-10-06T05:07:12.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.511+0000 7f37727fc700 1 -- 172.21.15.98:0/1558810778 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f3774003040 con 0x7f377c067800 2023-10-06T05:07:12.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.512+0000 7f37818fd700 1 --2- 172.21.15.98:0/1558810778 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f377c067bd0 0x7f377c0c0b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:07:12.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.512+0000 7f37727fc700 1 --2- 172.21.15.98:0/1558810778 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f3760066fb0 0x7f3760069460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.512+0000 7f37727fc700 1 -- 172.21.15.98:0/1558810778 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f3774086330 con 0x7f377c067800 2023-10-06T05:07:12.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.512+0000 7f37818fd700 1 --2- 172.21.15.98:0/1558810778 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f3760066fb0 0x7f3760069460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.513+0000 7f3784362700 1 --2- 172.21.15.98:0/1558810778 >> [v2:172.21.15.98:6816/90662,v1:172.21.15.98:6818/90662] conn(0x7f377c176c60 0x7f377c0caf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.513+0000 7f3784362700 1 -- 172.21.15.98:0/1558810778 --> [v2:172.21.15.98:6816/90662,v1:172.21.15.98:6818/90662] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f377c06e560 con 0x7f377c176c60 2023-10-06T05:07:12.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.513+0000 7f37818fd700 1 --2- 172.21.15.98:0/1558810778 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f3760066fb0 0x7f3760069460 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3778006fd0 tx=0x7f3778008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:12.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.524+0000 7f745b16a700 1 -- 172.21.15.98:0/2328044221 --> [v2:172.21.15.98:6817/90663,v1:172.21.15.98:6819/90663] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7434005c80 con 0x7f74340015f0 2023-10-06T05:07:12.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.525+0000 7f7450ff9700 1 -- 172.21.15.98:0/2328044221 <== osd.3 v2:172.21.15.98:6817/90663 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f7434005c80 con 0x7f74340015f0 2023-10-06T05:07:12.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.525+0000 7f745b16a700 1 -- 172.21.15.98:0/2328044221 >> [v2:172.21.15.98:6817/90663,v1:172.21.15.98:6819/90663] conn(0x7f74340015f0 msgr2=0x7f7434003aa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.525+0000 7f745b16a700 1 --2- 172.21.15.98:0/2328044221 >> [v2:172.21.15.98:6817/90663,v1:172.21.15.98:6819/90663] conn(0x7f74340015f0 0x7f7434003aa0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.526+0000 7f745b16a700 1 -- 172.21.15.98:0/2328044221 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f7438066fb0 msgr2=0x7f7438069460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.526+0000 7f745b16a700 1 --2- 172.21.15.98:0/2328044221 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f7438066fb0 0x7f7438069460 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f745406e6f0 tx=0x7f7440016000 comp rx=0 tx=0).stop 2023-10-06T05:07:12.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.526+0000 7f745b16a700 1 -- 172.21.15.98:0/2328044221 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7454160df0 msgr2=0x7f74541591a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.526+0000 7f745b16a700 1 --2- 172.21.15.98:0/2328044221 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7454160df0 0x7f74541591a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f744c00ceb0 tx=0x7f744c007870 comp rx=0 tx=0).stop 2023-10-06T05:07:12.573 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.526+0000 7f37828ff700 1 --2- 172.21.15.98:0/1558810778 >> [v2:172.21.15.98:6816/90662,v1:172.21.15.98:6818/90662] conn(0x7f377c176c60 0x7f377c0caf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.573 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.526+0000 7f1009769700 1 -- 172.21.15.98:0/96809018 >> [v2:172.21.15.174:6824/90878,v1:172.21.15.174:6825/90878] conn(0x7f0ff0070c50 msgr2=0x7f0ff0073080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.573 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.526+0000 7f1009769700 1 --2- 172.21.15.98:0/96809018 >> [v2:172.21.15.174:6824/90878,v1:172.21.15.174:6825/90878] conn(0x7f0ff0070c50 0x7f0ff0073080 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.526+0000 7f37828ff700 1 --2- 172.21.15.98:0/1558810778 >> [v2:172.21.15.98:6816/90662,v1:172.21.15.98:6818/90662] conn(0x7f377c176c60 0x7f377c0caf30 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:12.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.526+0000 7f5f55126700 1 -- 172.21.15.98:0/3777412 --> [v2:172.21.15.98:6808/90628,v1:172.21.15.98:6809/90628] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5f500c0120 con 0x7f5f50156250 2023-10-06T05:07:12.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.526+0000 7f1009769700 1 -- 172.21.15.98:0/96809018 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f0ff0067000 msgr2=0x7f0ff00694b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.526+0000 7f1009769700 1 --2- 172.21.15.98:0/96809018 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f0ff0067000 0x7f0ff00694b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f100406a670 tx=0x7f0ffc009370 comp rx=0 tx=0).stop 2023-10-06T05:07:12.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.526+0000 7f1009769700 1 -- 172.21.15.98:0/96809018 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f10040696e0 msgr2=0x7f1004154670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.526+0000 7f1009769700 1 --2- 172.21.15.98:0/96809018 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f10040696e0 0x7f1004154670 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0ff4003080 tx=0x7f0ff4016000 comp rx=0 tx=0).stop 2023-10-06T05:07:12.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.526+0000 7f10037fe700 1 -- 172.21.15.98:0/96809018 reap_dead start 2023-10-06T05:07:12.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.527+0000 7f1009769700 1 -- 172.21.15.98:0/96809018 shutdown_connections 2023-10-06T05:07:12.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.527+0000 7f1009769700 1 -- 172.21.15.98:0/96809018 >> 172.21.15.98:0/96809018 conn(0x7f1004068080 msgr2=0x7f10040c5a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:12.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.527+0000 7f1009769700 1 -- 172.21.15.98:0/96809018 shutdown_connections 2023-10-06T05:07:12.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.527+0000 7f1009769700 1 -- 172.21.15.98:0/96809018 wait complete. 2023-10-06T05:07:12.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.527+0000 7f5f36ffd700 1 -- 172.21.15.98:0/3777412 <== osd.1 v2:172.21.15.98:6808/90628 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f5f500c0120 con 0x7f5f50156250 2023-10-06T05:07:12.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.527+0000 7f9594a5e700 1 -- 172.21.15.98:0/4288607882 --> [v2:172.21.15.98:6800/90627,v1:172.21.15.98:6801/90627] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9590064a90 con 0x7f9570070c00 2023-10-06T05:07:12.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.527+0000 7f0b0e35f700 1 -- 172.21.15.98:0/1439664353 --> [v2:172.21.15.174:6800/90847,v1:172.21.15.174:6801/90847] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0b08064a90 con 0x7f0ae8070d50 2023-10-06T05:07:12.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.527+0000 7f5f55126700 1 -- 172.21.15.98:0/3777412 >> [v2:172.21.15.98:6808/90628,v1:172.21.15.98:6809/90628] conn(0x7f5f50156250 msgr2=0x7f5f5016e130 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.527+0000 7f5f55126700 1 --2- 172.21.15.98:0/3777412 >> [v2:172.21.15.98:6808/90628,v1:172.21.15.98:6809/90628] conn(0x7f5f50156250 0x7f5f5016e130 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.527+0000 7f37727fc700 1 -- 172.21.15.98:0/1558810778 <== osd.2 v2:172.21.15.98:6816/90662 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f377c06e560 con 0x7f377c176c60 2023-10-06T05:07:12.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.527+0000 7f5f55126700 1 -- 172.21.15.98:0/3777412 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f5f3c067000 msgr2=0x7f5f3c0694b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.527+0000 7f5f55126700 1 --2- 172.21.15.98:0/3777412 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f5f3c067000 0x7f5f3c0694b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5f400078a0 tx=0x7f5f40007830 comp rx=0 tx=0).stop 2023-10-06T05:07:12.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.527+0000 7f5f55126700 1 -- 172.21.15.98:0/3777412 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5f500cc7f0 msgr2=0x7f5f500c2530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.527+0000 7f5f55126700 1 --2- 172.21.15.98:0/3777412 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5f500cc7f0 0x7f5f500c2530 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f5f4400c590 tx=0x7f5f44007d50 comp rx=0 tx=0).stop 2023-10-06T05:07:12.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.527+0000 7f5f4f59e700 1 -- 172.21.15.98:0/3777412 reap_dead start 2023-10-06T05:07:12.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.528+0000 7f957e7fc700 1 -- 172.21.15.98:0/4288607882 <== osd.0 v2:172.21.15.98:6800/90627 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f9590064a90 con 0x7f9570070c00 2023-10-06T05:07:12.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.528+0000 7f5f55126700 1 -- 172.21.15.98:0/3777412 shutdown_connections 2023-10-06T05:07:12.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.528+0000 7f5f55126700 1 -- 172.21.15.98:0/3777412 >> 172.21.15.98:0/3777412 conn(0x7f5f50068080 msgr2=0x7f5f500b0ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:12.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.528+0000 7f0afffff700 1 -- 172.21.15.98:0/1439664353 <== osd.4 v2:172.21.15.174:6800/90847 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f0b08064a90 con 0x7f0ae8070d50 2023-10-06T05:07:12.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.528+0000 7f5f55126700 1 -- 172.21.15.98:0/3777412 shutdown_connections 2023-10-06T05:07:12.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.528+0000 7f0b0e35f700 1 -- 172.21.15.98:0/1439664353 >> [v2:172.21.15.174:6800/90847,v1:172.21.15.174:6801/90847] conn(0x7f0ae8070d50 msgr2=0x7f0ae8073180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.528+0000 7f0b0e35f700 1 --2- 172.21.15.98:0/1439664353 >> [v2:172.21.15.174:6800/90847,v1:172.21.15.174:6801/90847] conn(0x7f0ae8070d50 0x7f0ae8073180 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.528+0000 7f5f55126700 1 -- 172.21.15.98:0/3777412 wait complete. 2023-10-06T05:07:12.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.528+0000 7f0b0e35f700 1 -- 172.21.15.98:0/1439664353 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f0ae8067080 msgr2=0x7f0ae8069530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.528+0000 7f0b0e35f700 1 --2- 172.21.15.98:0/1439664353 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f0ae8067080 0x7f0ae8069530 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0b080c9b90 tx=0x7f0af8009040 comp rx=0 tx=0).stop 2023-10-06T05:07:12.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.528+0000 7f0b0e35f700 1 -- 172.21.15.98:0/1439664353 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0b08161b70 msgr2=0x7f0b08157030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.528+0000 7f0b0e35f700 1 --2- 172.21.15.98:0/1439664353 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0b08161b70 0x7f0b08157030 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0b0000cb10 tx=0x7f0b00001750 comp rx=0 tx=0).stop 2023-10-06T05:07:12.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.528+0000 7f0b0c8fc700 1 -- 172.21.15.98:0/1439664353 reap_dead start 2023-10-06T05:07:12.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.528+0000 7f0b0e35f700 1 -- 172.21.15.98:0/1439664353 shutdown_connections 2023-10-06T05:07:12.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.528+0000 7f0b0e35f700 1 -- 172.21.15.98:0/1439664353 >> 172.21.15.98:0/1439664353 conn(0x7f0b080cdeb0 msgr2=0x7f0b080c38a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:12.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.529+0000 7f0b0e35f700 1 -- 172.21.15.98:0/1439664353 shutdown_connections 2023-10-06T05:07:12.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.529+0000 7f0b0e35f700 1 -- 172.21.15.98:0/1439664353 wait complete. 2023-10-06T05:07:12.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.529+0000 7f7459707700 1 -- 172.21.15.98:0/2328044221 reap_dead start 2023-10-06T05:07:12.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.529+0000 7ff3e24c1700 1 -- 172.21.15.98:0/3292772014 --> [v2:172.21.15.174:6808/90848,v1:172.21.15.174:6809/90848] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff3dc15be80 con 0x7ff3c8070cd0 2023-10-06T05:07:12.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.529+0000 7f745b16a700 1 -- 172.21.15.98:0/2328044221 shutdown_connections 2023-10-06T05:07:12.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.529+0000 7f745b16a700 1 -- 172.21.15.98:0/2328044221 >> 172.21.15.98:0/2328044221 conn(0x7f7454068080 msgr2=0x7f74540cb5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:12.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.530+0000 7f745b16a700 1 -- 172.21.15.98:0/2328044221 shutdown_connections 2023-10-06T05:07:12.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.530+0000 7f745b16a700 1 -- 172.21.15.98:0/2328044221 wait complete. 2023-10-06T05:07:12.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.532+0000 7f9594a5e700 1 -- 172.21.15.98:0/4288607882 >> [v2:172.21.15.98:6800/90627,v1:172.21.15.98:6801/90627] conn(0x7f9570070c00 msgr2=0x7f9570073030 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.532+0000 7f9594a5e700 1 --2- 172.21.15.98:0/4288607882 >> [v2:172.21.15.98:6800/90627,v1:172.21.15.98:6801/90627] conn(0x7f9570070c00 0x7f9570073030 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.532+0000 7f9594a5e700 1 -- 172.21.15.98:0/4288607882 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f9570066f30 msgr2=0x7f95700693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.532+0000 7f9594a5e700 1 --2- 172.21.15.98:0/4288607882 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f9570066f30 0x7f95700693e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f95800099b0 tx=0x7f9580017000 comp rx=0 tx=0).stop 2023-10-06T05:07:12.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.532+0000 7f9594a5e700 1 -- 172.21.15.98:0/4288607882 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f95900ca5b0 msgr2=0x7f959016aff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.532+0000 7f9594a5e700 1 --2- 172.21.15.98:0/4288607882 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f95900ca5b0 0x7f959016aff0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f958800cb40 tx=0x7f9588001770 comp rx=0 tx=0).stop 2023-10-06T05:07:12.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.532+0000 7f958ed9d700 1 -- 172.21.15.98:0/4288607882 reap_dead start 2023-10-06T05:07:12.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.530+0000 7ff3c3fff700 1 -- 172.21.15.98:0/3292772014 <== osd.5 v2:172.21.15.174:6808/90848 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ff3dc15be80 con 0x7ff3c8070cd0 2023-10-06T05:07:12.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.530+0000 7ff3e24c1700 1 -- 172.21.15.98:0/3292772014 >> [v2:172.21.15.174:6808/90848,v1:172.21.15.174:6809/90848] conn(0x7ff3c8070cd0 msgr2=0x7ff3c8073100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.530+0000 7ff3e24c1700 1 --2- 172.21.15.98:0/3292772014 >> [v2:172.21.15.174:6808/90848,v1:172.21.15.174:6809/90848] conn(0x7ff3c8070cd0 0x7ff3c8073100 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.530+0000 7ff3e24c1700 1 -- 172.21.15.98:0/3292772014 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7ff3c8067000 msgr2=0x7ff3c80694b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.530+0000 7ff3e24c1700 1 --2- 172.21.15.98:0/3292772014 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7ff3c8067000 0x7ff3c80694b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff3d0000f60 tx=0x7ff3d0007b40 comp rx=0 tx=0).stop 2023-10-06T05:07:12.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.530+0000 7ff3e24c1700 1 -- 172.21.15.98:0/3292772014 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3dc157730 msgr2=0x7ff3dc149040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.530+0000 7ff3e24c1700 1 --2- 172.21.15.98:0/3292772014 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff3dc157730 0x7ff3dc149040 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff3cc00c7c0 tx=0x7ff3cc007f70 comp rx=0 tx=0).stop 2023-10-06T05:07:12.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.530+0000 7ff3e0a5e700 1 -- 172.21.15.98:0/3292772014 reap_dead start 2023-10-06T05:07:12.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.530+0000 7ff3e24c1700 1 -- 172.21.15.98:0/3292772014 shutdown_connections 2023-10-06T05:07:12.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.530+0000 7ff3e24c1700 1 -- 172.21.15.98:0/3292772014 >> 172.21.15.98:0/3292772014 conn(0x7ff3dc0cc6f0 msgr2=0x7ff3dc0b1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:12.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.531+0000 7ff3e24c1700 1 -- 172.21.15.98:0/3292772014 shutdown_connections 2023-10-06T05:07:12.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.531+0000 7ff3e24c1700 1 -- 172.21.15.98:0/3292772014 wait complete. 2023-10-06T05:07:12.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.536+0000 7f9594a5e700 1 -- 172.21.15.98:0/4288607882 shutdown_connections 2023-10-06T05:07:12.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.536+0000 7f9594a5e700 1 -- 172.21.15.98:0/4288607882 >> 172.21.15.98:0/4288607882 conn(0x7f9590087fc0 msgr2=0x7f95900cabc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:12.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.540+0000 7f61cbde3700 1 -- 172.21.15.98:0/110896907 --> [v2:172.21.15.174:6816/90877,v1:172.21.15.174:6817/90877] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f61a4002d30 con 0x7f61a8070d50 2023-10-06T05:07:12.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.538+0000 7f9594a5e700 1 -- 172.21.15.98:0/4288607882 shutdown_connections 2023-10-06T05:07:12.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.538+0000 7f9594a5e700 1 -- 172.21.15.98:0/4288607882 wait complete. 2023-10-06T05:07:12.587 INFO:teuthology.orchestra.run.smithi098.stdout:60129542152 2023-10-06T05:07:12.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.546+0000 7f3784362700 1 -- 172.21.15.98:0/1558810778 --> [v2:172.21.15.98:6816/90662,v1:172.21.15.98:6818/90662] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f377c04bb40 con 0x7f377c176c60 2023-10-06T05:07:12.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.545+0000 7f61b9ffb700 1 -- 172.21.15.98:0/110896907 <== osd.6 v2:172.21.15.174:6816/90877 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f61a4002d30 con 0x7f61a8070d50 2023-10-06T05:07:12.588 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-06T05:07:12.589 INFO:teuthology.orchestra.run.smithi098.stdout:64424509449 2023-10-06T05:07:12.589 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-10-06T05:07:12.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.550+0000 7f37727fc700 1 -- 172.21.15.98:0/1558810778 <== osd.2 v2:172.21.15.98:6816/90662 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f377c04bb40 con 0x7f377c176c60 2023-10-06T05:07:12.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.550+0000 7f3784362700 1 -- 172.21.15.98:0/1558810778 >> [v2:172.21.15.98:6816/90662,v1:172.21.15.98:6818/90662] conn(0x7f377c176c60 msgr2=0x7f377c0caf30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.550+0000 7f3784362700 1 --2- 172.21.15.98:0/1558810778 >> [v2:172.21.15.98:6816/90662,v1:172.21.15.98:6818/90662] conn(0x7f377c176c60 0x7f377c0caf30 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.550+0000 7f3784362700 1 -- 172.21.15.98:0/1558810778 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f3760066fb0 msgr2=0x7f3760069460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.550+0000 7f3784362700 1 --2- 172.21.15.98:0/1558810778 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f3760066fb0 0x7f3760069460 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3778006fd0 tx=0x7f3778008040 comp rx=0 tx=0).stop 2023-10-06T05:07:12.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.550+0000 7f3784362700 1 -- 172.21.15.98:0/1558810778 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f377c067800 msgr2=0x7f377c160d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.550+0000 7f3784362700 1 --2- 172.21.15.98:0/1558810778 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f377c067800 0x7f377c160d10 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f377c0af600 tx=0x7f3774018000 comp rx=0 tx=0).stop 2023-10-06T05:07:12.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.550+0000 7f37828ff700 1 -- 172.21.15.98:0/1558810778 reap_dead start 2023-10-06T05:07:12.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.551+0000 7f61cbde3700 1 -- 172.21.15.98:0/110896907 >> [v2:172.21.15.174:6816/90877,v1:172.21.15.174:6817/90877] conn(0x7f61a8070d50 msgr2=0x7f61a8073180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.551+0000 7f61cbde3700 1 --2- 172.21.15.98:0/110896907 >> [v2:172.21.15.174:6816/90877,v1:172.21.15.174:6817/90877] conn(0x7f61a8070d50 0x7f61a8073180 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.551+0000 7f61cbde3700 1 -- 172.21.15.98:0/110896907 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f61a8067080 msgr2=0x7f61a8069530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.551+0000 7f61cbde3700 1 --2- 172.21.15.98:0/110896907 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f61a8067080 0x7f61a8069530 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f61b4006fd0 tx=0x7f61b40090f0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.551+0000 7f61cbde3700 1 -- 172.21.15.98:0/110896907 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f61c40bf6f0 msgr2=0x7f61c415cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.551+0000 7f61cbde3700 1 --2- 172.21.15.98:0/110896907 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f61c40bf6f0 0x7f61c415cec0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f61c406fd70 tx=0x7f61c0016000 comp rx=0 tx=0).stop 2023-10-06T05:07:12.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.552+0000 7f61ca380700 1 -- 172.21.15.98:0/110896907 reap_dead start 2023-10-06T05:07:12.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.552+0000 7f61cbde3700 1 -- 172.21.15.98:0/110896907 shutdown_connections 2023-10-06T05:07:12.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.552+0000 7f61cbde3700 1 -- 172.21.15.98:0/110896907 >> 172.21.15.98:0/110896907 conn(0x7f61c4068080 msgr2=0x7f61c41617b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:12.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.552+0000 7f61cbde3700 1 -- 172.21.15.98:0/110896907 shutdown_connections 2023-10-06T05:07:12.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.552+0000 7f61cbde3700 1 -- 172.21.15.98:0/110896907 wait complete. 2023-10-06T05:07:12.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.553+0000 7f3784362700 1 -- 172.21.15.98:0/1558810778 shutdown_connections 2023-10-06T05:07:12.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.553+0000 7f3784362700 1 -- 172.21.15.98:0/1558810778 >> 172.21.15.98:0/1558810778 conn(0x7f377c0cc6f0 msgr2=0x7f377c06a270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:12.594 INFO:teuthology.orchestra.run.smithi098.stdout:64424509448 2023-10-06T05:07:12.594 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-10-06T05:07:12.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.554+0000 7f3784362700 1 -- 172.21.15.98:0/1558810778 shutdown_connections 2023-10-06T05:07:12.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.554+0000 7f3784362700 1 -- 172.21.15.98:0/1558810778 wait complete. 2023-10-06T05:07:12.595 INFO:teuthology.orchestra.run.smithi098.stdout:55834574856 2023-10-06T05:07:12.596 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-10-06T05:07:12.596 INFO:teuthology.orchestra.run.smithi098.stdout:64424509449 2023-10-06T05:07:12.597 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-10-06T05:07:12.597 INFO:teuthology.orchestra.run.smithi098.stdout:60129542153 2023-10-06T05:07:12.598 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-06T05:07:12.598 INFO:teuthology.orchestra.run.smithi098.stdout:64424509449 2023-10-06T05:07:12.599 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-10-06T05:07:12.600 INFO:teuthology.orchestra.run.smithi098.stdout:64424509448 2023-10-06T05:07:12.600 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-06T05:07:12.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.828+0000 7fce88daa700 1 Processor -- start 2023-10-06T05:07:12.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.828+0000 7f6bc6a3e700 1 Processor -- start 2023-10-06T05:07:12.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.828+0000 7fce88daa700 1 -- start start 2023-10-06T05:07:12.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.828+0000 7f6bc6a3e700 1 -- start start 2023-10-06T05:07:12.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.829+0000 7fce88daa700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fce840b49a0 0x7fce840aac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.829+0000 7fce88daa700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fce840b0760 0x7fce840b1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.829+0000 7fce88daa700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce840b0c60 con 0x7fce84159560 2023-10-06T05:07:12.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.829+0000 7fce88daa700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce840b0f40 con 0x7fce840b49a0 2023-10-06T05:07:12.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.829+0000 7fce88daa700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce840b1220 con 0x7fce840b0760 2023-10-06T05:07:12.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.829+0000 7f6bc6a3e700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6bc00ca9f0 0x7f6bc006c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.829+0000 7f6bc6a3e700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f6bc006c970 0x7f6bc0160b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.829+0000 7f6bc6a3e700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bc006bfa0 con 0x7f6bc0067800 2023-10-06T05:07:12.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.829+0000 7f6bc6a3e700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bc0162040 con 0x7f6bc006c970 2023-10-06T05:07:12.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.829+0000 7f6bc6a3e700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc0162320 con 0x7f6bc00ca9f0 2023-10-06T05:07:12.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.829+0000 7fce81d9b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fce840b49a0 0x7fce840aac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.829+0000 7fce8259c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fce84159560 0x7fce840aa530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:41174/0 (socket says 172.21.15.98:41174) 2023-10-06T05:07:12.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.829+0000 7fce8259c700 1 -- 172.21.15.98:0/3435000630 learned_addr learned my addr 172.21.15.98:0/3435000630 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:07:12.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.829+0000 7fce82d9d700 1 --2- 172.21.15.98:0/3435000630 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fce840b0760 0x7fce840b1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.829+0000 7fce81d9b700 1 -- 172.21.15.98:0/3435000630 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fce840b0760 msgr2=0x7fce840b1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:12.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.829+0000 7fce81d9b700 1 --2- 172.21.15.98:0/3435000630 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fce840b0760 0x7fce840b1bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.830+0000 7fce81d9b700 1 -- 172.21.15.98:0/3435000630 >> v1:172.21.15.98:6789/0 conn(0x7fce84159560 legacy=0x7fce840aa530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.829+0000 7f6bbf7fe700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6bc00ca9f0 0x7f6bc006c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.829+0000 7f6bbffff700 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f6bc0067800 0x7f6bc00c7a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.98:55498/0 (socket says 172.21.15.98:55498) 2023-10-06T05:07:12.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.829+0000 7f6bbffff700 1 -- 172.21.15.98:0/2976507889 learned_addr learned my addr 172.21.15.98:0/2976507889 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:07:12.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.830+0000 7fce81d9b700 1 -- 172.21.15.98:0/3435000630 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce840b1500 con 0x7fce840b49a0 2023-10-06T05:07:12.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.830+0000 7f6bc4fdb700 1 --2- 172.21.15.98:0/2976507889 >> v2:172.21.15.174:3300/0 conn(0x7f6bc006c970 0x7f6bc0160b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.830+0000 7fce81d9b700 1 --2- 172.21.15.98:0/3435000630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce840b49a0 0x7fce840aac40 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fce78009910 tx=0x7fce780091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=765bf73f461cb882 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:12.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.830+0000 7f6bbf7fe700 1 -- 172.21.15.98:0/2976507889 >> v2:172.21.15.174:3300/0 conn(0x7f6bc006c970 msgr2=0x7f6bc0160b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:12.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.830+0000 7f6bbf7fe700 1 --2- 172.21.15.98:0/2976507889 >> v2:172.21.15.174:3300/0 conn(0x7f6bc006c970 0x7f6bc0160b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.830+0000 7f6bbf7fe700 1 -- 172.21.15.98:0/2976507889 >> v1:172.21.15.174:6789/0 conn(0x7f6bc0067800 legacy=0x7f6bc00c7a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.830+0000 7f6bbf7fe700 1 -- 172.21.15.98:0/2976507889 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bc0162600 con 0x7f6bc00ca9f0 2023-10-06T05:07:12.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.830+0000 7f6bc4fdb700 1 --2- 172.21.15.98:0/2976507889 >> v2:172.21.15.174:3300/0 conn(0x7f6bc006c970 0x7f6bc0160b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:07:12.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.830+0000 7f6bbf7fe700 1 --2- 172.21.15.98:0/2976507889 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6bc00ca9f0 0x7f6bc006c430 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f6bb40098e0 tx=0x7f6bb40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf701adbb4292927 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:12.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.830+0000 7f6bbeffd700 1 -- 172.21.15.98:0/2976507889 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bb4009e80 con 0x7f6bc00ca9f0 2023-10-06T05:07:12.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.830+0000 7f6bbeffd700 1 -- 172.21.15.98:0/2976507889 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bb400b860 con 0x7f6bc00ca9f0 2023-10-06T05:07:12.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.830+0000 7f6bbeffd700 1 -- 172.21.15.98:0/2976507889 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bb400bbe0 con 0x7f6bc00ca9f0 2023-10-06T05:07:12.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.831+0000 7fce8159a700 1 -- 172.21.15.98:0/3435000630 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce78009eb0 con 0x7fce840b49a0 2023-10-06T05:07:12.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.831+0000 7fce8159a700 1 -- 172.21.15.98:0/3435000630 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce7800b760 con 0x7fce840b49a0 2023-10-06T05:07:12.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.831+0000 7fce8159a700 1 -- 172.21.15.98:0/3435000630 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce7800bb00 con 0x7fce840b49a0 2023-10-06T05:07:12.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.831+0000 7f6bc6a3e700 1 -- 172.21.15.98:0/2976507889 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6bc00ca9f0 msgr2=0x7f6bc006c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.831+0000 7f6bc6a3e700 1 --2- 172.21.15.98:0/2976507889 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6bc00ca9f0 0x7f6bc006c430 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f6bb40098e0 tx=0x7f6bb40091a0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.831+0000 7f6bc6a3e700 1 -- 172.21.15.98:0/2976507889 shutdown_connections 2023-10-06T05:07:12.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.831+0000 7f6bc6a3e700 1 --2- 172.21.15.98:0/2976507889 >> v2:172.21.15.174:3300/0 conn(0x7f6bc006c970 0x7f6bc0160b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.831+0000 7f6bc6a3e700 1 --2- 172.21.15.98:0/2976507889 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6bc00ca9f0 0x7f6bc006c430 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.831+0000 7f6bc6a3e700 1 -- 172.21.15.98:0/2976507889 >> 172.21.15.98:0/2976507889 conn(0x7f6bc00cdeb0 msgr2=0x7f6bc00c9e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:12.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.831+0000 7fce88daa700 1 -- 172.21.15.98:0/3435000630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce840b49a0 msgr2=0x7fce840aac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.831+0000 7fce88daa700 1 --2- 172.21.15.98:0/3435000630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce840b49a0 0x7fce840aac40 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fce78009910 tx=0x7fce780091a0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.831+0000 7fce88daa700 1 -- 172.21.15.98:0/3435000630 shutdown_connections 2023-10-06T05:07:12.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.831+0000 7fce88daa700 1 --2- 172.21.15.98:0/3435000630 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fce840b0760 0x7fce840b1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.831+0000 7fce88daa700 1 --2- 172.21.15.98:0/3435000630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce840b49a0 0x7fce840aac40 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.831+0000 7fce88daa700 1 -- 172.21.15.98:0/3435000630 >> 172.21.15.98:0/3435000630 conn(0x7fce840997f0 msgr2=0x7fce8406f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:12.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.832+0000 7fce88daa700 1 -- 172.21.15.98:0/3435000630 shutdown_connections 2023-10-06T05:07:12.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.832+0000 7fce88daa700 1 -- 172.21.15.98:0/3435000630 wait complete. 2023-10-06T05:07:12.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.833+0000 7f6bc6a3e700 1 -- 172.21.15.98:0/2976507889 shutdown_connections 2023-10-06T05:07:12.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.833+0000 7f56fa969700 1 Processor -- start 2023-10-06T05:07:12.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.833+0000 7f56fa969700 1 -- start start 2023-10-06T05:07:12.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.833+0000 7fce88daa700 1 Processor -- start 2023-10-06T05:07:12.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.833+0000 7f13bab17700 1 Processor -- start 2023-10-06T05:07:12.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.833+0000 7fa76cbe4700 1 Processor -- start 2023-10-06T05:07:12.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.833+0000 7fa76cbe4700 1 -- start start 2023-10-06T05:07:12.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.834+0000 7f56fa969700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f56f40ca9f0 0x7f56f406c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.834+0000 7f56fa969700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f56f406c970 0x7f56f4160b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.834+0000 7f56fa969700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56f406bfa0 con 0x7f56f4067800 2023-10-06T05:07:12.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.834+0000 7f56fa969700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56f4162040 con 0x7f56f40ca9f0 2023-10-06T05:07:12.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.834+0000 7f56fa969700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56f4162320 con 0x7f56f406c970 2023-10-06T05:07:12.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.833+0000 7fce88daa700 1 -- start start 2023-10-06T05:07:12.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.834+0000 7fce88daa700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fce840b0760 0x7fce841569f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.834+0000 7fce88daa700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fce840b49a0 0x7fce84153ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.834+0000 7fce88daa700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce84159560 0x7fce8414f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.834+0000 7fce88daa700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce84169bf0 con 0x7fce84159560 2023-10-06T05:07:12.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.834+0000 7fce88daa700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce8416a740 con 0x7fce840b0760 2023-10-06T05:07:12.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.834+0000 7fce88daa700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce841543e0 con 0x7fce840b49a0 2023-10-06T05:07:12.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.834+0000 7f56f3fff700 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f56f4067800 0x7f56f40c7a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.98:55506/0 (socket says 172.21.15.98:55506) 2023-10-06T05:07:12.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.834+0000 7f56f3fff700 1 -- 172.21.15.98:0/1680992592 learned_addr learned my addr 172.21.15.98:0/1680992592 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:07:12.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.834+0000 7f56f37fe700 1 --2- 172.21.15.98:0/1680992592 >> v2:172.21.15.98:3300/0 conn(0x7f56f40ca9f0 0x7f56f406c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.834+0000 7f56f8f06700 1 --2- 172.21.15.98:0/1680992592 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f56f406c970 0x7f56f4160b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.834+0000 7fce81d9b700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fce840b49a0 0x7fce84153ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.834+0000 7fce8259c700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fce840b0760 0x7fce841569f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.834+0000 7fce82d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce84159560 0x7fce8414f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.834+0000 7fce82d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce84159560 0x7fce8414f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45818/0 (socket says 172.21.15.98:45818) 2023-10-06T05:07:12.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.834+0000 7fce82d9d700 1 -- 172.21.15.98:0/1083354292 learned_addr learned my addr 172.21.15.98:0/1083354292 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:12.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.833+0000 7f6bc6a3e700 1 -- 172.21.15.98:0/2976507889 wait complete. 2023-10-06T05:07:12.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.835+0000 7f6bc6a3e700 1 Processor -- start 2023-10-06T05:07:12.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.834+0000 7fa76cbe4700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa7680b2350 0x7fa7680b2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.834+0000 7fa76cbe4700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa768162330 0x7fa7680cb410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.834+0000 7fa76cbe4700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa76815e130 con 0x7fa768161780 2023-10-06T05:07:12.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.834+0000 7fa76cbe4700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa76815e300 con 0x7fa7680b2350 2023-10-06T05:07:12.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.834+0000 7fa76cbe4700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7680cbcc0 con 0x7fa768162330 2023-10-06T05:07:12.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.835+0000 7fa76659c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa7680b2350 0x7fa7680b2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.833+0000 7f13bab17700 1 -- start start 2023-10-06T05:07:12.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.834+0000 7f13bab17700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f13b40ca9f0 0x7f13b406c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.834+0000 7f13bab17700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13b406bfa0 con 0x7f13b406c970 2023-10-06T05:07:12.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.834+0000 7f13bab17700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13b4162040 con 0x7f13b4067800 2023-10-06T05:07:12.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.834+0000 7f13bab17700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13b4162320 con 0x7f13b40ca9f0 2023-10-06T05:07:12.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.834+0000 7f13b88b3700 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f13b4067800 0x7f13b40c7a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.98:55504/0 (socket says 172.21.15.98:55504) 2023-10-06T05:07:12.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.834+0000 7f13b88b3700 1 -- 172.21.15.98:0/1578891850 learned_addr learned my addr 172.21.15.98:0/1578891850 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:07:12.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.834+0000 7f13b3fff700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f13b40ca9f0 0x7f13b406c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.835+0000 7fa76659c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa7680b2350 0x7fa7680b2720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45840/0 (socket says 172.21.15.98:45840) 2023-10-06T05:07:12.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.835+0000 7fa76659c700 1 -- 172.21.15.98:0/1541367980 learned_addr learned my addr 172.21.15.98:0/1541367980 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:12.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.835+0000 7f13b3fff700 1 -- 172.21.15.98:0/1578891850 >> v1:172.21.15.174:6789/0 conn(0x7f13b4067800 legacy=0x7f13b40c7a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.835+0000 7f13b3fff700 1 -- 172.21.15.98:0/1578891850 >> v1:172.21.15.98:6789/0 conn(0x7f13b406c970 legacy=0x7f13b4160b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.835+0000 7f13b3fff700 1 -- 172.21.15.98:0/1578891850 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13b4162600 con 0x7f13b40ca9f0 2023-10-06T05:07:12.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.835+0000 7f56f8f06700 1 -- 172.21.15.98:0/1680992592 >> v2:172.21.15.98:3300/0 conn(0x7f56f40ca9f0 msgr2=0x7f56f406c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:12.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.835+0000 7f56f8f06700 1 --2- 172.21.15.98:0/1680992592 >> v2:172.21.15.98:3300/0 conn(0x7f56f40ca9f0 0x7f56f406c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.835+0000 7f56f2ffd700 1 -- 172.21.15.98:0/1680992592 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 655496981 0 0) 0x7f56f406bfa0 con 0x7f56f4067800 2023-10-06T05:07:12.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.835+0000 7f56f8f06700 1 -- 172.21.15.98:0/1680992592 >> v1:172.21.15.174:6789/0 conn(0x7f56f4067800 legacy=0x7f56f40c7a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.835+0000 7f56f8f06700 1 -- 172.21.15.98:0/1680992592 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56f4162600 con 0x7f56f406c970 2023-10-06T05:07:12.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.835+0000 7f6bc6a3e700 1 -- start start 2023-10-06T05:07:12.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.835+0000 7fa765d9b700 1 --2- 172.21.15.98:0/1541367980 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa768162330 0x7fa7680cb410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.835+0000 7fce82d9d700 1 -- 172.21.15.98:0/1083354292 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fce840b49a0 msgr2=0x7fce84153ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.835+0000 7fce82d9d700 1 --2- 172.21.15.98:0/1083354292 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fce840b49a0 0x7fce84153ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.835+0000 7fce82d9d700 1 -- 172.21.15.98:0/1083354292 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fce840b0760 msgr2=0x7fce841569f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.835+0000 7fce82d9d700 1 --2- 172.21.15.98:0/1083354292 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fce840b0760 0x7fce841569f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.835+0000 7fce82d9d700 1 -- 172.21.15.98:0/1083354292 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce78009420 con 0x7fce84159560 2023-10-06T05:07:12.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.835+0000 7f13b3fff700 1 --2- 172.21.15.98:0/1578891850 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f13b40ca9f0 0x7f13b406c430 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f13a0009860 tx=0x7f13a00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d1857b15dea93e6 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:12.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.835+0000 7f56f8f06700 1 --2- 172.21.15.98:0/1680992592 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f56f406c970 0x7f56f4160b60 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f56e40098e0 tx=0x7f56e40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85cf921ce5c72689 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:12.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.835+0000 7f56f2ffd700 1 -- 172.21.15.98:0/1680992592 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56e400b640 con 0x7f56f406c970 2023-10-06T05:07:12.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.835+0000 7fa76659c700 1 -- 172.21.15.98:0/1541367980 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa768162330 msgr2=0x7fa7680cb410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:12.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.835+0000 7fa76659c700 1 --2- 172.21.15.98:0/1541367980 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa768162330 0x7fa7680cb410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.835+0000 7fa76559a700 1 -- 172.21.15.98:0/1541367980 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 988649605 0 0) 0x7fa76815e130 con 0x7fa768161780 2023-10-06T05:07:12.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.835+0000 7fa76659c700 1 -- 172.21.15.98:0/1541367980 >> v1:172.21.15.174:6789/0 conn(0x7fa768161780 legacy=0x7fa7680b4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.835+0000 7fa76659c700 1 -- 172.21.15.98:0/1541367980 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7680cbfa0 con 0x7fa7680b2350 2023-10-06T05:07:12.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.835+0000 7f2246104700 1 Processor -- start 2023-10-06T05:07:12.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.835+0000 7f2246104700 1 -- start start 2023-10-06T05:07:12.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.836+0000 7fce82d9d700 1 --2- 172.21.15.98:0/1083354292 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce84159560 0x7fce8414f660 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fce7c007f30 tx=0x7fce7c0098e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:12.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.836+0000 7fa76659c700 1 --2- 172.21.15.98:0/1541367980 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7680b2350 0x7fa7680b2720 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa7580099b0 tx=0x7fa758009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6a96d735ffef96 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:12.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.836+0000 7fc58dd88700 1 Processor -- start 2023-10-06T05:07:12.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.836+0000 7fc58dd88700 1 -- start start 2023-10-06T05:07:12.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.836+0000 7fa76559a700 1 -- 172.21.15.98:0/1541367980 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa758009ea0 con 0x7fa7680b2350 2023-10-06T05:07:12.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.836+0000 7f13b37fe700 1 -- 172.21.15.98:0/1578891850 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a0011630 con 0x7f13b40ca9f0 2023-10-06T05:07:12.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.836+0000 7fa76559a700 1 -- 172.21.15.98:0/1541367980 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa75800b860 con 0x7fa7680b2350 2023-10-06T05:07:12.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.836+0000 7fa76559a700 1 -- 172.21.15.98:0/1541367980 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa75800bc00 con 0x7fa7680b2350 2023-10-06T05:07:12.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.836+0000 7fd912b2d700 1 Processor -- start 2023-10-06T05:07:12.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.836+0000 7f13b37fe700 1 -- 172.21.15.98:0/1578891850 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13a0011790 con 0x7f13b40ca9f0 2023-10-06T05:07:12.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.836+0000 7f13b37fe700 1 -- 172.21.15.98:0/1578891850 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a0011b50 con 0x7f13b40ca9f0 2023-10-06T05:07:12.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.836+0000 7f6bc6a3e700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6bc0067800 0x7f6bc01283b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.836+0000 7f6bc6a3e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bc006c970 0x7f6bc00c3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.836+0000 7f6bc6a3e700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6bc00ca9f0 0x7f6bc00c3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.836+0000 7f6bc6a3e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc01775a0 con 0x7f6bc006c970 2023-10-06T05:07:12.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.836+0000 7f6bc6a3e700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc017a900 con 0x7f6bc0067800 2023-10-06T05:07:12.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.836+0000 7f6bc6a3e700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc01288f0 con 0x7f6bc00ca9f0 2023-10-06T05:07:12.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.836+0000 7f2246104700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f22400ca370 0x7f2240158270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.836+0000 7f2246104700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f22400ce8a0 0x7f224015aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.836+0000 7f2246104700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2240157de0 con 0x7f22400adb60 2023-10-06T05:07:12.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.836+0000 7f2246104700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22400cbb90 con 0x7f22400ca370 2023-10-06T05:07:12.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.836+0000 7f2246104700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22400cb5d0 con 0x7f22400ce8a0 2023-10-06T05:07:12.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.837+0000 7f56f2ffd700 1 -- 172.21.15.98:0/1680992592 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56e400b7a0 con 0x7f56f406c970 2023-10-06T05:07:12.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.837+0000 7f56f2ffd700 1 -- 172.21.15.98:0/1680992592 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56e400bb40 con 0x7f56f406c970 2023-10-06T05:07:12.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.837+0000 7f223effd700 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f22400adb60 0x7f22400ce190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.98:55534/0 (socket says 172.21.15.98:55534) 2023-10-06T05:07:12.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.837+0000 7f223effd700 1 -- 172.21.15.98:0/2915025176 learned_addr learned my addr 172.21.15.98:0/2915025176 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:07:12.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.837+0000 7f223ffff700 1 --2- 172.21.15.98:0/2915025176 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f22400ce8a0 0x7f224015aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.836+0000 7fce727fc700 1 -- 172.21.15.98:0/1083354292 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce7c00ec40 con 0x7fce84159560 2023-10-06T05:07:12.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.836+0000 7fce727fc700 1 -- 172.21.15.98:0/1083354292 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce7c00eda0 con 0x7fce84159560 2023-10-06T05:07:12.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.836+0000 7fce727fc700 1 -- 172.21.15.98:0/1083354292 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce7c014670 con 0x7fce84159560 2023-10-06T05:07:12.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.837+0000 7f223e7fc700 1 -- 172.21.15.98:0/2915025176 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 500477730 0 0) 0x7f2240157de0 con 0x7f22400adb60 2023-10-06T05:07:12.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.837+0000 7f223e7fc700 1 -- 172.21.15.98:0/2915025176 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2228003570 con 0x7f22400adb60 2023-10-06T05:07:12.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.837+0000 7fc58dd88700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc588159cc0 0x7fc58806e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.837+0000 7fc58dd88700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc58806edc0 0x7fc5880c7030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.837+0000 7fc58dd88700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5880ac940 con 0x7fc5880c7570 2023-10-06T05:07:12.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.837+0000 7fc58dd88700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5880c8510 con 0x7fc588159cc0 2023-10-06T05:07:12.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.837+0000 7fc58dd88700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5880ccd10 con 0x7fc58806edc0 2023-10-06T05:07:12.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.837+0000 7fa76cbe4700 1 -- 172.21.15.98:0/1541367980 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7680b2350 msgr2=0x7fa7680b2720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.837+0000 7fa76cbe4700 1 --2- 172.21.15.98:0/1541367980 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7680b2350 0x7fa7680b2720 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa7580099b0 tx=0x7fa758009250 comp rx=0 tx=0).stop 2023-10-06T05:07:12.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.837+0000 7fa76cbe4700 1 -- 172.21.15.98:0/1541367980 shutdown_connections 2023-10-06T05:07:12.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.837+0000 7fa76cbe4700 1 --2- 172.21.15.98:0/1541367980 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7680b2350 0x7fa7680b2720 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.837+0000 7fa76cbe4700 1 --2- 172.21.15.98:0/1541367980 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa768162330 0x7fa7680cb410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.837+0000 7fa76cbe4700 1 -- 172.21.15.98:0/1541367980 >> 172.21.15.98:0/1541367980 conn(0x7fa7680ccf30 msgr2=0x7fa7680cd330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:12.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.837+0000 7f13bab17700 1 -- 172.21.15.98:0/1578891850 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f13b40ca9f0 msgr2=0x7f13b406c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.837+0000 7f13bab17700 1 --2- 172.21.15.98:0/1578891850 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f13b40ca9f0 0x7f13b406c430 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f13a0009860 tx=0x7f13a00091a0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.837+0000 7f223ffff700 1 -- 172.21.15.98:0/2915025176 >> v2:172.21.15.98:3300/0 conn(0x7f22400ca370 msgr2=0x7f2240158270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:12.864 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.837+0000 7f223ffff700 1 --2- 172.21.15.98:0/2915025176 >> v2:172.21.15.98:3300/0 conn(0x7f22400ca370 0x7f2240158270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.864 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.837+0000 7f223ffff700 1 -- 172.21.15.98:0/2915025176 >> v1:172.21.15.174:6789/0 conn(0x7f22400adb60 legacy=0x7f22400ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.864 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.837+0000 7f223ffff700 1 -- 172.21.15.98:0/2915025176 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22400cb8b0 con 0x7f22400ce8a0 2023-10-06T05:07:12.864 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.836+0000 7fd912b2d700 1 -- start start 2023-10-06T05:07:12.864 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.837+0000 7fc586ffd700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc58806edc0 0x7fc5880c7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.837+0000 7fd912b2d700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd90c068be0 0x7fd90c155d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.837+0000 7fd912b2d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd90c0af110 con 0x7fd90c0ac440 2023-10-06T05:07:12.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.837+0000 7fd912b2d700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd90c0ac8d0 con 0x7fd90c068810 2023-10-06T05:07:12.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.837+0000 7fd912b2d700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd90c0acbb0 con 0x7fd90c068be0 2023-10-06T05:07:12.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.837+0000 7fc586ffd700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc58806edc0 0x7fc5880c7030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.98:55976/0 (socket says 172.21.15.98:55976) 2023-10-06T05:07:12.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.837+0000 7fc586ffd700 1 -- 172.21.15.98:0/1862581699 learned_addr learned my addr 172.21.15.98:0/1862581699 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:12.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.837+0000 7fc587fff700 1 --1- 172.21.15.98:0/1862581699 >> v1:172.21.15.174:6789/0 conn(0x7fc5880c7570 0x7fc58806d350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.98:55540/0 (socket says 172.21.15.98:55540) 2023-10-06T05:07:12.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.837+0000 7fa76cbe4700 1 -- 172.21.15.98:0/1541367980 shutdown_connections 2023-10-06T05:07:12.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.837+0000 7fa76cbe4700 1 -- 172.21.15.98:0/1541367980 wait complete. 2023-10-06T05:07:12.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.837+0000 7f6bc4fdb700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6bc00ca9f0 0x7f6bc00c3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.837+0000 7f13bab17700 1 -- 172.21.15.98:0/1578891850 shutdown_connections 2023-10-06T05:07:12.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.837+0000 7f13bab17700 1 --2- 172.21.15.98:0/1578891850 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f13b40ca9f0 0x7f13b406c430 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.837+0000 7f13bab17700 1 -- 172.21.15.98:0/1578891850 >> 172.21.15.98:0/1578891850 conn(0x7f13b40cdeb0 msgr2=0x7f13b40c9e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:12.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.837+0000 7fd9108c9700 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fd90c068810 0x7fd90c0afb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.98:55546/0 (socket says 172.21.15.98:55546) 2023-10-06T05:07:12.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.838+0000 7fd9108c9700 1 -- 172.21.15.98:0/2401582762 learned_addr learned my addr 172.21.15.98:0/2401582762 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:07:12.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.838+0000 7f223ffff700 1 --2- 172.21.15.98:0/2915025176 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f22400ce8a0 0x7f224015aed0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f2238009e60 tx=0x7f2238009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88408381e5f3e0e2 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:12.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.838+0000 7f56fa969700 1 -- 172.21.15.98:0/1680992592 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f56f406c970 msgr2=0x7f56f4160b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.838+0000 7f56fa969700 1 --2- 172.21.15.98:0/1680992592 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f56f406c970 0x7f56f4160b60 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f56e40098e0 tx=0x7f56e40091a0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.838+0000 7f6bbffff700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6bc0067800 0x7f6bc01283b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.838+0000 7f6bbffff700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6bc0067800 0x7f6bc01283b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.98:51338/0 (socket says 172.21.15.98:51338) 2023-10-06T05:07:12.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.838+0000 7f6bbffff700 1 -- 172.21.15.98:0/2786613481 learned_addr learned my addr 172.21.15.98:0/2786613481 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:12.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.838+0000 7f13bab17700 1 -- 172.21.15.98:0/1578891850 shutdown_connections 2023-10-06T05:07:12.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.838+0000 7fc5867fc700 1 -- 172.21.15.98:0/1862581699 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1835716935 0 0) 0x7fc5880ac940 con 0x7fc5880c7570 2023-10-06T05:07:12.870 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.838+0000 7fc5867fc700 1 -- 172.21.15.98:0/1862581699 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc570003570 con 0x7fc5880c7570 2023-10-06T05:07:12.870 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.838+0000 7f13bab17700 1 -- 172.21.15.98:0/1578891850 wait complete. 2023-10-06T05:07:12.870 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.838+0000 7fd90b7fe700 1 -- 172.21.15.98:0/2401582762 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4116827001 0 0) 0x7fd90c0ac8d0 con 0x7fd90c068810 2023-10-06T05:07:12.871 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.838+0000 7fd90b7fe700 1 -- 172.21.15.98:0/2401582762 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8f4003570 con 0x7fd90c068810 2023-10-06T05:07:12.871 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.838+0000 7f56fa969700 1 -- 172.21.15.98:0/1680992592 shutdown_connections 2023-10-06T05:07:12.871 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.838+0000 7f56fa969700 1 --2- 172.21.15.98:0/1680992592 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f56f406c970 0x7f56f4160b60 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.871 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.838+0000 7f56fa969700 1 --2- 172.21.15.98:0/1680992592 >> v2:172.21.15.98:3300/0 conn(0x7f56f40ca9f0 0x7f56f406c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.871 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.838+0000 7f56fa969700 1 -- 172.21.15.98:0/1680992592 >> 172.21.15.98:0/1680992592 conn(0x7f56f40cdeb0 msgr2=0x7f56f40c9e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:12.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.838+0000 7fc586ffd700 1 -- 172.21.15.98:0/1862581699 >> v2:172.21.15.98:3300/0 conn(0x7fc588159cc0 msgr2=0x7fc58806e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:12.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.838+0000 7fc586ffd700 1 --2- 172.21.15.98:0/1862581699 >> v2:172.21.15.98:3300/0 conn(0x7fc588159cc0 0x7fc58806e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.838+0000 7fc586ffd700 1 -- 172.21.15.98:0/1862581699 >> v1:172.21.15.174:6789/0 conn(0x7fc5880c7570 legacy=0x7fc58806d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.838+0000 7fc586ffd700 1 -- 172.21.15.98:0/1862581699 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc58806f230 con 0x7fc58806edc0 2023-10-06T05:07:12.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.838+0000 7f6bbffff700 1 -- 172.21.15.98:0/2786613481 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6bc00ca9f0 msgr2=0x7f6bc00c3dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.838+0000 7f6bbffff700 1 --2- 172.21.15.98:0/2786613481 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6bc00ca9f0 0x7f6bc00c3dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.838+0000 7f6bbffff700 1 -- 172.21.15.98:0/2786613481 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bc006c970 msgr2=0x7f6bc00c3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.838+0000 7f6bbffff700 1 --2- 172.21.15.98:0/2786613481 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bc006c970 0x7f6bc00c3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.838+0000 7f6bbffff700 1 -- 172.21.15.98:0/2786613481 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bb4009420 con 0x7f6bc0067800 2023-10-06T05:07:12.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.838+0000 7fd90b7fe700 1 -- 172.21.15.98:0/2401582762 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 169534251 0 0) 0x7fd8f4003570 con 0x7fd90c068810 2023-10-06T05:07:12.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.838+0000 7fd90b7fe700 1 -- 172.21.15.98:0/2401582762 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd90c068be0 msgr2=0x7fd90c155d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:12.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.838+0000 7fd90b7fe700 1 --2- 172.21.15.98:0/2401582762 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd90c068be0 0x7fd90c155d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.838+0000 7fd90b7fe700 1 -- 172.21.15.98:0/2401582762 >> v1:172.21.15.98:6789/0 conn(0x7fd90c0ac440 legacy=0x7fd90c155600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:12.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.838+0000 7fd90b7fe700 1 -- 172.21.15.98:0/2401582762 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd90c0ace90 con 0x7fd90c068810 2023-10-06T05:07:12.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.838+0000 7fd90b7fe700 1 -- 172.21.15.98:0/2401582762 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1543991876 0 0) 0x7fd8fc0014c0 con 0x7fd90c068810 2023-10-06T05:07:12.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.838+0000 7fd90b7fe700 1 -- 172.21.15.98:0/2401582762 >> v1:172.21.15.174:6789/0 conn(0x7fd90c068810 legacy=0x7fd90c0afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.838+0000 7f223e7fc700 1 -- 172.21.15.98:0/2915025176 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f223800e070 con 0x7f22400ce8a0 2023-10-06T05:07:12.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.838+0000 7f223e7fc700 1 -- 172.21.15.98:0/2915025176 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f223800bb20 con 0x7f22400ce8a0 2023-10-06T05:07:12.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.838+0000 7f223e7fc700 1 -- 172.21.15.98:0/2915025176 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2238012430 con 0x7f22400ce8a0 2023-10-06T05:07:12.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.838+0000 7f6bbffff700 1 --2- 172.21.15.98:0/2786613481 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6bc0067800 0x7f6bc01283b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6bb000c7c0 tx=0x7f6bb0007f70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:12.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.838+0000 7fd90b7fe700 1 --2- 172.21.15.98:0/2401582762 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd8f4003ae0 0x7fd8f4003ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.838+0000 7fd90b7fe700 1 --2- 172.21.15.98:0/2401582762 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd8f4004810 0x7fd8f4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.838+0000 7fd90b7fe700 1 --2- 172.21.15.98:0/2401582762 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8f40053c0 0x7fd8f4005a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.838+0000 7fd90b7fe700 1 -- 172.21.15.98:0/2401582762 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd90c0acbb0 con 0x7fd8f40053c0 2023-10-06T05:07:12.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.838+0000 7fd90b7fe700 1 -- 172.21.15.98:0/2401582762 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f4006220 con 0x7fd8f4003ae0 2023-10-06T05:07:12.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.838+0000 7fd90b7fe700 1 -- 172.21.15.98:0/2401582762 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f40063e0 con 0x7fd8f4004810 2023-10-06T05:07:12.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.839+0000 7fd9108c9700 1 --2- 172.21.15.98:0/2401582762 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd8f4003ae0 0x7fd8f4003ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.877 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.837+0000 7fce88daa700 1 -- 172.21.15.98:0/1083354292 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce84169e10 con 0x7fce84159560 2023-10-06T05:07:12.877 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.837+0000 7fce88daa700 1 -- 172.21.15.98:0/1083354292 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce8416a3d0 con 0x7fce84159560 2023-10-06T05:07:12.877 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.839+0000 7fce88daa700 1 -- 172.21.15.98:0/1083354292 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce8416a8d0 con 0x7fce84159560 2023-10-06T05:07:12.877 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.839+0000 7fd9110ca700 1 --2- 172.21.15.98:0/2401582762 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8f40053c0 0x7fd8f4005a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.877 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.839+0000 7f56fa969700 1 -- 172.21.15.98:0/1680992592 shutdown_connections 2023-10-06T05:07:12.878 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.839+0000 7fc586ffd700 1 --2- 172.21.15.98:0/1862581699 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc58806edc0 0x7fc5880c7030 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc57c00bc50 tx=0x7fc57c00b4f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fded0b7db36aa306 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:12.878 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.839+0000 7fa76cbe4700 1 Processor -- start 2023-10-06T05:07:12.878 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.839+0000 7f2246104700 1 -- 172.21.15.98:0/2915025176 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f22400ce8a0 msgr2=0x7f224015aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.878 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.839+0000 7f2246104700 1 --2- 172.21.15.98:0/2915025176 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f22400ce8a0 0x7f224015aed0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f2238009e60 tx=0x7f2238009780 comp rx=0 tx=0).stop 2023-10-06T05:07:12.878 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.839+0000 7f2246104700 1 -- 172.21.15.98:0/2915025176 shutdown_connections 2023-10-06T05:07:12.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.839+0000 7f2246104700 1 --2- 172.21.15.98:0/2915025176 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f22400ce8a0 0x7f224015aed0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.839+0000 7f2246104700 1 --2- 172.21.15.98:0/2915025176 >> v2:172.21.15.98:3300/0 conn(0x7f22400ca370 0x7f2240158270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.839+0000 7f2246104700 1 -- 172.21.15.98:0/2915025176 >> 172.21.15.98:0/2915025176 conn(0x7f22400707d0 msgr2=0x7f2240070bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:12.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.839+0000 7fd9110ca700 1 -- 172.21.15.98:0/2401582762 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd8f4004810 msgr2=0x7fd8f4004c80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-06T05:07:12.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.839+0000 7fd9110ca700 1 --2- 172.21.15.98:0/2401582762 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd8f4004810 0x7fd8f4004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.839+0000 7fd9110ca700 1 -- 172.21.15.98:0/2401582762 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd8f4003ae0 msgr2=0x7fd8f4003ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:12.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.839+0000 7fd9110ca700 1 --2- 172.21.15.98:0/2401582762 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd8f4003ae0 0x7fd8f4003ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.839+0000 7fd9110ca700 1 -- 172.21.15.98:0/2401582762 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8f4003570 con 0x7fd8f40053c0 2023-10-06T05:07:12.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.839+0000 7f2246104700 1 -- 172.21.15.98:0/2915025176 shutdown_connections 2023-10-06T05:07:12.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.839+0000 7f13bab17700 1 Processor -- start 2023-10-06T05:07:12.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.839+0000 7f2246104700 1 -- 172.21.15.98:0/2915025176 wait complete. 2023-10-06T05:07:12.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.839+0000 7f6ba7fff700 1 -- 172.21.15.98:0/2786613481 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bb0005b30 con 0x7f6bc0067800 2023-10-06T05:07:12.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.839+0000 7f6ba7fff700 1 -- 172.21.15.98:0/2786613481 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bb0005c90 con 0x7f6bc0067800 2023-10-06T05:07:12.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.839+0000 7f6ba7fff700 1 -- 172.21.15.98:0/2786613481 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bb0004a80 con 0x7f6bc0067800 2023-10-06T05:07:12.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.839+0000 7fc5867fc700 1 -- 172.21.15.98:0/1862581699 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc57c00dbb0 con 0x7fc58806edc0 2023-10-06T05:07:12.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.839+0000 7fc5867fc700 1 -- 172.21.15.98:0/1862581699 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc57c00dd10 con 0x7fc58806edc0 2023-10-06T05:07:12.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.839+0000 7fc5867fc700 1 -- 172.21.15.98:0/1862581699 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc57c0106a0 con 0x7fc58806edc0 2023-10-06T05:07:12.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.839+0000 7fd9110ca700 1 --2- 172.21.15.98:0/2401582762 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8f40053c0 0x7fd8f4005a30 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd90c068fb0 tx=0x7fd9040095f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14b0cd4574263d77 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:12.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.839+0000 7f56fa969700 1 -- 172.21.15.98:0/1680992592 wait complete. 2023-10-06T05:07:12.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.839+0000 7f6bc6a3e700 1 -- 172.21.15.98:0/2786613481 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bc0179cf0 con 0x7f6bc0067800 2023-10-06T05:07:12.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.840+0000 7f6bc6a3e700 1 -- 172.21.15.98:0/2786613481 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bc017a2b0 con 0x7f6bc0067800 2023-10-06T05:07:12.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.839+0000 7fa76cbe4700 1 -- start start 2023-10-06T05:07:12.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.840+0000 7fa76cbe4700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa7680b2350 0x7fa7680c3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.840+0000 7fa76cbe4700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa768161780 0x7fa76815a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.840+0000 7fa76cbe4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa768162330 0x7fa768157de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.841+0000 7fa76cbe4700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7680c4bd0 con 0x7fa768162330 2023-10-06T05:07:12.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.841+0000 7fa76cbe4700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa76816dc60 con 0x7fa7680b2350 2023-10-06T05:07:12.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.841+0000 7fa76cbe4700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7680c4330 con 0x7fa768161780 2023-10-06T05:07:12.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.841+0000 7f6bc6a3e700 1 -- 172.21.15.98:0/2786613481 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bc0064a90 con 0x7f6bc0067800 2023-10-06T05:07:12.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.841+0000 7fa765d9b700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa768161780 0x7fa76815a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.841+0000 7f56fa969700 1 Processor -- start 2023-10-06T05:07:12.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.841+0000 7fa76659c700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa7680b2350 0x7fa7680c3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.841+0000 7fa766d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa768162330 0x7fa768157de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.841+0000 7fa766d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa768162330 0x7fa768157de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45872/0 (socket says 172.21.15.98:45872) 2023-10-06T05:07:12.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.841+0000 7fa766d9d700 1 -- 172.21.15.98:0/1369658387 learned_addr learned my addr 172.21.15.98:0/1369658387 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:12.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.841+0000 7f2246104700 1 Processor -- start 2023-10-06T05:07:12.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.841+0000 7f6ba7fff700 1 -- 172.21.15.98:0/2786613481 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f6bb00070d0 con 0x7f6bc0067800 2023-10-06T05:07:12.886 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.841+0000 7f2246104700 1 -- start start 2023-10-06T05:07:12.886 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.841+0000 7fce727fc700 1 -- 172.21.15.98:0/1083354292 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fce7c0149e0 con 0x7fce84159560 2023-10-06T05:07:12.886 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.841+0000 7fce727fc700 1 --2- 172.21.15.98:0/1083354292 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fce64067000 0x7fce640694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.886 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.841+0000 7fce727fc700 1 -- 172.21.15.98:0/1083354292 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fce7c085a10 con 0x7fce84159560 2023-10-06T05:07:12.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.841+0000 7fa766d9d700 1 -- 172.21.15.98:0/1369658387 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa768161780 msgr2=0x7fa76815a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.841+0000 7fa766d9d700 1 --2- 172.21.15.98:0/1369658387 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa768161780 0x7fa76815a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.841+0000 7fa766d9d700 1 -- 172.21.15.98:0/1369658387 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa7680b2350 msgr2=0x7fa7680c3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.841+0000 7fa766d9d700 1 --2- 172.21.15.98:0/1369658387 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa7680b2350 0x7fa7680c3ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.841+0000 7fa766d9d700 1 -- 172.21.15.98:0/1369658387 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7580094f0 con 0x7fa768162330 2023-10-06T05:07:12.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.841+0000 7fc58dd88700 1 -- 172.21.15.98:0/1862581699 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc58806edc0 msgr2=0x7fc5880c7030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.841+0000 7fc58dd88700 1 --2- 172.21.15.98:0/1862581699 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc58806edc0 0x7fc5880c7030 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc57c00bc50 tx=0x7fc57c00b4f0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.842+0000 7fa765d9b700 1 --2- 172.21.15.98:0/1369658387 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa768161780 0x7fa76815a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:07:12.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.842+0000 7fa76659c700 1 --2- 172.21.15.98:0/1369658387 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa7680b2350 0x7fa7680c3ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:07:12.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.842+0000 7f6ba7fff700 1 --2- 172.21.15.98:0/2786613481 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f6bac066f30 0x7f6bac0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.842+0000 7fc58dd88700 1 -- 172.21.15.98:0/1862581699 shutdown_connections 2023-10-06T05:07:12.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.842+0000 7fc58dd88700 1 --2- 172.21.15.98:0/1862581699 >> v2:172.21.15.98:3300/0 conn(0x7fc588159cc0 0x7fc58806e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.842+0000 7fc58dd88700 1 --2- 172.21.15.98:0/1862581699 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc58806edc0 0x7fc5880c7030 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.842+0000 7fc58dd88700 1 -- 172.21.15.98:0/1862581699 >> 172.21.15.98:0/1862581699 conn(0x7fc588068080 msgr2=0x7fc5880cc0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:12.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.842+0000 7f56fa969700 1 -- start start 2023-10-06T05:07:12.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.842+0000 7fa766d9d700 1 --2- 172.21.15.98:0/1369658387 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa768162330 0x7fa768157de0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fa76000c8e0 tx=0x7fa7600016c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:12.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.842+0000 7f6ba7fff700 1 -- 172.21.15.98:0/2786613481 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f6bb0083d70 con 0x7f6bc0067800 2023-10-06T05:07:12.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.842+0000 7f6bbf7fe700 1 --2- 172.21.15.98:0/2786613481 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f6bac066f30 0x7f6bac0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.842+0000 7f13bab17700 1 -- start start 2023-10-06T05:07:12.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.842+0000 7fc58dd88700 1 -- 172.21.15.98:0/1862581699 shutdown_connections 2023-10-06T05:07:12.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.842+0000 7f6bbf7fe700 1 --2- 172.21.15.98:0/2786613481 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f6bac066f30 0x7f6bac0693e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6bc0160720 tx=0x7f6bb4017000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:12.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.842+0000 7fc58dd88700 1 -- 172.21.15.98:0/1862581699 wait complete. 2023-10-06T05:07:12.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.843+0000 7f56fa969700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f56f4067800 0x7f56f40b1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.843+0000 7f56fa969700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56f406c970 0x7f56f4151740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.843+0000 7f56fa969700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f56f40ca9f0 0x7f56f415e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.843+0000 7f56fa969700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56f4151c80 con 0x7f56f406c970 2023-10-06T05:07:12.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.843+0000 7f56fa969700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56f40c5bf0 con 0x7f56f4067800 2023-10-06T05:07:12.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.843+0000 7f56fa969700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56f40c5ed0 con 0x7f56f40ca9f0 2023-10-06T05:07:12.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.842+0000 7f2246104700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f22400adb60 0x7f22400b1080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.842+0000 7f2246104700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f22400ca370 0x7f22400b19a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.842+0000 7f2246104700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22400ce8a0 0x7f22400ca990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.842+0000 7f2246104700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22400c1d10 con 0x7f22400ce8a0 2023-10-06T05:07:12.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.842+0000 7f2246104700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22400b1ee0 con 0x7f22400ca370 2023-10-06T05:07:12.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.842+0000 7f2246104700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22400c1f60 con 0x7f22400adb60 2023-10-06T05:07:12.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.842+0000 7f223effd700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f22400ca370 0x7f22400b19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.842+0000 7f223f7fe700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f22400adb60 0x7f22400b1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.842+0000 7f223effd700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f22400ca370 0x7f22400b19a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.98:51364/0 (socket says 172.21.15.98:51364) 2023-10-06T05:07:12.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.843+0000 7f56f37fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56f406c970 0x7f56f4151740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.842+0000 7f223effd700 1 -- 172.21.15.98:0/2257644142 learned_addr learned my addr 172.21.15.98:0/2257644142 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:12.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.843+0000 7f223effd700 1 -- 172.21.15.98:0/2257644142 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f22400adb60 msgr2=0x7f22400b1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.843+0000 7f223effd700 1 --2- 172.21.15.98:0/2257644142 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f22400adb60 0x7f22400b1080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.843+0000 7f223effd700 1 -- 172.21.15.98:0/2257644142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22400ce8a0 msgr2=0x7f22400ca990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-06T05:07:12.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.843+0000 7f223effd700 1 --2- 172.21.15.98:0/2257644142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22400ce8a0 0x7f22400ca990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.843+0000 7f223effd700 1 -- 172.21.15.98:0/2257644142 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2238009a20 con 0x7f22400ca370 2023-10-06T05:07:12.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.843+0000 7f56f37fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56f406c970 0x7f56f4151740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45878/0 (socket says 172.21.15.98:45878) 2023-10-06T05:07:12.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.843+0000 7f223f7fe700 1 --2- 172.21.15.98:0/2257644142 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f22400adb60 0x7f22400b1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T05:07:12.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.843+0000 7f223effd700 1 --2- 172.21.15.98:0/2257644142 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f22400ca370 0x7f22400b19a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f223001d540 tx=0x7f2230007f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:12.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.843+0000 7f56f37fe700 1 -- 172.21.15.98:0/1089095321 learned_addr learned my addr 172.21.15.98:0/1089095321 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:12.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.843+0000 7f13bab17700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13b4067800 0x7f13b414f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.843+0000 7f13bab17700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f13b406c970 0x7f13b40c7050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.843+0000 7f13bab17700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f13b40ca9f0 0x7f13b4167c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.843+0000 7f13bab17700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13b4153cb0 con 0x7f13b4067800 2023-10-06T05:07:12.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.843+0000 7f13bab17700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13b4154830 con 0x7f13b406c970 2023-10-06T05:07:12.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.843+0000 7f13bab17700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13b4153f00 con 0x7f13b40ca9f0 2023-10-06T05:07:12.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.843+0000 7f13b88b3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13b4067800 0x7f13b414f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.843+0000 7f13b88b3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13b4067800 0x7f13b414f2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45884/0 (socket says 172.21.15.98:45884) 2023-10-06T05:07:12.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.843+0000 7f13b88b3700 1 -- 172.21.15.98:0/1066719863 learned_addr learned my addr 172.21.15.98:0/1066719863 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:12.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.844+0000 7f56f37fe700 1 -- 172.21.15.98:0/1089095321 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f56f40ca9f0 msgr2=0x7f56f415e760 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-06T05:07:12.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.844+0000 7f56f37fe700 1 --2- 172.21.15.98:0/1089095321 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f56f40ca9f0 0x7f56f415e760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.844+0000 7f56f37fe700 1 -- 172.21.15.98:0/1089095321 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f56f4067800 msgr2=0x7f56f40b1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.844+0000 7f56f37fe700 1 --2- 172.21.15.98:0/1089095321 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f56f4067800 0x7f56f40b1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.844+0000 7f56f37fe700 1 -- 172.21.15.98:0/1089095321 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56e4009420 con 0x7f56f406c970 2023-10-06T05:07:12.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.844+0000 7f13b90b4700 1 --2- 172.21.15.98:0/1066719863 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f13b40ca9f0 0x7f13b4167c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.844+0000 7f13b3fff700 1 --2- 172.21.15.98:0/1066719863 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f13b406c970 0x7f13b40c7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.844+0000 7f13b88b3700 1 -- 172.21.15.98:0/1066719863 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f13b40ca9f0 msgr2=0x7f13b4167c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.844+0000 7f13b88b3700 1 --2- 172.21.15.98:0/1066719863 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f13b40ca9f0 0x7f13b4167c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.844+0000 7f13b88b3700 1 -- 172.21.15.98:0/1066719863 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f13b406c970 msgr2=0x7f13b40c7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.844+0000 7f13b88b3700 1 --2- 172.21.15.98:0/1066719863 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f13b406c970 0x7f13b40c7050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.844+0000 7f56f37fe700 1 --2- 172.21.15.98:0/1089095321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56f406c970 0x7f56f4151740 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f56e800e930 tx=0x7f56e80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:12.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.844+0000 7f13b88b3700 1 -- 172.21.15.98:0/1066719863 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13a0009420 con 0x7f13b4067800 2023-10-06T05:07:12.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.844+0000 7f13b90b4700 1 --2- 172.21.15.98:0/1066719863 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f13b40ca9f0 0x7f13b4167c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:07:12.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.844+0000 7f13b3fff700 1 --2- 172.21.15.98:0/1066719863 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f13b406c970 0x7f13b40c7050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:07:12.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.844+0000 7f13b88b3700 1 --2- 172.21.15.98:0/1066719863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13b4067800 0x7f13b414f2c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f13a8001eb0 tx=0x7f13a8018f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:12.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.844+0000 7fc58dd88700 1 Processor -- start 2023-10-06T05:07:12.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.844+0000 7fc58dd88700 1 -- start start 2023-10-06T05:07:12.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.845+0000 7fce8259c700 1 --2- 172.21.15.98:0/1083354292 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fce64067000 0x7fce640694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.845+0000 7fc58dd88700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc58806edc0 0x7fc5881689c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.845+0000 7fce8259c700 1 --2- 172.21.15.98:0/1083354292 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fce64067000 0x7fce640694b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fce74000ef0 tx=0x7fce74009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:12.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.845+0000 7fc58dd88700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc5880c7570 0x7fc58815b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.845+0000 7fc58dd88700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc588159cc0 0x7fc58815b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.845+0000 7fc58dd88700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5880c4ca0 con 0x7fc58806edc0 2023-10-06T05:07:12.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.845+0000 7fc58dd88700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5880c4090 con 0x7fc5880c7570 2023-10-06T05:07:12.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.845+0000 7fc58dd88700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5880c4370 con 0x7fc588159cc0 2023-10-06T05:07:12.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.845+0000 7fc5877fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc58806edc0 0x7fc5881689c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.845+0000 7fc5877fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc58806edc0 0x7fc5881689c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45898/0 (socket says 172.21.15.98:45898) 2023-10-06T05:07:12.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.845+0000 7fc5877fe700 1 -- 172.21.15.98:0/1021791739 learned_addr learned my addr 172.21.15.98:0/1021791739 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:12.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.845+0000 7fc587fff700 1 --2- 172.21.15.98:0/1021791739 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc588159cc0 0x7fc58815b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.845+0000 7fc586ffd700 1 --2- 172.21.15.98:0/1021791739 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc5880c7570 0x7fc58815b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.845+0000 7fc5877fe700 1 -- 172.21.15.98:0/1021791739 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc588159cc0 msgr2=0x7fc58815b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.845+0000 7fc5877fe700 1 --2- 172.21.15.98:0/1021791739 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc588159cc0 0x7fc58815b7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.845+0000 7fc5877fe700 1 -- 172.21.15.98:0/1021791739 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc5880c7570 msgr2=0x7fc58815b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.845+0000 7fc5877fe700 1 --2- 172.21.15.98:0/1021791739 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc5880c7570 0x7fc58815b270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.845+0000 7fc5877fe700 1 -- 172.21.15.98:0/1021791739 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc57c00b790 con 0x7fc58806edc0 2023-10-06T05:07:12.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.845+0000 7fc5877fe700 1 --2- 172.21.15.98:0/1021791739 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc58806edc0 0x7fc5881689c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc57800b810 tx=0x7fc5780091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:12.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.846+0000 7fd90b7fe700 1 -- 172.21.15.98:0/2401582762 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd90400bc20 con 0x7fd8f40053c0 2023-10-06T05:07:12.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.846+0000 7fa7567fc700 1 -- 172.21.15.98:0/1369658387 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa76000ce80 con 0x7fa768162330 2023-10-06T05:07:12.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.846+0000 7fc56f7fe700 1 -- 172.21.15.98:0/1021791739 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc57800f510 con 0x7fc58806edc0 2023-10-06T05:07:12.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.846+0000 7fc58dd88700 1 -- 172.21.15.98:0/1021791739 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5880c4650 con 0x7fc58806edc0 2023-10-06T05:07:12.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.846+0000 7f13b0ff9700 1 -- 172.21.15.98:0/1066719863 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a8016b40 con 0x7f13b4067800 2023-10-06T05:07:12.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.846+0000 7fc58dd88700 1 -- 172.21.15.98:0/1021791739 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc588168f00 con 0x7fc58806edc0 2023-10-06T05:07:12.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.846+0000 7f13bab17700 1 -- 172.21.15.98:0/1066719863 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13b41541e0 con 0x7f13b4067800 2023-10-06T05:07:12.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.846+0000 7f13bab17700 1 -- 172.21.15.98:0/1066719863 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13b414f800 con 0x7f13b4067800 2023-10-06T05:07:12.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.846+0000 7f56dbfff700 1 -- 172.21.15.98:0/1089095321 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56e800eed0 con 0x7f56f406c970 2023-10-06T05:07:12.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.846+0000 7fa76cbe4700 1 -- 172.21.15.98:0/1369658387 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7680c4670 con 0x7fa768162330 2023-10-06T05:07:12.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.846+0000 7fa76cbe4700 1 -- 172.21.15.98:0/1369658387 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7680c4020 con 0x7fa768162330 2023-10-06T05:07:12.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.846+0000 7f56fa969700 1 -- 172.21.15.98:0/1089095321 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56f40b0c30 con 0x7f56f406c970 2023-10-06T05:07:12.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.846+0000 7f56fa969700 1 -- 172.21.15.98:0/1089095321 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56f40b11f0 con 0x7f56f406c970 2023-10-06T05:07:12.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.846+0000 7fce727fc700 1 -- 172.21.15.98:0/1083354292 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fce7c052990 con 0x7fce84159560 2023-10-06T05:07:12.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.847+0000 7f22277fe700 1 -- 172.21.15.98:0/2257644142 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2230005b30 con 0x7f22400ca370 2023-10-06T05:07:12.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.847+0000 7f22277fe700 1 -- 172.21.15.98:0/2257644142 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2230005c90 con 0x7f22400ca370 2023-10-06T05:07:12.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.847+0000 7f2246104700 1 -- 172.21.15.98:0/2257644142 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22400c2240 con 0x7f22400ca370 2023-10-06T05:07:12.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.847+0000 7fc58dd88700 1 -- 172.21.15.98:0/1021791739 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5880c4f80 con 0x7fc58806edc0 2023-10-06T05:07:12.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.847+0000 7f13b0ff9700 1 -- 172.21.15.98:0/1066719863 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13a8016cc0 con 0x7f13b4067800 2023-10-06T05:07:12.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.847+0000 7f13b0ff9700 1 -- 172.21.15.98:0/1066719863 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a8018340 con 0x7f13b4067800 2023-10-06T05:07:12.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.848+0000 7f6ba7fff700 1 -- 172.21.15.98:0/2786613481 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6bb004b5a0 con 0x7f6bc0067800 2023-10-06T05:07:12.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.848+0000 7fd90b7fe700 1 -- 172.21.15.98:0/2401582762 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd90400bd80 con 0x7fd8f40053c0 2023-10-06T05:07:12.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.848+0000 7fd90b7fe700 1 -- 172.21.15.98:0/2401582762 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd904010a90 con 0x7fd8f40053c0 2023-10-06T05:07:12.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.848+0000 7f56fa969700 1 -- 172.21.15.98:0/1089095321 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56f40b1400 con 0x7f56f406c970 2023-10-06T05:07:12.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.847+0000 7fa7567fc700 1 -- 172.21.15.98:0/1369658387 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa76000a7e0 con 0x7fa768162330 2023-10-06T05:07:12.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.847+0000 7fa7567fc700 1 -- 172.21.15.98:0/1369658387 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa76000aba0 con 0x7fa768162330 2023-10-06T05:07:12.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.848+0000 7fa76cbe4700 1 -- 172.21.15.98:0/1369658387 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa76816e0b0 con 0x7fa768162330 2023-10-06T05:07:12.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.847+0000 7f22277fe700 1 -- 172.21.15.98:0/2257644142 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2230007300 con 0x7f22400ca370 2023-10-06T05:07:12.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.847+0000 7f2246104700 1 -- 172.21.15.98:0/2257644142 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22400c2800 con 0x7f22400ca370 2023-10-06T05:07:12.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.848+0000 7f2246104700 1 -- 172.21.15.98:0/2257644142 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22400c29f0 con 0x7f22400ca370 2023-10-06T05:07:12.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.849+0000 7fd912b2d700 1 -- 172.21.15.98:0/2401582762 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8f40053c0 msgr2=0x7fd8f4005a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.849+0000 7fd912b2d700 1 --2- 172.21.15.98:0/2401582762 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8f40053c0 0x7fd8f4005a30 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd90c068fb0 tx=0x7fd9040095f0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.849+0000 7fd9110ca700 1 -- 172.21.15.98:0/2401582762 reap_dead start 2023-10-06T05:07:12.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.849+0000 7f22277fe700 1 -- 172.21.15.98:0/2257644142 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f2230023410 con 0x7f22400ca370 2023-10-06T05:07:12.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.849+0000 7f22277fe700 1 --2- 172.21.15.98:0/2257644142 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f22280673b0 0x7f2228069860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.849+0000 7f22277fe700 1 -- 172.21.15.98:0/2257644142 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f2230094d70 con 0x7f22400ca370 2023-10-06T05:07:12.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.849+0000 7f223f7fe700 1 --2- 172.21.15.98:0/2257644142 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f22280673b0 0x7f2228069860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.850+0000 7f223f7fe700 1 --2- 172.21.15.98:0/2257644142 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f22280673b0 0x7f2228069860 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2234009f20 tx=0x7f22340090f0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:12.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.851+0000 7f13b0ff9700 1 -- 172.21.15.98:0/1066719863 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f13a802c400 con 0x7f13b4067800 2023-10-06T05:07:12.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.851+0000 7fa7567fc700 1 -- 172.21.15.98:0/1369658387 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fa760003030 con 0x7fa768162330 2023-10-06T05:07:12.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.851+0000 7f13b0ff9700 1 --2- 172.21.15.98:0/1066719863 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f1398066f30 0x7f13980693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.852+0000 7fa7567fc700 1 --2- 172.21.15.98:0/1369658387 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fa748066f30 0x7fa7480693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.852+0000 7fa7567fc700 1 -- 172.21.15.98:0/1369658387 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fa760084b20 con 0x7fa768162330 2023-10-06T05:07:12.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.852+0000 7fa76659c700 1 --2- 172.21.15.98:0/1369658387 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fa748066f30 0x7fa7480693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.852+0000 7fa76659c700 1 --2- 172.21.15.98:0/1369658387 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fa748066f30 0x7fa7480693e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa7580099b0 tx=0x7fa758017000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:12.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.853+0000 7f56dbfff700 1 -- 172.21.15.98:0/1089095321 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56e800ca60 con 0x7f56f406c970 2023-10-06T05:07:12.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.853+0000 7f56dbfff700 1 -- 172.21.15.98:0/1089095321 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56e800cbd0 con 0x7f56f406c970 2023-10-06T05:07:12.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.852+0000 7f13b0ff9700 1 -- 172.21.15.98:0/1066719863 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f13a80954d0 con 0x7f13b4067800 2023-10-06T05:07:12.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.852+0000 7f13b3fff700 1 --2- 172.21.15.98:0/1066719863 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f1398066f30 0x7f13980693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.852+0000 7f13b3fff700 1 --2- 172.21.15.98:0/1066719863 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f1398066f30 0x7f13980693e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f13a00093f0 tx=0x7f13a0016000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:12.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.853+0000 7f56dbfff700 1 -- 172.21.15.98:0/1089095321 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f56e8005a00 con 0x7f56f406c970 2023-10-06T05:07:12.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.853+0000 7f13bab17700 1 -- 172.21.15.98:0/1066719863 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13940052b0 con 0x7f13b4067800 2023-10-06T05:07:12.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.853+0000 7f56dbfff700 1 --2- 172.21.15.98:0/1089095321 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f56e0066f30 0x7f56e00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.853+0000 7f56f3fff700 1 --2- 172.21.15.98:0/1089095321 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f56e0066f30 0x7f56e00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.853+0000 7f56dbfff700 1 -- 172.21.15.98:0/1089095321 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f56e8084940 con 0x7f56f406c970 2023-10-06T05:07:12.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.854+0000 7f22277fe700 1 -- 172.21.15.98:0/2257644142 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f223005c5a0 con 0x7f22400ca370 2023-10-06T05:07:12.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.854+0000 7f56f3fff700 1 --2- 172.21.15.98:0/1089095321 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f56e0066f30 0x7f56e00693e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f56ec001560 tx=0x7f56ec0016c0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:12.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.855+0000 7fd912b2d700 1 -- 172.21.15.98:0/2401582762 shutdown_connections 2023-10-06T05:07:12.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.855+0000 7fd912b2d700 1 -- 172.21.15.98:0/2401582762 >> 172.21.15.98:0/2401582762 conn(0x7fd90c06de40 msgr2=0x7fd90c06e240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:12.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.855+0000 7fd912b2d700 1 -- 172.21.15.98:0/2401582762 shutdown_connections 2023-10-06T05:07:12.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.855+0000 7fd912b2d700 1 -- 172.21.15.98:0/2401582762 wait complete. 2023-10-06T05:07:12.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.856+0000 7fd912b2d700 1 Processor -- start 2023-10-06T05:07:12.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.857+0000 7fc56f7fe700 1 -- 172.21.15.98:0/1021791739 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc57800f670 con 0x7fc58806edc0 2023-10-06T05:07:12.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.857+0000 7fc56f7fe700 1 -- 172.21.15.98:0/1021791739 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc57800fa30 con 0x7fc58806edc0 2023-10-06T05:07:12.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.857+0000 7fc56f7fe700 1 -- 172.21.15.98:0/1021791739 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fc578024430 con 0x7fc58806edc0 2023-10-06T05:07:12.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.857+0000 7fc56f7fe700 1 --2- 172.21.15.98:0/1021791739 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fc574067030 0x7fc5740694e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.857+0000 7fc56f7fe700 1 -- 172.21.15.98:0/1021791739 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fc578084ef0 con 0x7fc58806edc0 2023-10-06T05:07:12.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.857+0000 7fc56f7fe700 1 -- 172.21.15.98:0/1021791739 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc578085330 con 0x7fc58806edc0 2023-10-06T05:07:12.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.857+0000 7fd912b2d700 1 -- start start 2023-10-06T05:07:12.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.858+0000 7fd912b2d700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd8f40053c0 0x7fd90c0c1920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.858+0000 7fd912b2d700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd90c0c3ae0 0x7fd90c0c8110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.858+0000 7fd912b2d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd90c153ce0 0x7fd90c154130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.858+0000 7fd912b2d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd90c0c3690 con 0x7fd90c153ce0 2023-10-06T05:07:12.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.858+0000 7fd912b2d700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd90c0c3f50 con 0x7fd90c0c3ae0 2023-10-06T05:07:12.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.858+0000 7fd912b2d700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd90c0c2b10 con 0x7fd8f40053c0 2023-10-06T05:07:12.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.858+0000 7fd9110ca700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd90c153ce0 0x7fd90c154130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.858+0000 7fd9110ca700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd90c153ce0 0x7fd90c154130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45910/0 (socket says 172.21.15.98:45910) 2023-10-06T05:07:12.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.858+0000 7fd9110ca700 1 -- 172.21.15.98:0/1432344714 learned_addr learned my addr 172.21.15.98:0/1432344714 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:12.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.858+0000 7fd9108c9700 1 --2- 172.21.15.98:0/1432344714 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd8f40053c0 0x7fd90c0c1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.858+0000 7fd90bfff700 1 --2- 172.21.15.98:0/1432344714 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd90c0c3ae0 0x7fd90c0c8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.859+0000 7fc586ffd700 1 --2- 172.21.15.98:0/1021791739 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fc574067030 0x7fc5740694e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.859+0000 7fd9108c9700 1 -- 172.21.15.98:0/1432344714 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd90c0c3ae0 msgr2=0x7fd90c0c8110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.859+0000 7fd9108c9700 1 --2- 172.21.15.98:0/1432344714 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd90c0c3ae0 0x7fd90c0c8110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.859+0000 7fd9108c9700 1 -- 172.21.15.98:0/1432344714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd90c153ce0 msgr2=0x7fd90c154130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:12.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.859+0000 7fd9108c9700 1 --2- 172.21.15.98:0/1432344714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd90c153ce0 0x7fd90c154130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:12.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.859+0000 7fd9108c9700 1 -- 172.21.15.98:0/1432344714 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd90c0c1f00 con 0x7fd8f40053c0 2023-10-06T05:07:12.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.859+0000 7fd9108c9700 1 --2- 172.21.15.98:0/1432344714 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd8f40053c0 0x7fd90c0c1920 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd8fc007110 tx=0x7fd8fc001df0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:12.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.859+0000 7fc586ffd700 1 --2- 172.21.15.98:0/1021791739 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fc574067030 0x7fc5740694e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc57c00b760 tx=0x7fc57c01a000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:12.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.860+0000 7fd908ff9700 1 -- 172.21.15.98:0/1432344714 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8fc011590 con 0x7fd8f40053c0 2023-10-06T05:07:12.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.860+0000 7fd908ff9700 1 -- 172.21.15.98:0/1432344714 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8fc0116f0 con 0x7fd8f40053c0 2023-10-06T05:07:12.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.860+0000 7fd908ff9700 1 -- 172.21.15.98:0/1432344714 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8fc011a90 con 0x7fd8f40053c0 2023-10-06T05:07:12.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.861+0000 7fd912b2d700 1 -- 172.21.15.98:0/1432344714 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd90c0c21e0 con 0x7fd8f40053c0 2023-10-06T05:07:12.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.861+0000 7fd912b2d700 1 -- 172.21.15.98:0/1432344714 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd90c0c27a0 con 0x7fd8f40053c0 2023-10-06T05:07:12.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.862+0000 7fa7567fc700 1 -- 172.21.15.98:0/1369658387 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa76816e0b0 con 0x7fa768162330 2023-10-06T05:07:12.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.862+0000 7fd908ff9700 1 -- 172.21.15.98:0/1432344714 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fd8fc01d460 con 0x7fd8f40053c0 2023-10-06T05:07:12.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.863+0000 7fd908ff9700 1 --2- 172.21.15.98:0/1432344714 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fd8f8067000 0x7fd8f80694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:12.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.863+0000 7fd908ff9700 1 -- 172.21.15.98:0/1432344714 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fd8fc084a10 con 0x7fd8f40053c0 2023-10-06T05:07:12.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.863+0000 7fd90bfff700 1 --2- 172.21.15.98:0/1432344714 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fd8f8067000 0x7fd8f80694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:12.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.864+0000 7fd90bfff700 1 --2- 172.21.15.98:0/1432344714 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fd8f8067000 0x7fd8f80694b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd90c0c5030 tx=0x7fd900009060 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:12.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.866+0000 7fd912b2d700 1 -- 172.21.15.98:0/1432344714 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd90c064a90 con 0x7fd8f40053c0 2023-10-06T05:07:12.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.870+0000 7f56dbfff700 1 -- 172.21.15.98:0/1089095321 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f56e8051850 con 0x7f56f406c970 2023-10-06T05:07:12.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.871+0000 7fd908ff9700 1 -- 172.21.15.98:0/1432344714 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd8fc0518b0 con 0x7fd8f40053c0 2023-10-06T05:07:12.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:12.872+0000 7f13b0ff9700 1 -- 172.21.15.98:0/1066719863 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f13a80623a0 con 0x7f13b4067800 2023-10-06T05:07:13.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.226+0000 7fce88daa700 1 -- 172.21.15.98:0/1083354292 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fce84064a90 con 0x7fce84159560 2023-10-06T05:07:13.233 INFO:teuthology.orchestra.run.smithi098.stdout:64424509447 2023-10-06T05:07:13.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.228+0000 7fce727fc700 1 -- 172.21.15.98:0/1083354292 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fce7c009a20 con 0x7fce84159560 2023-10-06T05:07:13.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.238+0000 7fce88daa700 1 -- 172.21.15.98:0/1083354292 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fce64067000 msgr2=0x7fce640694b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:13.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.238+0000 7fce88daa700 1 --2- 172.21.15.98:0/1083354292 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fce64067000 0x7fce640694b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fce74000ef0 tx=0x7fce74009040 comp rx=0 tx=0).stop 2023-10-06T05:07:13.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.238+0000 7fce88daa700 1 -- 172.21.15.98:0/1083354292 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce84159560 msgr2=0x7fce8414f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:13.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.238+0000 7fce88daa700 1 --2- 172.21.15.98:0/1083354292 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce84159560 0x7fce8414f660 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fce7c007f30 tx=0x7fce7c0098e0 comp rx=0 tx=0).stop 2023-10-06T05:07:13.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.240+0000 7fce88daa700 1 -- 172.21.15.98:0/1083354292 shutdown_connections 2023-10-06T05:07:13.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.240+0000 7fce88daa700 1 --2- 172.21.15.98:0/1083354292 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fce64067000 0x7fce640694b0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:13.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.240+0000 7fce88daa700 1 --2- 172.21.15.98:0/1083354292 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce84159560 0x7fce8414f660 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:13.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.240+0000 7fce88daa700 1 --2- 172.21.15.98:0/1083354292 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fce840b0760 0x7fce841569f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:13.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.240+0000 7fce88daa700 1 --2- 172.21.15.98:0/1083354292 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fce840b49a0 0x7fce84153ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:13.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.240+0000 7fce88daa700 1 -- 172.21.15.98:0/1083354292 >> 172.21.15.98:0/1083354292 conn(0x7fce840997f0 msgr2=0x7fce84069890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:13.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.240+0000 7fce88daa700 1 -- 172.21.15.98:0/1083354292 shutdown_connections 2023-10-06T05:07:13.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.240+0000 7fce88daa700 1 -- 172.21.15.98:0/1083354292 wait complete. 2023-10-06T05:07:13.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.243+0000 7fd912b2d700 1 -- 172.21.15.98:0/1432344714 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fd90c04bb40 con 0x7fd8f40053c0 2023-10-06T05:07:13.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.243+0000 7fd908ff9700 1 -- 172.21.15.98:0/1432344714 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd8fc054ed0 con 0x7fd8f40053c0 2023-10-06T05:07:13.246 INFO:teuthology.orchestra.run.smithi098.stdout:64424509448 2023-10-06T05:07:13.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.244+0000 7f6bc6a3e700 1 -- 172.21.15.98:0/2786613481 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f6bc006bc30 con 0x7f6bc0067800 2023-10-06T05:07:13.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.244+0000 7f2246104700 1 -- 172.21.15.98:0/2257644142 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f2240064a90 con 0x7f22400ca370 2023-10-06T05:07:13.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.244+0000 7fc58dd88700 1 -- 172.21.15.98:0/1021791739 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fc588064a90 con 0x7fc58806edc0 2023-10-06T05:07:13.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.245+0000 7f22277fe700 1 -- 172.21.15.98:0/2257644142 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2230028020 con 0x7f22400ca370 2023-10-06T05:07:13.248 INFO:teuthology.orchestra.run.smithi098.stdout:60129542151 2023-10-06T05:07:13.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.245+0000 7fc56f7fe700 1 -- 172.21.15.98:0/1021791739 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc578051d70 con 0x7fc58806edc0 2023-10-06T05:07:13.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.246+0000 7f6ba7fff700 1 -- 172.21.15.98:0/2786613481 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6bb0050b90 con 0x7f6bc0067800 2023-10-06T05:07:13.249 INFO:teuthology.orchestra.run.smithi098.stdout:64424509448 2023-10-06T05:07:13.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.248+0000 7fd912b2d700 1 -- 172.21.15.98:0/1432344714 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fd8f8067000 msgr2=0x7fd8f80694b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:13.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.248+0000 7fd912b2d700 1 --2- 172.21.15.98:0/1432344714 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fd8f8067000 0x7fd8f80694b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd90c0c5030 tx=0x7fd900009060 comp rx=0 tx=0).stop 2023-10-06T05:07:13.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.248+0000 7fd912b2d700 1 -- 172.21.15.98:0/1432344714 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd8f40053c0 msgr2=0x7fd90c0c1920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:13.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.248+0000 7fd912b2d700 1 --2- 172.21.15.98:0/1432344714 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd8f40053c0 0x7fd90c0c1920 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd8fc007110 tx=0x7fd8fc001df0 comp rx=0 tx=0).stop 2023-10-06T05:07:13.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.248+0000 7f2246104700 1 -- 172.21.15.98:0/2257644142 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f22280673b0 msgr2=0x7f2228069860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:13.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.248+0000 7f2246104700 1 --2- 172.21.15.98:0/2257644142 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f22280673b0 0x7f2228069860 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2234009f20 tx=0x7f22340090f0 comp rx=0 tx=0).stop 2023-10-06T05:07:13.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.248+0000 7f2246104700 1 -- 172.21.15.98:0/2257644142 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f22400ca370 msgr2=0x7f22400b19a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:13.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.248+0000 7f2246104700 1 --2- 172.21.15.98:0/2257644142 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f22400ca370 0x7f22400b19a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f223001d540 tx=0x7f2230007f90 comp rx=0 tx=0).stop 2023-10-06T05:07:13.251 INFO:teuthology.orchestra.run.smithi098.stdout:55834574856 2023-10-06T05:07:13.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.248+0000 7fd912b2d700 1 -- 172.21.15.98:0/1432344714 shutdown_connections 2023-10-06T05:07:13.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.248+0000 7fd912b2d700 1 --2- 172.21.15.98:0/1432344714 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fd8f8067000 0x7fd8f80694b0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:13.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.248+0000 7fd912b2d700 1 --2- 172.21.15.98:0/1432344714 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd90c153ce0 0x7fd90c154130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:13.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.248+0000 7fd912b2d700 1 --2- 172.21.15.98:0/1432344714 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd8f40053c0 0x7fd90c0c1920 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:13.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.248+0000 7fd912b2d700 1 --2- 172.21.15.98:0/1432344714 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd90c0c3ae0 0x7fd90c0c8110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:13.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.248+0000 7fd912b2d700 1 -- 172.21.15.98:0/1432344714 >> 172.21.15.98:0/1432344714 conn(0x7fd90c06de40 msgr2=0x7fd90c0c9530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:13.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.248+0000 7f2246104700 1 -- 172.21.15.98:0/2257644142 shutdown_connections 2023-10-06T05:07:13.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.248+0000 7f2246104700 1 --2- 172.21.15.98:0/2257644142 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f22280673b0 0x7f2228069860 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:13.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.248+0000 7f2246104700 1 --2- 172.21.15.98:0/2257644142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f22400ce8a0 0x7f22400ca990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:13.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.248+0000 7f2246104700 1 --2- 172.21.15.98:0/2257644142 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f22400adb60 0x7f22400b1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:13.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.248+0000 7f2246104700 1 --2- 172.21.15.98:0/2257644142 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f22400ca370 0x7f22400b19a0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:13.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.248+0000 7f2246104700 1 -- 172.21.15.98:0/2257644142 >> 172.21.15.98:0/2257644142 conn(0x7f22400707d0 msgr2=0x7f22400cd630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:13.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.248+0000 7f6bc6a3e700 1 -- 172.21.15.98:0/2786613481 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f6bac066f30 msgr2=0x7f6bac0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:13.254 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.248+0000 7f6bc6a3e700 1 --2- 172.21.15.98:0/2786613481 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f6bac066f30 0x7f6bac0693e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6bc0160720 tx=0x7f6bb4017000 comp rx=0 tx=0).stop 2023-10-06T05:07:13.254 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.248+0000 7f6bc6a3e700 1 -- 172.21.15.98:0/2786613481 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6bc0067800 msgr2=0x7f6bc01283b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:13.254 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.248+0000 7f6bc6a3e700 1 --2- 172.21.15.98:0/2786613481 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6bc0067800 0x7f6bc01283b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6bb000c7c0 tx=0x7f6bb0007f70 comp rx=0 tx=0).stop 2023-10-06T05:07:13.254 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.249+0000 7fd912b2d700 1 -- 172.21.15.98:0/1432344714 shutdown_connections 2023-10-06T05:07:13.255 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.249+0000 7f2246104700 1 -- 172.21.15.98:0/2257644142 shutdown_connections 2023-10-06T05:07:13.255 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.249+0000 7f2246104700 1 -- 172.21.15.98:0/2257644142 wait complete. 2023-10-06T05:07:13.255 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.249+0000 7fd912b2d700 1 -- 172.21.15.98:0/1432344714 wait complete. 2023-10-06T05:07:13.255 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.249+0000 7f6bc6a3e700 1 -- 172.21.15.98:0/2786613481 shutdown_connections 2023-10-06T05:07:13.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.249+0000 7f6bc6a3e700 1 --2- 172.21.15.98:0/2786613481 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6bc00ca9f0 0x7f6bc00c3dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:13.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.249+0000 7f6bc6a3e700 1 --2- 172.21.15.98:0/2786613481 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6bc0067800 0x7f6bc01283b0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:13.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.249+0000 7f6bc6a3e700 1 --2- 172.21.15.98:0/2786613481 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f6bac066f30 0x7f6bac0693e0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:13.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.249+0000 7f6bc6a3e700 1 --2- 172.21.15.98:0/2786613481 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6bc006c970 0x7f6bc00c3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:13.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.249+0000 7f6bc6a3e700 1 -- 172.21.15.98:0/2786613481 >> 172.21.15.98:0/2786613481 conn(0x7f6bc00cdeb0 msgr2=0x7f6bc01611d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:13.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.249+0000 7f13bab17700 1 -- 172.21.15.98:0/1066719863 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f1394005830 con 0x7f13b4067800 2023-10-06T05:07:13.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.249+0000 7f6bc6a3e700 1 -- 172.21.15.98:0/2786613481 shutdown_connections 2023-10-06T05:07:13.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.249+0000 7f6bc6a3e700 1 -- 172.21.15.98:0/2786613481 wait complete. 2023-10-06T05:07:13.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.250+0000 7f13b0ff9700 1 -- 172.21.15.98:0/1066719863 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f13a80659c0 con 0x7f13b4067800 2023-10-06T05:07:13.257 INFO:teuthology.orchestra.run.smithi098.stdout:64424509447 2023-10-06T05:07:13.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.257+0000 7fa76cbe4700 1 -- 172.21.15.98:0/1369658387 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fa76816e0b0 con 0x7fa768162330 2023-10-06T05:07:13.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.257+0000 7fc58dd88700 1 -- 172.21.15.98:0/1021791739 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fc574067030 msgr2=0x7fc5740694e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:13.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.257+0000 7fc58dd88700 1 --2- 172.21.15.98:0/1021791739 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fc574067030 0x7fc5740694e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc57c00b760 tx=0x7fc57c01a000 comp rx=0 tx=0).stop 2023-10-06T05:07:13.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.257+0000 7fc58dd88700 1 -- 172.21.15.98:0/1021791739 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc58806edc0 msgr2=0x7fc5881689c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:13.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.257+0000 7fc58dd88700 1 --2- 172.21.15.98:0/1021791739 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc58806edc0 0x7fc5881689c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc57800b810 tx=0x7fc5780091d0 comp rx=0 tx=0).stop 2023-10-06T05:07:13.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.258+0000 7fc58dd88700 1 -- 172.21.15.98:0/1021791739 shutdown_connections 2023-10-06T05:07:13.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.258+0000 7fc58dd88700 1 --2- 172.21.15.98:0/1021791739 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc588159cc0 0x7fc58815b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:13.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.258+0000 7fc58dd88700 1 --2- 172.21.15.98:0/1021791739 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fc574067030 0x7fc5740694e0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:13.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.258+0000 7fc58dd88700 1 --2- 172.21.15.98:0/1021791739 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc58806edc0 0x7fc5881689c0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:13.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.258+0000 7fc58dd88700 1 --2- 172.21.15.98:0/1021791739 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc5880c7570 0x7fc58815b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:13.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.258+0000 7fc58dd88700 1 -- 172.21.15.98:0/1021791739 >> 172.21.15.98:0/1021791739 conn(0x7fc588068080 msgr2=0x7fc5880cb3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:13.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.258+0000 7fa7567fc700 1 -- 172.21.15.98:0/1369658387 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa76816e0b0 con 0x7fa768162330 2023-10-06T05:07:13.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.258+0000 7f13bab17700 1 -- 172.21.15.98:0/1066719863 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f1398066f30 msgr2=0x7f13980693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:13.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.258+0000 7f13bab17700 1 --2- 172.21.15.98:0/1066719863 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f1398066f30 0x7f13980693e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f13a00093f0 tx=0x7f13a0016000 comp rx=0 tx=0).stop 2023-10-06T05:07:13.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.258+0000 7f13bab17700 1 -- 172.21.15.98:0/1066719863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13b4067800 msgr2=0x7f13b414f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:13.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.258+0000 7f13bab17700 1 --2- 172.21.15.98:0/1066719863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13b4067800 0x7f13b414f2c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f13a8001eb0 tx=0x7f13a8018f40 comp rx=0 tx=0).stop 2023-10-06T05:07:13.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.258+0000 7f13bab17700 1 -- 172.21.15.98:0/1066719863 shutdown_connections 2023-10-06T05:07:13.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.258+0000 7f13bab17700 1 --2- 172.21.15.98:0/1066719863 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f13b40ca9f0 0x7f13b4167c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:13.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.258+0000 7f13bab17700 1 --2- 172.21.15.98:0/1066719863 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f1398066f30 0x7f13980693e0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:13.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.258+0000 7f13bab17700 1 --2- 172.21.15.98:0/1066719863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13b4067800 0x7f13b414f2c0 secure :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f13a8001eb0 tx=0x7f13a8018f40 comp rx=0 tx=0).stop 2023-10-06T05:07:13.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.258+0000 7f13bab17700 1 --2- 172.21.15.98:0/1066719863 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f13b406c970 0x7f13b40c7050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:13.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.258+0000 7f13bab17700 1 -- 172.21.15.98:0/1066719863 >> 172.21.15.98:0/1066719863 conn(0x7f13b40cdeb0 msgr2=0x7f13b40c96f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:13.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.258+0000 7f13bab17700 1 -- 172.21.15.98:0/1066719863 shutdown_connections 2023-10-06T05:07:13.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.258+0000 7f13bab17700 1 -- 172.21.15.98:0/1066719863 wait complete. 2023-10-06T05:07:13.263 INFO:teuthology.orchestra.run.smithi098.stdout:60129542152 2023-10-06T05:07:13.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.261+0000 7fc58dd88700 1 -- 172.21.15.98:0/1021791739 shutdown_connections 2023-10-06T05:07:13.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.262+0000 7fc58dd88700 1 -- 172.21.15.98:0/1021791739 wait complete. 2023-10-06T05:07:13.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.262+0000 7fa76cbe4700 1 -- 172.21.15.98:0/1369658387 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fa748066f30 msgr2=0x7fa7480693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:13.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.262+0000 7fa76cbe4700 1 --2- 172.21.15.98:0/1369658387 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fa748066f30 0x7fa7480693e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa7580099b0 tx=0x7fa758017000 comp rx=0 tx=0).stop 2023-10-06T05:07:13.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.262+0000 7fa76cbe4700 1 -- 172.21.15.98:0/1369658387 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa768162330 msgr2=0x7fa768157de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:13.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.262+0000 7fa76cbe4700 1 --2- 172.21.15.98:0/1369658387 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa768162330 0x7fa768157de0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fa76000c8e0 tx=0x7fa7600016c0 comp rx=0 tx=0).stop 2023-10-06T05:07:13.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.263+0000 7fa76cbe4700 1 -- 172.21.15.98:0/1369658387 shutdown_connections 2023-10-06T05:07:13.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.263+0000 7fa76cbe4700 1 --2- 172.21.15.98:0/1369658387 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fa748066f30 0x7fa7480693e0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:13.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.263+0000 7fa76cbe4700 1 --2- 172.21.15.98:0/1369658387 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa768162330 0x7fa768157de0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:13.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.263+0000 7fa76cbe4700 1 --2- 172.21.15.98:0/1369658387 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa7680b2350 0x7fa7680c3ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:13.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.263+0000 7fa76cbe4700 1 --2- 172.21.15.98:0/1369658387 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa768161780 0x7fa76815a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:13.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.263+0000 7fa76cbe4700 1 -- 172.21.15.98:0/1369658387 >> 172.21.15.98:0/1369658387 conn(0x7fa7680ccf30 msgr2=0x7fa7680b4f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:13.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.263+0000 7fa76cbe4700 1 -- 172.21.15.98:0/1369658387 shutdown_connections 2023-10-06T05:07:13.266 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509448 got 64424509447 for osd.3 2023-10-06T05:07:13.267 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.264+0000 7fa76cbe4700 1 -- 172.21.15.98:0/1369658387 wait complete. 2023-10-06T05:07:13.269 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509449 got 64424509448 for osd.6 2023-10-06T05:07:13.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.268+0000 7f56fa969700 1 -- 172.21.15.98:0/1089095321 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f56f4064a90 con 0x7f56f406c970 2023-10-06T05:07:13.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.269+0000 7f56dbfff700 1 -- 172.21.15.98:0/1089095321 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f56e8054e70 con 0x7f56f406c970 2023-10-06T05:07:13.272 INFO:teuthology.orchestra.run.smithi098.stdout:64424509448 2023-10-06T05:07:13.274 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509449 got 64424509448 for osd.5 2023-10-06T05:07:13.275 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542151 for osd.1 2023-10-06T05:07:13.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.273+0000 7f56fa969700 1 -- 172.21.15.98:0/1089095321 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f56e0066f30 msgr2=0x7f56e00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:13.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.273+0000 7f56fa969700 1 --2- 172.21.15.98:0/1089095321 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f56e0066f30 0x7f56e00693e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f56ec001560 tx=0x7f56ec0016c0 comp rx=0 tx=0).stop 2023-10-06T05:07:13.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.273+0000 7f56fa969700 1 -- 172.21.15.98:0/1089095321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56f406c970 msgr2=0x7f56f4151740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:13.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.273+0000 7f56fa969700 1 --2- 172.21.15.98:0/1089095321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56f406c970 0x7f56f4151740 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f56e800e930 tx=0x7f56e80091a0 comp rx=0 tx=0).stop 2023-10-06T05:07:13.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.273+0000 7f56fa969700 1 -- 172.21.15.98:0/1089095321 shutdown_connections 2023-10-06T05:07:13.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.273+0000 7f56fa969700 1 --2- 172.21.15.98:0/1089095321 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f56f40ca9f0 0x7f56f415e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:13.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.273+0000 7f56fa969700 1 --2- 172.21.15.98:0/1089095321 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f56f4067800 0x7f56f40b1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:13.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.273+0000 7f56fa969700 1 --2- 172.21.15.98:0/1089095321 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f56e0066f30 0x7f56e00693e0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:13.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.273+0000 7f56fa969700 1 --2- 172.21.15.98:0/1089095321 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f56f406c970 0x7f56f4151740 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:13.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.273+0000 7f56fa969700 1 -- 172.21.15.98:0/1089095321 >> 172.21.15.98:0/1089095321 conn(0x7f56f40cdeb0 msgr2=0x7f56f40c37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:13.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.273+0000 7f56fa969700 1 -- 172.21.15.98:0/1089095321 shutdown_connections 2023-10-06T05:07:13.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:13.273+0000 7f56fa969700 1 -- 172.21.15.98:0/1089095321 wait complete. 2023-10-06T05:07:13.281 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509448 got 64424509447 for osd.2 2023-10-06T05:07:13.287 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542153 got 60129542152 for osd.0 2023-10-06T05:07:13.290 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.7 2023-10-06T05:07:13.290 DEBUG:teuthology.parallel:result is None 2023-10-06T05:07:13.293 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509449 got 64424509448 for osd.4 2023-10-06T05:07:14.267 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-10-06T05:07:14.270 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-10-06T05:07:14.274 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-10-06T05:07:14.276 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-06T05:07:14.281 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-06T05:07:14.287 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-06T05:07:14.294 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-10-06T05:07:14.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.435+0000 7fd1ef1cf700 1 Processor -- start 2023-10-06T05:07:14.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.435+0000 7fd1ef1cf700 1 -- start start 2023-10-06T05:07:14.438 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.436+0000 7fd1ef1cf700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fd1e80707d0 0x7fd1e8068780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:14.438 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.436+0000 7fd1ef1cf700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd1e8070c50 0x7fd1e8155600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:14.438 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.436+0000 7fd1ef1cf700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1e80ac8d0 con 0x7fd1e80ac440 2023-10-06T05:07:14.439 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.436+0000 7fd1ef1cf700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1e80af110 con 0x7fd1e80707d0 2023-10-06T05:07:14.439 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.436+0000 7fd1ef1cf700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e80acbb0 con 0x7fd1e8070c50 2023-10-06T05:07:14.440 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.437+0000 7fd1ed76c700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd1e8070c50 0x7fd1e8155600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:14.440 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.437+0000 7fd1e7fff700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fd1e80707d0 0x7fd1e8068780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:14.440 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.437+0000 7fd1ed76c700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd1e8070c50 0x7fd1e8155600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.98:56048/0 (socket says 172.21.15.98:56048) 2023-10-06T05:07:14.440 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.437+0000 7fd1ed76c700 1 -- 172.21.15.98:0/4191732515 learned_addr learned my addr 172.21.15.98:0/4191732515 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:14.440 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.437+0000 7fd1ed76c700 1 -- 172.21.15.98:0/4191732515 >> v2:172.21.15.174:3300/0 conn(0x7fd1e80707d0 msgr2=0x7fd1e8068780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:14.440 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.437+0000 7fd1ed76c700 1 --2- 172.21.15.98:0/4191732515 >> v2:172.21.15.174:3300/0 conn(0x7fd1e80707d0 0x7fd1e8068780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:14.440 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.437+0000 7fd1ed76c700 1 -- 172.21.15.98:0/4191732515 >> v1:172.21.15.174:6789/0 conn(0x7fd1e80ac440 legacy=0x7fd1e80afb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:14.440 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.437+0000 7fd1ed76c700 1 -- 172.21.15.98:0/4191732515 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1e80ace90 con 0x7fd1e8070c50 2023-10-06T05:07:14.440 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.437+0000 7fd1e7fff700 1 --2- 172.21.15.98:0/4191732515 >> v2:172.21.15.174:3300/0 conn(0x7fd1e80707d0 0x7fd1e8068780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T05:07:14.440 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.437+0000 7fd1ed76c700 1 --2- 172.21.15.98:0/4191732515 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd1e8070c50 0x7fd1e8155600 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd1e0009f20 tx=0x7fd1e0009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b1aa27676d5c1dd9 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:14.440 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.438+0000 7fd1e77fe700 1 -- 172.21.15.98:0/4191732515 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1e000e070 con 0x7fd1e8070c50 2023-10-06T05:07:14.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.438+0000 7fd1e77fe700 1 -- 172.21.15.98:0/4191732515 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1e000ba20 con 0x7fd1e8070c50 2023-10-06T05:07:14.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.438+0000 7fd1e77fe700 1 -- 172.21.15.98:0/4191732515 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1e00092e0 con 0x7fd1e8070c50 2023-10-06T05:07:14.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.438+0000 7fd1ef1cf700 1 -- 172.21.15.98:0/4191732515 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd1e8070c50 msgr2=0x7fd1e8155600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:14.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.439+0000 7fd1ef1cf700 1 --2- 172.21.15.98:0/4191732515 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd1e8070c50 0x7fd1e8155600 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd1e0009f20 tx=0x7fd1e0009840 comp rx=0 tx=0).stop 2023-10-06T05:07:14.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.439+0000 7fd1ef1cf700 1 -- 172.21.15.98:0/4191732515 shutdown_connections 2023-10-06T05:07:14.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.439+0000 7fd1ef1cf700 1 --2- 172.21.15.98:0/4191732515 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd1e8070c50 0x7fd1e8155600 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:14.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.439+0000 7fd1ef1cf700 1 --2- 172.21.15.98:0/4191732515 >> v2:172.21.15.174:3300/0 conn(0x7fd1e80707d0 0x7fd1e8068780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:14.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.439+0000 7fd1ef1cf700 1 -- 172.21.15.98:0/4191732515 >> 172.21.15.98:0/4191732515 conn(0x7fd1e806de40 msgr2=0x7fd1e806e240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:14.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.442+0000 7fd1ef1cf700 1 -- 172.21.15.98:0/4191732515 shutdown_connections 2023-10-06T05:07:14.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.446+0000 7fd1ef1cf700 1 -- 172.21.15.98:0/4191732515 wait complete. 2023-10-06T05:07:14.449 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.447+0000 7fd1ef1cf700 1 Processor -- start 2023-10-06T05:07:14.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.448+0000 7fd1ef1cf700 1 -- start start 2023-10-06T05:07:14.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.449+0000 7fd1ef1cf700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd1e80707d0 0x7fd1e8154890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:14.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.449+0000 7fd1ef1cf700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd1e8070c50 0x7fd1e80c2280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:14.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.449+0000 7fd1ef1cf700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1e813fd00 0x7fd1e8140180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:14.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.449+0000 7fd1ef1cf700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e80a8750 con 0x7fd1e813fd00 2023-10-06T05:07:14.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.449+0000 7fd1ef1cf700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e80a7e60 con 0x7fd1e80707d0 2023-10-06T05:07:14.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.449+0000 7fd1ef1cf700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e80c2c20 con 0x7fd1e8070c50 2023-10-06T05:07:14.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.449+0000 7fd1ed76c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1e813fd00 0x7fd1e8140180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:14.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.449+0000 7fd1ed76c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1e813fd00 0x7fd1e8140180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45918/0 (socket says 172.21.15.98:45918) 2023-10-06T05:07:14.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.449+0000 7fd1ed76c700 1 -- 172.21.15.98:0/516069118 learned_addr learned my addr 172.21.15.98:0/516069118 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:14.453 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.450+0000 7fd1e7fff700 1 --2- 172.21.15.98:0/516069118 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd1e8070c50 0x7fd1e80c2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:14.453 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.450+0000 7fd1ecf6b700 1 --2- 172.21.15.98:0/516069118 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd1e80707d0 0x7fd1e8154890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:14.453 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.450+0000 7fd1ed76c700 1 -- 172.21.15.98:0/516069118 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd1e8070c50 msgr2=0x7fd1e80c2280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:14.454 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.450+0000 7fd1ed76c700 1 --2- 172.21.15.98:0/516069118 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd1e8070c50 0x7fd1e80c2280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:14.454 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.450+0000 7fd1ed76c700 1 -- 172.21.15.98:0/516069118 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd1e80707d0 msgr2=0x7fd1e8154890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:14.455 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.450+0000 7fd1ed76c700 1 --2- 172.21.15.98:0/516069118 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd1e80707d0 0x7fd1e8154890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:14.455 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.450+0000 7fd1ed76c700 1 -- 172.21.15.98:0/516069118 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1dc009420 con 0x7fd1e813fd00 2023-10-06T05:07:14.455 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.450+0000 7fd1ed76c700 1 --2- 172.21.15.98:0/516069118 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1e813fd00 0x7fd1e8140180 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd1e80accf0 tx=0x7fd1e0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:14.455 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.450+0000 7fd1e4ff9700 1 -- 172.21.15.98:0/516069118 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1e00094e0 con 0x7fd1e813fd00 2023-10-06T05:07:14.455 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.450+0000 7fd1ef1cf700 1 -- 172.21.15.98:0/516069118 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1e0009ae0 con 0x7fd1e813fd00 2023-10-06T05:07:14.455 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.450+0000 7fd1e4ff9700 1 -- 172.21.15.98:0/516069118 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1e0012e60 con 0x7fd1e813fd00 2023-10-06T05:07:14.455 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.450+0000 7fd1e4ff9700 1 -- 172.21.15.98:0/516069118 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1e001e680 con 0x7fd1e813fd00 2023-10-06T05:07:14.455 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.450+0000 7fd1ef1cf700 1 -- 172.21.15.98:0/516069118 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1e80a83e0 con 0x7fd1e813fd00 2023-10-06T05:07:14.457 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.452+0000 7fd1ce7fc700 1 -- 172.21.15.98:0/516069118 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1c00052b0 con 0x7fd1e813fd00 2023-10-06T05:07:14.457 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.454+0000 7fd1e4ff9700 1 -- 172.21.15.98:0/516069118 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fd1e001ea30 con 0x7fd1e813fd00 2023-10-06T05:07:14.457 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.454+0000 7fc9bfe82700 1 Processor -- start 2023-10-06T05:07:14.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.454+0000 7fd1e4ff9700 1 --2- 172.21.15.98:0/516069118 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fd1d4066f30 0x7fd1d40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:14.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.454+0000 7fd1e4ff9700 1 -- 172.21.15.98:0/516069118 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fd1e00864a0 con 0x7fd1e813fd00 2023-10-06T05:07:14.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.454+0000 7fc9bfe82700 1 -- start start 2023-10-06T05:07:14.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.455+0000 7fc9bfe82700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc9b80cbc20 0x7fc9b80ae480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:14.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.456+0000 7fc9bfe82700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9b80c7dd0 con 0x7fc9b806e3d0 2023-10-06T05:07:14.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.456+0000 7fc9bfe82700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9b80ae9c0 con 0x7fc9b80c8260 2023-10-06T05:07:14.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.456+0000 7fc9bdc1e700 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fc9b80c8260 0x7fc9b8155820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.98:55558/0 (socket says 172.21.15.98:55558) 2023-10-06T05:07:14.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.456+0000 7fc9bdc1e700 1 -- 172.21.15.98:0/2864149879 learned_addr learned my addr 172.21.15.98:0/2864149879 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:07:14.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.456+0000 7fc9bd41d700 1 --2- 172.21.15.98:0/2864149879 >> v2:172.21.15.98:3300/0 conn(0x7fc9b80cbc20 0x7fc9b80ae480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:14.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.456+0000 7fc9bfe82700 1 -- 172.21.15.98:0/2864149879 --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9b80acca0 con 0x7fc9b80cbc20 2023-10-06T05:07:14.460 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.456+0000 7fc9bcc1c700 1 -- 172.21.15.98:0/2864149879 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4083675436 0 0) 0x7fc9b80ae9c0 con 0x7fc9b80c8260 2023-10-06T05:07:14.460 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.456+0000 7fc9bcc1c700 1 -- 172.21.15.98:0/2864149879 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9a4003570 con 0x7fc9b80c8260 2023-10-06T05:07:14.460 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.456+0000 7fc9bd41d700 1 -- 172.21.15.98:0/2864149879 >> v1:172.21.15.174:6789/0 conn(0x7fc9b80c8260 legacy=0x7fc9b8155820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:14.460 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.456+0000 7fc9bd41d700 1 -- 172.21.15.98:0/2864149879 >> v1:172.21.15.98:6789/0 conn(0x7fc9b806e3d0 legacy=0x7fc9b8069520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:14.460 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.456+0000 7fc9bd41d700 1 -- 172.21.15.98:0/2864149879 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9b80c71c0 con 0x7fc9b80cbc20 2023-10-06T05:07:14.460 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.457+0000 7fc9bd41d700 1 --2- 172.21.15.98:0/2864149879 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9b80cbc20 0x7fc9b80ae480 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc9b40098e0 tx=0x7fc9b40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4f89136622b5204 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:14.460 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.456+0000 7fd1ecf6b700 1 --2- 172.21.15.98:0/516069118 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fd1d4066f30 0x7fd1d40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:14.461 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.457+0000 7fd1ecf6b700 1 --2- 172.21.15.98:0/516069118 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fd1d4066f30 0x7fd1d40693e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd1e806ac10 tx=0x7fd1d8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:14.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.462+0000 7fc9bcc1c700 1 -- 172.21.15.98:0/2864149879 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9b4011630 con 0x7fc9b80cbc20 2023-10-06T05:07:14.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.462+0000 7fc9bcc1c700 1 -- 172.21.15.98:0/2864149879 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9b4011790 con 0x7fc9b80cbc20 2023-10-06T05:07:14.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.462+0000 7fc9bcc1c700 1 -- 172.21.15.98:0/2864149879 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9b4011b50 con 0x7fc9b80cbc20 2023-10-06T05:07:14.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.462+0000 7fd1e4ff9700 1 -- 172.21.15.98:0/516069118 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd1e00532c0 con 0x7fd1e813fd00 2023-10-06T05:07:14.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.463+0000 7fc9bfe82700 1 -- 172.21.15.98:0/2864149879 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9b80cbc20 msgr2=0x7fc9b80ae480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:14.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.463+0000 7fc9bfe82700 1 --2- 172.21.15.98:0/2864149879 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9b80cbc20 0x7fc9b80ae480 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc9b40098e0 tx=0x7fc9b40091a0 comp rx=0 tx=0).stop 2023-10-06T05:07:14.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.463+0000 7fc9bfe82700 1 -- 172.21.15.98:0/2864149879 shutdown_connections 2023-10-06T05:07:14.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.463+0000 7fc9bfe82700 1 --2- 172.21.15.98:0/2864149879 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9b80cbc20 0x7fc9b80ae480 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:14.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.463+0000 7fc9bfe82700 1 -- 172.21.15.98:0/2864149879 >> 172.21.15.98:0/2864149879 conn(0x7fc9b8068080 msgr2=0x7fc9b80cc0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:14.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.463+0000 7fc9bfe82700 1 -- 172.21.15.98:0/2864149879 shutdown_connections 2023-10-06T05:07:14.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.463+0000 7fc9bfe82700 1 -- 172.21.15.98:0/2864149879 wait complete. 2023-10-06T05:07:14.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.465+0000 7fc9bfe82700 1 Processor -- start 2023-10-06T05:07:14.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.465+0000 7fc9bfe82700 1 -- start start 2023-10-06T05:07:14.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.466+0000 7fc9bfe82700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc9b806e3d0 0x7fc9b814ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:14.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.466+0000 7fc9bfe82700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc9b80c8260 0x7fc9b80bf340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:14.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.466+0000 7fc9bfe82700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9b80cbc20 0x7fc9b81714f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:14.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.466+0000 7fc9bfe82700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9b81723e0 con 0x7fc9b80cbc20 2023-10-06T05:07:14.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.466+0000 7fc9bfe82700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9b80bf880 con 0x7fc9b80c8260 2023-10-06T05:07:14.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.466+0000 7fc9bfe82700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9b8151210 con 0x7fc9b806e3d0 2023-10-06T05:07:14.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.467+0000 7fc9be41f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9b80cbc20 0x7fc9b81714f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:14.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.467+0000 7fc9be41f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9b80cbc20 0x7fc9b81714f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45932/0 (socket says 172.21.15.98:45932) 2023-10-06T05:07:14.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.467+0000 7fc9be41f700 1 -- 172.21.15.98:0/1867976768 learned_addr learned my addr 172.21.15.98:0/1867976768 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:14.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.467+0000 7fc9bdc1e700 1 --2- 172.21.15.98:0/1867976768 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc9b806e3d0 0x7fc9b814ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:14.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.467+0000 7fc9be41f700 1 -- 172.21.15.98:0/1867976768 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc9b806e3d0 msgr2=0x7fc9b814ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:14.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.467+0000 7fc9be41f700 1 --2- 172.21.15.98:0/1867976768 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc9b806e3d0 0x7fc9b814ff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:14.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.467+0000 7fc9be41f700 1 -- 172.21.15.98:0/1867976768 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc9b80c8260 msgr2=0x7fc9b80bf340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:14.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.467+0000 7fc9be41f700 1 --2- 172.21.15.98:0/1867976768 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc9b80c8260 0x7fc9b80bf340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:14.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.467+0000 7fc9be41f700 1 -- 172.21.15.98:0/1867976768 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9b4009420 con 0x7fc9b80cbc20 2023-10-06T05:07:14.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.467+0000 7fc9be41f700 1 --2- 172.21.15.98:0/1867976768 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9b80cbc20 0x7fc9b81714f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc9b000bb80 tx=0x7fc9b0009810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:14.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.469+0000 7fc9adffb700 1 -- 172.21.15.98:0/1867976768 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9b000c8c0 con 0x7fc9b80cbc20 2023-10-06T05:07:14.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.469+0000 7fc9adffb700 1 -- 172.21.15.98:0/1867976768 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9b000ca60 con 0x7fc9b80cbc20 2023-10-06T05:07:14.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.469+0000 7fc9adffb700 1 -- 172.21.15.98:0/1867976768 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9b000ce20 con 0x7fc9b80cbc20 2023-10-06T05:07:14.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.469+0000 7fc9bfe82700 1 -- 172.21.15.98:0/1867976768 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9b8150f30 con 0x7fc9b80cbc20 2023-10-06T05:07:14.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.469+0000 7fc9bfe82700 1 -- 172.21.15.98:0/1867976768 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9b81508e0 con 0x7fc9b80cbc20 2023-10-06T05:07:14.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.471+0000 7fc9bfe82700 1 -- 172.21.15.98:0/1867976768 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9b8151380 con 0x7fc9b80cbc20 2023-10-06T05:07:14.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.484+0000 7fc9adffb700 1 -- 172.21.15.98:0/1867976768 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fc9b0014620 con 0x7fc9b80cbc20 2023-10-06T05:07:14.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.484+0000 7fc9adffb700 1 --2- 172.21.15.98:0/1867976768 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fc99c067080 0x7fc99c069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:14.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.484+0000 7fc9adffb700 1 -- 172.21.15.98:0/1867976768 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fc9b0085360 con 0x7fc9b80cbc20 2023-10-06T05:07:14.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.485+0000 7fc9adffb700 1 -- 172.21.15.98:0/1867976768 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc9b00521e0 con 0x7fc9b80cbc20 2023-10-06T05:07:14.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.485+0000 7fc9bdc1e700 1 --2- 172.21.15.98:0/1867976768 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fc99c067080 0x7fc99c069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:14.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.485+0000 7fc9bdc1e700 1 --2- 172.21.15.98:0/1867976768 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fc99c067080 0x7fc99c069530 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc9a8007fd0 tx=0x7fc9a8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:14.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.485+0000 7f2bb3841700 1 Processor -- start 2023-10-06T05:07:14.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.486+0000 7f2bb3841700 1 -- start start 2023-10-06T05:07:14.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.487+0000 7f2bb3841700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f2bac15acd0 0x7f2bac0ac0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:14.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.487+0000 7f2bb3841700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bac06d580 con 0x7f2bac069520 2023-10-06T05:07:14.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.487+0000 7f2bb3841700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bac0ae250 con 0x7f2bac15a900 2023-10-06T05:07:14.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.487+0000 7f2bb3841700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bac0aead0 con 0x7f2bac15acd0 2023-10-06T05:07:14.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.487+0000 7f2bb1dde700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f2bac069520 0x7f2bac069910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:41214/0 (socket says 172.21.15.98:41214) 2023-10-06T05:07:14.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.487+0000 7f2bb1dde700 1 -- 172.21.15.98:0/1674898093 learned_addr learned my addr 172.21.15.98:0/1674898093 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:07:14.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.487+0000 7f2bb0ddc700 1 --2- 172.21.15.98:0/1674898093 >> v2:172.21.15.174:3300/0 conn(0x7f2bac15acd0 0x7f2bac0ac0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:14.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.488+0000 7f2babfff700 1 -- 172.21.15.98:0/1674898093 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1607278283 0 0) 0x7f2bac06d580 con 0x7f2bac069520 2023-10-06T05:07:14.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.488+0000 7f2babfff700 1 -- 172.21.15.98:0/1674898093 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b94003570 con 0x7f2bac069520 2023-10-06T05:07:14.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.488+0000 7f2bb0ddc700 1 -- 172.21.15.98:0/1674898093 >> v1:172.21.15.174:6789/0 conn(0x7f2bac15a900 legacy=0x7f2bac0ad980 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-06T05:07:14.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.488+0000 7f2bb0ddc700 1 -- 172.21.15.98:0/1674898093 >> v1:172.21.15.98:6789/0 conn(0x7f2bac069520 legacy=0x7f2bac069910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:14.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.488+0000 7f2bb0ddc700 1 -- 172.21.15.98:0/1674898093 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bac0ae510 con 0x7f2bac15acd0 2023-10-06T05:07:14.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.488+0000 7f2bb0ddc700 1 --2- 172.21.15.98:0/1674898093 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2bac15acd0 0x7f2bac0ac0d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2ba00098e0 tx=0x7f2ba00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f60c73e6feec844b server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:14.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.488+0000 7f2babfff700 1 -- 172.21.15.98:0/1674898093 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba0011630 con 0x7f2bac15acd0 2023-10-06T05:07:14.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.488+0000 7f2babfff700 1 -- 172.21.15.98:0/1674898093 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ba0011790 con 0x7f2bac15acd0 2023-10-06T05:07:14.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.488+0000 7f2babfff700 1 -- 172.21.15.98:0/1674898093 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba0011b30 con 0x7f2bac15acd0 2023-10-06T05:07:14.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.489+0000 7f2bb3841700 1 -- 172.21.15.98:0/1674898093 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2bac15acd0 msgr2=0x7f2bac0ac0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:14.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.489+0000 7f2bb3841700 1 --2- 172.21.15.98:0/1674898093 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2bac15acd0 0x7f2bac0ac0d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2ba00098e0 tx=0x7f2ba00091a0 comp rx=0 tx=0).stop 2023-10-06T05:07:14.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.489+0000 7f2bb3841700 1 -- 172.21.15.98:0/1674898093 shutdown_connections 2023-10-06T05:07:14.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.489+0000 7f2bb3841700 1 --2- 172.21.15.98:0/1674898093 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2bac15acd0 0x7f2bac0ac0d0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:14.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.489+0000 7f2bb3841700 1 -- 172.21.15.98:0/1674898093 >> 172.21.15.98:0/1674898093 conn(0x7f2bac068080 msgr2=0x7f2bac0cc0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:14.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.490+0000 7f2bb3841700 1 -- 172.21.15.98:0/1674898093 shutdown_connections 2023-10-06T05:07:14.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.490+0000 7f2bb3841700 1 -- 172.21.15.98:0/1674898093 wait complete. 2023-10-06T05:07:14.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.491+0000 7f2bb3841700 1 Processor -- start 2023-10-06T05:07:14.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.492+0000 7f2bb3841700 1 -- start start 2023-10-06T05:07:14.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.493+0000 7f2bb3841700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2bac069520 0x7f2bac0aee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:14.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.493+0000 7f2bb3841700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2bac15acd0 0x7f2bac14b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:14.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.493+0000 7f2bb3841700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2bac0c6ba0 0x7f2bac0cabe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:14.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.493+0000 7f2bb3841700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bac16f4c0 con 0x7f2bac0c6ba0 2023-10-06T05:07:14.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.493+0000 7f2bb3841700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bac0b0230 con 0x7f2bac069520 2023-10-06T05:07:14.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.493+0000 7f2bb3841700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bac0c96e0 con 0x7f2bac15acd0 2023-10-06T05:07:14.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.493+0000 7f2bb1dde700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2bac0c6ba0 0x7f2bac0cabe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:14.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.493+0000 7f2bb1dde700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2bac0c6ba0 0x7f2bac0cabe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45934/0 (socket says 172.21.15.98:45934) 2023-10-06T05:07:14.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.493+0000 7f2bb1dde700 1 -- 172.21.15.98:0/1627614499 learned_addr learned my addr 172.21.15.98:0/1627614499 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:14.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.494+0000 7f2bb1dde700 1 -- 172.21.15.98:0/1627614499 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2bac15acd0 msgr2=0x7f2bac14b110 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-06T05:07:14.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.494+0000 7f2bb1dde700 1 --2- 172.21.15.98:0/1627614499 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2bac15acd0 0x7f2bac14b110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:14.497 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.494+0000 7f2bb1dde700 1 -- 172.21.15.98:0/1627614499 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2bac069520 msgr2=0x7f2bac0aee90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-06T05:07:14.497 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.494+0000 7f2bb1dde700 1 --2- 172.21.15.98:0/1627614499 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2bac069520 0x7f2bac0aee90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:14.497 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.494+0000 7f2bb1dde700 1 -- 172.21.15.98:0/1627614499 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ba0009420 con 0x7f2bac0c6ba0 2023-10-06T05:07:14.497 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.494+0000 7f2bb1dde700 1 --2- 172.21.15.98:0/1627614499 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2bac0c6ba0 0x7f2bac0cabe0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f2ba40014b0 tx=0x7f2ba4001a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:14.497 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.494+0000 7f2ba97fa700 1 -- 172.21.15.98:0/1627614499 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba400a930 con 0x7f2bac0c6ba0 2023-10-06T05:07:14.497 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.494+0000 7f2bb3841700 1 -- 172.21.15.98:0/1627614499 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bac14b650 con 0x7f2bac0c6ba0 2023-10-06T05:07:14.497 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.494+0000 7f2bb3841700 1 -- 172.21.15.98:0/1627614499 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bac0af870 con 0x7f2bac0c6ba0 2023-10-06T05:07:14.498 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.494+0000 7f2ba97fa700 1 -- 172.21.15.98:0/1627614499 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ba400aca0 con 0x7f2bac0c6ba0 2023-10-06T05:07:14.498 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.494+0000 7f2ba97fa700 1 -- 172.21.15.98:0/1627614499 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba400ae10 con 0x7f2bac0c6ba0 2023-10-06T05:07:14.498 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.496+0000 7f2bb3841700 1 -- 172.21.15.98:0/1627614499 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b8c0052b0 con 0x7f2bac0c6ba0 2023-10-06T05:07:14.499 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.497+0000 7f2ba97fa700 1 -- 172.21.15.98:0/1627614499 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f2ba401b610 con 0x7f2bac0c6ba0 2023-10-06T05:07:14.500 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.498+0000 7f2ba97fa700 1 --2- 172.21.15.98:0/1627614499 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f2b98066f30 0x7f2b980693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:14.500 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.498+0000 7f2ba97fa700 1 -- 172.21.15.98:0/1627614499 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f2ba4085760 con 0x7f2bac0c6ba0 2023-10-06T05:07:14.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.501+0000 7f2bb15dd700 1 --2- 172.21.15.98:0/1627614499 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f2b98066f30 0x7f2b980693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:14.503 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.501+0000 7f2bb15dd700 1 --2- 172.21.15.98:0/1627614499 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f2b98066f30 0x7f2b980693e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f2bac15ba30 tx=0x7f2b9c006c60 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:14.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.504+0000 7fc1ef06f700 1 Processor -- start 2023-10-06T05:07:14.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.501+0000 7f745fc0b700 1 Processor -- start 2023-10-06T05:07:14.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.501+0000 7f745fc0b700 1 -- start start 2023-10-06T05:07:14.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.502+0000 7f745fc0b700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f745806d510 0x7f745806d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:14.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.502+0000 7f745fc0b700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f74580c82e0 0x7f74580878b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:14.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.502+0000 7f745fc0b700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7458087df0 con 0x7f7458069520 2023-10-06T05:07:14.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.502+0000 7f745fc0b700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74580c89e0 con 0x7f745806d510 2023-10-06T05:07:14.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.502+0000 7f745fc0b700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74580c7040 con 0x7f74580c82e0 2023-10-06T05:07:14.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.502+0000 7f745e1a8700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f7458069520 0x7f7458069940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:41220/0 (socket says 172.21.15.98:41220) 2023-10-06T05:07:14.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.502+0000 7f745e1a8700 1 -- 172.21.15.98:0/1280552195 learned_addr learned my addr 172.21.15.98:0/1280552195 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:07:14.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.502+0000 7f745d9a7700 1 --2- 172.21.15.98:0/1280552195 >> v2:172.21.15.174:3300/0 conn(0x7f745806d510 0x7f745806d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:14.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.502+0000 7f745d1a6700 1 --2- 172.21.15.98:0/1280552195 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f74580c82e0 0x7f74580878b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:14.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.504+0000 7f745d9a7700 1 -- 172.21.15.98:0/1280552195 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f74580c82e0 msgr2=0x7f74580878b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:14.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.504+0000 7f745d9a7700 1 --2- 172.21.15.98:0/1280552195 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f74580c82e0 0x7f74580878b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:14.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.504+0000 7f745d9a7700 1 -- 172.21.15.98:0/1280552195 >> v1:172.21.15.98:6789/0 conn(0x7f7458069520 legacy=0x7f7458069940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:14.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.504+0000 7f745d9a7700 1 -- 172.21.15.98:0/1280552195 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74580c8750 con 0x7f745806d510 2023-10-06T05:07:14.508 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.505+0000 7f2ba97fa700 1 -- 172.21.15.98:0/1627614499 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2ba40526e0 con 0x7f2bac0c6ba0 2023-10-06T05:07:14.508 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.505+0000 7f2aee853700 1 Processor -- start 2023-10-06T05:07:14.508 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.505+0000 7f2aee853700 1 -- start start 2023-10-06T05:07:14.509 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.504+0000 7fc1ef06f700 1 -- start start 2023-10-06T05:07:14.509 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.505+0000 7fc1ef06f700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fc1e80c8260 0x7fc1e8069520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:14.509 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.505+0000 7fc1ef06f700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1e80c7dd0 con 0x7fc1e8155820 2023-10-06T05:07:14.509 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.505+0000 7f745d9a7700 1 --2- 172.21.15.98:0/1280552195 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f745806d510 0x7f745806d8e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f74480078e0 tx=0x7f7448007200 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4c3356cf4c445998 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:14.511 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.505+0000 7fc1ef06f700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1e80c71c0 con 0x7fc1e806e3d0 2023-10-06T05:07:14.511 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.505+0000 7fc1ef06f700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1e80c74a0 con 0x7fc1e80c8260 2023-10-06T05:07:14.511 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.505+0000 7fc1ece0b700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fc1e8155820 0x7fc1e80ae480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:41222/0 (socket says 172.21.15.98:41222) 2023-10-06T05:07:14.511 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.505+0000 7fc1ece0b700 1 -- 172.21.15.98:0/2827380405 learned_addr learned my addr 172.21.15.98:0/2827380405 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:07:14.511 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.506+0000 7fc1e77fe700 1 -- 172.21.15.98:0/2827380405 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1407882707 0 0) 0x7fc1e80c7dd0 con 0x7fc1e8155820 2023-10-06T05:07:14.511 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.506+0000 7fc1e77fe700 1 -- 172.21.15.98:0/2827380405 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1d0003570 con 0x7fc1e8155820 2023-10-06T05:07:14.511 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.505+0000 7fc1e7fff700 1 --2- 172.21.15.98:0/2827380405 >> v2:172.21.15.174:3300/0 conn(0x7fc1e80c8260 0x7fc1e8069520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:14.511 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.506+0000 7fc1e77fe700 1 -- 172.21.15.98:0/2827380405 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 856216464 0 0) 0x7fc1e80c71c0 con 0x7fc1e806e3d0 2023-10-06T05:07:14.511 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.506+0000 7fc1e77fe700 1 -- 172.21.15.98:0/2827380405 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1e80c7dd0 con 0x7fc1e806e3d0 2023-10-06T05:07:14.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.506+0000 7fc1e77fe700 1 -- 172.21.15.98:0/2827380405 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2029114869 0 0) 0x7fc1d0003570 con 0x7fc1e8155820 2023-10-06T05:07:14.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.506+0000 7fc1e77fe700 1 -- 172.21.15.98:0/2827380405 >> v2:172.21.15.174:3300/0 conn(0x7fc1e80c8260 msgr2=0x7fc1e8069520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:14.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.506+0000 7fc1e77fe700 1 --2- 172.21.15.98:0/2827380405 >> v2:172.21.15.174:3300/0 conn(0x7fc1e80c8260 0x7fc1e8069520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:14.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.506+0000 7fc1e77fe700 1 -- 172.21.15.98:0/2827380405 >> v1:172.21.15.174:6789/0 conn(0x7fc1e806e3d0 legacy=0x7fc1e80abc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:14.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.506+0000 7fc1e77fe700 1 -- 172.21.15.98:0/2827380405 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1e80c7780 con 0x7fc1e8155820 2023-10-06T05:07:14.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.506+0000 7fc1e7fff700 1 --2- 172.21.15.98:0/2827380405 >> v2:172.21.15.174:3300/0 conn(0x7fc1e80c8260 0x7fc1e8069520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:07:14.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.506+0000 7fc1e77fe700 1 -- 172.21.15.98:0/2827380405 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1543991876 0 0) 0x7fc1e00017f0 con 0x7fc1e8155820 2023-10-06T05:07:14.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.506+0000 7fc1e77fe700 1 -- 172.21.15.98:0/2827380405 >> v1:172.21.15.98:6789/0 conn(0x7fc1e8155820 legacy=0x7fc1e80ae480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:14.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.506+0000 7fc1e77fe700 1 --2- 172.21.15.98:0/2827380405 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc1d0003bf0 0x7fc1d0003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:14.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.506+0000 7fc1e77fe700 1 --2- 172.21.15.98:0/2827380405 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1d0004810 0x7fc1d0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:14.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.506+0000 7fc1e77fe700 1 --2- 172.21.15.98:0/2827380405 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1d0005230 0x7fc1d0007a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:14.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.506+0000 7fc1e77fe700 1 -- 172.21.15.98:0/2827380405 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e80c74a0 con 0x7fc1d0005230 2023-10-06T05:07:14.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.506+0000 7fc1e77fe700 1 -- 172.21.15.98:0/2827380405 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1d0008190 con 0x7fc1d0004810 2023-10-06T05:07:14.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.506+0000 7fc1e77fe700 1 -- 172.21.15.98:0/2827380405 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1d0008350 con 0x7fc1d0003bf0 2023-10-06T05:07:14.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.506+0000 7fc1ece0b700 1 --2- 172.21.15.98:0/2827380405 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc1d0003bf0 0x7fc1d0003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:14.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.506+0000 7fc1e7fff700 1 --2- 172.21.15.98:0/2827380405 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1d0004810 0x7fc1d0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:14.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.506+0000 7fc1ed60c700 1 --2- 172.21.15.98:0/2827380405 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1d0005230 0x7fc1d0007a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:14.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.506+0000 7f2aee853700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f2ae8156350 0x7f2ae80cc1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:14.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.506+0000 7f2aee853700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2ae80cc710 0x7f2ae806bc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:14.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.506+0000 7f2aee853700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ae806c9a0 con 0x7f2ae806c160 2023-10-06T05:07:14.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.506+0000 7f2aee853700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ae80ccb90 con 0x7f2ae80cc710 2023-10-06T05:07:14.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.506+0000 7f2aee853700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ae8156b20 con 0x7f2ae8156350 2023-10-06T05:07:14.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.507+0000 7f2ae7fff700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f2ae8156350 0x7f2ae80cc1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:14.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.507+0000 7f2aecdf0700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f2ae806c160 0x7f2ae80cafb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:41238/0 (socket says 172.21.15.98:41238) 2023-10-06T05:07:14.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.507+0000 7f2aecdf0700 1 -- 172.21.15.98:0/846328847 learned_addr learned my addr 172.21.15.98:0/846328847 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:07:14.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.506+0000 7fc1ece0b700 1 -- 172.21.15.98:0/2827380405 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1d0004810 msgr2=0x7fc1d0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:14.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.506+0000 7fc1ece0b700 1 --2- 172.21.15.98:0/2827380405 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1d0004810 0x7fc1d0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:14.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.506+0000 7fc1ece0b700 1 -- 172.21.15.98:0/2827380405 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1d0005230 msgr2=0x7fc1d0007a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:14.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.506+0000 7fc1ece0b700 1 --2- 172.21.15.98:0/2827380405 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1d0005230 0x7fc1d0007a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:14.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.507+0000 7fc1ece0b700 1 -- 172.21.15.98:0/2827380405 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1d0003570 con 0x7fc1d0003bf0 2023-10-06T05:07:14.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.507+0000 7fc1ed60c700 1 -- 172.21.15.98:0/2827380405 reap_dead start 2023-10-06T05:07:14.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.507+0000 7fc1ece0b700 1 --2- 172.21.15.98:0/2827380405 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc1d0003bf0 0x7fc1d0003fe0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc1e000a8f0 tx=0x7fc1e0002190 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c7370095bb36088c server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:14.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.507+0000 7fc1e77fe700 1 -- 172.21.15.98:0/2827380405 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1e00070d0 con 0x7fc1d0003bf0 2023-10-06T05:07:14.516 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.507+0000 7f2ae7fff700 1 -- 172.21.15.98:0/846328847 >> v2:172.21.15.98:3300/0 conn(0x7f2ae80cc710 msgr2=0x7f2ae806bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:14.516 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.507+0000 7f2ae7fff700 1 --2- 172.21.15.98:0/846328847 >> v2:172.21.15.98:3300/0 conn(0x7f2ae80cc710 0x7f2ae806bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:14.516 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.507+0000 7f2ae7fff700 1 -- 172.21.15.98:0/846328847 >> v1:172.21.15.98:6789/0 conn(0x7f2ae806c160 legacy=0x7f2ae80cafb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:14.516 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.507+0000 7f2ae7fff700 1 -- 172.21.15.98:0/846328847 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ae8156e00 con 0x7f2ae8156350 2023-10-06T05:07:14.516 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.507+0000 7fc1e77fe700 1 -- 172.21.15.98:0/2827380405 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1e0007250 con 0x7fc1d0003bf0 2023-10-06T05:07:14.517 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.507+0000 7fc1e77fe700 1 -- 172.21.15.98:0/2827380405 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1e0007610 con 0x7fc1d0003bf0 2023-10-06T05:07:14.517 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.507+0000 7f2ae7fff700 1 --2- 172.21.15.98:0/846328847 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2ae8156350 0x7f2ae80cc1d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f2ae8069f50 tx=0x7f2ad80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=26d97ccb85bae079 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:14.517 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.507+0000 7f2ae6ffd700 1 -- 172.21.15.98:0/846328847 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ad800b6b0 con 0x7f2ae8156350 2023-10-06T05:07:14.517 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.507+0000 7f2ae6ffd700 1 -- 172.21.15.98:0/846328847 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ad800b810 con 0x7f2ae8156350 2023-10-06T05:07:14.517 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.507+0000 7f2ae6ffd700 1 -- 172.21.15.98:0/846328847 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ad800bbb0 con 0x7f2ae8156350 2023-10-06T05:07:14.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.508+0000 7fe97d10b700 1 Processor -- start 2023-10-06T05:07:14.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.508+0000 7f745c9a5700 1 -- 172.21.15.98:0/1280552195 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f744800e070 con 0x7f745806d510 2023-10-06T05:07:14.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.508+0000 7f745c9a5700 1 -- 172.21.15.98:0/1280552195 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f744800b930 con 0x7f745806d510 2023-10-06T05:07:14.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.508+0000 7f745c9a5700 1 -- 172.21.15.98:0/1280552195 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f744800bcd0 con 0x7f745806d510 2023-10-06T05:07:14.519 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.508+0000 7fc1ef06f700 1 -- 172.21.15.98:0/2827380405 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc1d0003bf0 msgr2=0x7fc1d0003fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:14.519 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.508+0000 7fc1ef06f700 1 --2- 172.21.15.98:0/2827380405 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc1d0003bf0 0x7fc1d0003fe0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc1e000a8f0 tx=0x7fc1e0002190 comp rx=0 tx=0).stop 2023-10-06T05:07:14.519 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.508+0000 7fc1ef06f700 1 -- 172.21.15.98:0/2827380405 shutdown_connections 2023-10-06T05:07:14.519 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.508+0000 7fc1ef06f700 1 --2- 172.21.15.98:0/2827380405 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc1d0003bf0 0x7fc1d0003fe0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:14.519 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.508+0000 7fc1ef06f700 1 -- 172.21.15.98:0/2827380405 >> 172.21.15.98:0/2827380405 conn(0x7fc1e8068080 msgr2=0x7fc1e80cc0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:14.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.508+0000 7f2aee853700 1 -- 172.21.15.98:0/846328847 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2ae8156350 msgr2=0x7f2ae80cc1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:14.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.508+0000 7f2aee853700 1 --2- 172.21.15.98:0/846328847 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2ae8156350 0x7f2ae80cc1d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f2ae8069f50 tx=0x7f2ad80091a0 comp rx=0 tx=0).stop 2023-10-06T05:07:14.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.508+0000 7f2aee853700 1 -- 172.21.15.98:0/846328847 shutdown_connections 2023-10-06T05:07:14.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.508+0000 7f2aee853700 1 --2- 172.21.15.98:0/846328847 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2ae8156350 0x7f2ae80cc1d0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:14.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.508+0000 7f2aee853700 1 --2- 172.21.15.98:0/846328847 >> v2:172.21.15.98:3300/0 conn(0x7f2ae80cc710 0x7f2ae806bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:14.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.508+0000 7f2aee853700 1 -- 172.21.15.98:0/846328847 >> 172.21.15.98:0/846328847 conn(0x7f2ae80cdeb0 msgr2=0x7f2ae80c9e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:14.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.508+0000 7f2aee853700 1 -- 172.21.15.98:0/846328847 shutdown_connections 2023-10-06T05:07:14.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.508+0000 7f745fc0b700 1 -- 172.21.15.98:0/1280552195 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f745806d510 msgr2=0x7f745806d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:14.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.508+0000 7f745fc0b700 1 --2- 172.21.15.98:0/1280552195 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f745806d510 0x7f745806d8e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f74480078e0 tx=0x7f7448007200 comp rx=0 tx=0).stop 2023-10-06T05:07:14.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.508+0000 7f745fc0b700 1 -- 172.21.15.98:0/1280552195 shutdown_connections 2023-10-06T05:07:14.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.508+0000 7f745fc0b700 1 --2- 172.21.15.98:0/1280552195 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f745806d510 0x7f745806d8e0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:14.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.508+0000 7f745fc0b700 1 --2- 172.21.15.98:0/1280552195 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f74580c82e0 0x7f74580878b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:14.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.509+0000 7f2aee853700 1 -- 172.21.15.98:0/846328847 wait complete. 2023-10-06T05:07:14.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.508+0000 7f745fc0b700 1 -- 172.21.15.98:0/1280552195 >> 172.21.15.98:0/1280552195 conn(0x7f7458068080 msgr2=0x7f74580cc0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:14.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.508+0000 7fe97d10b700 1 -- start start 2023-10-06T05:07:14.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.509+0000 7fe97d10b700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe97815d8b0 0x7fe97815dc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:14.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.509+0000 7fe97d10b700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fe9780b1d10 0x7fe9780b2180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:14.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.509+0000 7fe97d10b700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9780c9710 con 0x7fe9780685c0 2023-10-06T05:07:14.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.509+0000 7fe97d10b700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9780c8eb0 con 0x7fe9780b1d10 2023-10-06T05:07:14.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.509+0000 7fe97d10b700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9780b26c0 con 0x7fe97815d8b0 2023-10-06T05:07:14.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.509+0000 7fe97659c700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fe9780b1d10 0x7fe9780b2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:14.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.509+0000 7fe976d9d700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe97815d8b0 0x7fe97815dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:14.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.509+0000 7fe97659c700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fe9780b1d10 0x7fe9780b2180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.98:51506/0 (socket says 172.21.15.98:51506) 2023-10-06T05:07:14.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.509+0000 7fe97659c700 1 -- 172.21.15.98:0/3912043739 learned_addr learned my addr 172.21.15.98:0/3912043739 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:14.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.510+0000 7fe97659c700 1 -- 172.21.15.98:0/3912043739 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe97815d8b0 msgr2=0x7fe97815dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:14.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.510+0000 7fe97659c700 1 --2- 172.21.15.98:0/3912043739 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe97815d8b0 0x7fe97815dc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:14.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.510+0000 7fe97659c700 1 -- 172.21.15.98:0/3912043739 >> v1:172.21.15.174:6789/0 conn(0x7fe9780685c0 legacy=0x7fe978161780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:14.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.510+0000 7fe97659c700 1 -- 172.21.15.98:0/3912043739 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9780b28d0 con 0x7fe9780b1d10 2023-10-06T05:07:14.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.510+0000 7f745fc0b700 1 -- 172.21.15.98:0/1280552195 shutdown_connections 2023-10-06T05:07:14.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.510+0000 7f745fc0b700 1 -- 172.21.15.98:0/1280552195 wait complete. 2023-10-06T05:07:14.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.510+0000 7fe97659c700 1 --2- 172.21.15.98:0/3912043739 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe9780b1d10 0x7fe9780b2180 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe9680099b0 tx=0x7fe968009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1259a529ad6e4ef8 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:14.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.510+0000 7fe975d9b700 1 -- 172.21.15.98:0/3912043739 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe968009ea0 con 0x7fe9780b1d10 2023-10-06T05:07:14.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.510+0000 7fe975d9b700 1 -- 172.21.15.98:0/3912043739 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe96800b860 con 0x7fe9780b1d10 2023-10-06T05:07:14.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.510+0000 7fe975d9b700 1 -- 172.21.15.98:0/3912043739 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe96800bc00 con 0x7fe9780b1d10 2023-10-06T05:07:14.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.510+0000 7f2aee853700 1 Processor -- start 2023-10-06T05:07:14.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.510+0000 7f2aee853700 1 -- start start 2023-10-06T05:07:14.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.511+0000 7fe97d10b700 1 -- 172.21.15.98:0/3912043739 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe9780b1d10 msgr2=0x7fe9780b2180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:14.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.511+0000 7fe97d10b700 1 --2- 172.21.15.98:0/3912043739 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe9780b1d10 0x7fe9780b2180 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe9680099b0 tx=0x7fe968009250 comp rx=0 tx=0).stop 2023-10-06T05:07:14.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.511+0000 7fc1ef06f700 1 -- 172.21.15.98:0/2827380405 shutdown_connections 2023-10-06T05:07:14.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.511+0000 7fe97d10b700 1 -- 172.21.15.98:0/3912043739 shutdown_connections 2023-10-06T05:07:14.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.511+0000 7fe97d10b700 1 --2- 172.21.15.98:0/3912043739 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe97815d8b0 0x7fe97815dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:14.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.511+0000 7fe97d10b700 1 --2- 172.21.15.98:0/3912043739 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe9780b1d10 0x7fe9780b2180 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:14.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.511+0000 7fe97d10b700 1 -- 172.21.15.98:0/3912043739 >> 172.21.15.98:0/3912043739 conn(0x7fe97806de40 msgr2=0x7fe97806e240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:14.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.511+0000 7f2aee853700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ae806c160 0x7f2ae8147ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:14.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.511+0000 7f2aee853700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2ae80cc710 0x7f2ae8148d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:14.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.511+0000 7f2aee853700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2ae8156350 0x7f2ae80b25d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:14.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.511+0000 7f2aee853700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ae8149820 con 0x7f2ae806c160 2023-10-06T05:07:14.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.511+0000 7f2aee853700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ae8149b00 con 0x7f2ae80cc710 2023-10-06T05:07:14.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.511+0000 7f2aee853700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ae8149de0 con 0x7f2ae8156350 2023-10-06T05:07:14.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.511+0000 7f2ae7fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ae806c160 0x7f2ae8147ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:14.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.511+0000 7f745fc0b700 1 Processor -- start 2023-10-06T05:07:14.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.511+0000 7f2ae7fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ae806c160 0x7f2ae8147ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45968/0 (socket says 172.21.15.98:45968) 2023-10-06T05:07:14.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.511+0000 7f2ae7fff700 1 -- 172.21.15.98:0/1163746025 learned_addr learned my addr 172.21.15.98:0/1163746025 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:14.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.511+0000 7f2ae77fe700 1 --2- 172.21.15.98:0/1163746025 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2ae80cc710 0x7f2ae8148d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:14.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.511+0000 7f2aecdf0700 1 --2- 172.21.15.98:0/1163746025 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2ae8156350 0x7f2ae80b25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:14.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.512+0000 7f745fc0b700 1 -- start start 2023-10-06T05:07:14.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.512+0000 7f2ae7fff700 1 -- 172.21.15.98:0/1163746025 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2ae8156350 msgr2=0x7f2ae80b25d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:14.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.512+0000 7f2ae7fff700 1 --2- 172.21.15.98:0/1163746025 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2ae8156350 0x7f2ae80b25d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:14.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.512+0000 7f2ae7fff700 1 -- 172.21.15.98:0/1163746025 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2ae80cc710 msgr2=0x7f2ae8148d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:14.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.512+0000 7f2ae7fff700 1 --2- 172.21.15.98:0/1163746025 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2ae80cc710 0x7f2ae8148d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:14.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.512+0000 7f2ae7fff700 1 -- 172.21.15.98:0/1163746025 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ad8009420 con 0x7f2ae806c160 2023-10-06T05:07:14.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.512+0000 7f2ae77fe700 1 --2- 172.21.15.98:0/1163746025 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2ae80cc710 0x7f2ae8148d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T05:07:14.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.512+0000 7f2ae7fff700 1 --2- 172.21.15.98:0/1163746025 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ae806c160 0x7f2ae8147ad0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f2ae8156c60 tx=0x7f2ad8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:14.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.512+0000 7f2acffff700 1 -- 172.21.15.98:0/1163746025 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ad800bb90 con 0x7f2ae806c160 2023-10-06T05:07:14.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.512+0000 7f2aee853700 1 -- 172.21.15.98:0/1163746025 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ae814a0c0 con 0x7f2ae806c160 2023-10-06T05:07:14.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.512+0000 7f2aee853700 1 -- 172.21.15.98:0/1163746025 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ae8148010 con 0x7f2ae806c160 2023-10-06T05:07:14.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.513+0000 7f745fc0b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7458069520 0x7f74580adaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:14.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.513+0000 7f745fc0b700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f745806d510 0x7f745817f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:14.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.513+0000 7f745fc0b700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f74580c82e0 0x7f745817fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:14.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.513+0000 7f745fc0b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74581696a0 con 0x7f7458069520 2023-10-06T05:07:14.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.513+0000 7f745fc0b700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7458180240 con 0x7f74580c82e0 2023-10-06T05:07:14.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.513+0000 7f745fc0b700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74580ad470 con 0x7f745806d510 2023-10-06T05:07:14.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.513+0000 7f745d9a7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7458069520 0x7f74580adaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:14.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.513+0000 7f745d9a7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7458069520 0x7f74580adaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45982/0 (socket says 172.21.15.98:45982) 2023-10-06T05:07:14.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.513+0000 7f745d9a7700 1 -- 172.21.15.98:0/241603522 learned_addr learned my addr 172.21.15.98:0/241603522 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:14.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.513+0000 7f745d1a6700 1 --2- 172.21.15.98:0/241603522 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f745806d510 0x7f745817f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:14.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.512+0000 7f2acffff700 1 -- 172.21.15.98:0/1163746025 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ad8011630 con 0x7f2ae806c160 2023-10-06T05:07:14.533 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.512+0000 7f2acffff700 1 -- 172.21.15.98:0/1163746025 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ad80117a0 con 0x7f2ae806c160 2023-10-06T05:07:14.533 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.513+0000 7f745d1a6700 1 -- 172.21.15.98:0/241603522 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f74580c82e0 msgr2=0x7f745817fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:14.533 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.513+0000 7f745d1a6700 1 --2- 172.21.15.98:0/241603522 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f74580c82e0 0x7f745817fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:14.533 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.513+0000 7f745d1a6700 1 -- 172.21.15.98:0/241603522 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7458069520 msgr2=0x7f74580adaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:14.533 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.513+0000 7f745d1a6700 1 --2- 172.21.15.98:0/241603522 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7458069520 0x7f74580adaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:14.533 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.513+0000 7f745d1a6700 1 -- 172.21.15.98:0/241603522 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74480074a0 con 0x7f745806d510 2023-10-06T05:07:14.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.514+0000 7fe97d10b700 1 -- 172.21.15.98:0/3912043739 shutdown_connections 2023-10-06T05:07:14.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.514+0000 7fe97d10b700 1 -- 172.21.15.98:0/3912043739 wait complete. 2023-10-06T05:07:14.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.514+0000 7f745d1a6700 1 --2- 172.21.15.98:0/241603522 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f745806d510 0x7f745817f7c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f745400dc70 tx=0x7f745400f040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:14.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.514+0000 7f744dffb700 1 -- 172.21.15.98:0/241603522 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f745400de80 con 0x7f745806d510 2023-10-06T05:07:14.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.514+0000 7f744dffb700 1 -- 172.21.15.98:0/241603522 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7454009b50 con 0x7f745806d510 2023-10-06T05:07:14.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.514+0000 7f745fc0b700 1 -- 172.21.15.98:0/241603522 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7458168600 con 0x7f745806d510 2023-10-06T05:07:14.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.514+0000 7f744dffb700 1 -- 172.21.15.98:0/241603522 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74540173f0 con 0x7f745806d510 2023-10-06T05:07:14.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.514+0000 7f745fc0b700 1 -- 172.21.15.98:0/241603522 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7458168990 con 0x7f745806d510 2023-10-06T05:07:14.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.513+0000 7fc1ef06f700 1 -- 172.21.15.98:0/2827380405 wait complete. 2023-10-06T05:07:14.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.515+0000 7fc1ef06f700 1 Processor -- start 2023-10-06T05:07:14.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.515+0000 7fe97d10b700 1 Processor -- start 2023-10-06T05:07:14.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.515+0000 7f2acffff700 1 -- 172.21.15.98:0/1163746025 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f2ad8011b10 con 0x7f2ae806c160 2023-10-06T05:07:14.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.516+0000 7f2acffff700 1 --2- 172.21.15.98:0/1163746025 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f2ad4066f30 0x7f2ad40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:14.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.516+0000 7f2acffff700 1 -- 172.21.15.98:0/1163746025 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f2ad8086400 con 0x7f2ae806c160 2023-10-06T05:07:14.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.516+0000 7f744dffb700 1 -- 172.21.15.98:0/241603522 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f7454017760 con 0x7f745806d510 2023-10-06T05:07:14.536 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.515+0000 7fc1ef06f700 1 -- start start 2023-10-06T05:07:14.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.516+0000 7fc1ef06f700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1d0003bf0 0x7fc1e816d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:14.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.516+0000 7fc1ef06f700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc1e816d760 0x7fc1e80cabe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:14.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.516+0000 7fc1ef06f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1e80c07c0 0x7fc1e80cb120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:14.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.516+0000 7fc1ef06f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e8177ad0 con 0x7fc1e80c07c0 2023-10-06T05:07:14.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.516+0000 7fc1ef06f700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e81777f0 con 0x7fc1d0003bf0 2023-10-06T05:07:14.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.516+0000 7fc1ef06f700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e80bfce0 con 0x7fc1e816d760 2023-10-06T05:07:14.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.517+0000 7f745fc0b700 1 -- 172.21.15.98:0/241603522 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74380052b0 con 0x7f745806d510 2023-10-06T05:07:14.538 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.517+0000 7fc1e7fff700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc1e816d760 0x7fc1e80cabe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:14.538 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.517+0000 7fc1e7fff700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc1e816d760 0x7fc1e80cabe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.98:56120/0 (socket says 172.21.15.98:56120) 2023-10-06T05:07:14.538 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.517+0000 7fc1e7fff700 1 -- 172.21.15.98:0/3336907641 learned_addr learned my addr 172.21.15.98:0/3336907641 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:14.538 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.517+0000 7f2aee853700 1 -- 172.21.15.98:0/1163746025 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ac80052b0 con 0x7f2ae806c160 2023-10-06T05:07:14.538 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.517+0000 7f744dffb700 1 --2- 172.21.15.98:0/241603522 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f743c066f30 0x7f743c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:14.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.517+0000 7f744dffb700 1 -- 172.21.15.98:0/241603522 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f7454084de0 con 0x7f745806d510 2023-10-06T05:07:14.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.517+0000 7f745d9a7700 1 --2- 172.21.15.98:0/241603522 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f743c066f30 0x7f743c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:14.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.517+0000 7fc1e7fff700 1 -- 172.21.15.98:0/3336907641 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1d0003bf0 msgr2=0x7fc1e816d220 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-06T05:07:14.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.517+0000 7fc1e7fff700 1 --2- 172.21.15.98:0/3336907641 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1d0003bf0 0x7fc1e816d220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:14.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.517+0000 7fc1e7fff700 1 -- 172.21.15.98:0/3336907641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1e80c07c0 msgr2=0x7fc1e80cb120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:14.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.517+0000 7fc1e7fff700 1 --2- 172.21.15.98:0/3336907641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1e80c07c0 0x7fc1e80cb120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:14.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.517+0000 7fc1e7fff700 1 -- 172.21.15.98:0/3336907641 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1e80bffc0 con 0x7fc1e816d760 2023-10-06T05:07:14.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.517+0000 7fc1e7fff700 1 --2- 172.21.15.98:0/3336907641 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc1e816d760 0x7fc1e80cabe0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc1d80107f0 tx=0x7fc1d8008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:14.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.518+0000 7f2ae77fe700 1 --2- 172.21.15.98:0/1163746025 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f2ad4066f30 0x7f2ad40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:14.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.516+0000 7fe97d10b700 1 -- start start 2023-10-06T05:07:14.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.517+0000 7fe97d10b700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe9780685c0 0x7fe9780a5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:14.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.517+0000 7fe97d10b700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe9780b1d10 0x7fe9781414a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:14.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.517+0000 7fe97d10b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe97815d8b0 0x7fe97806ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:14.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.517+0000 7fe97d10b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9780aa9f0 con 0x7fe97815d8b0 2023-10-06T05:07:14.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.517+0000 7fe97d10b700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9780a59d0 con 0x7fe9780685c0 2023-10-06T05:07:14.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.517+0000 7fe97d10b700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe97813dc60 con 0x7fe9780b1d10 2023-10-06T05:07:14.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.517+0000 7fe97659c700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe9780b1d10 0x7fe9781414a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:14.542 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.517+0000 7fe97659c700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe9780b1d10 0x7fe9781414a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.98:56132/0 (socket says 172.21.15.98:56132) 2023-10-06T05:07:14.542 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.517+0000 7fe97659c700 1 -- 172.21.15.98:0/3872589253 learned_addr learned my addr 172.21.15.98:0/3872589253 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:14.542 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.517+0000 7fe976d9d700 1 --2- 172.21.15.98:0/3872589253 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe9780685c0 0x7fe9780a5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:14.542 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.518+0000 7fe97659c700 1 -- 172.21.15.98:0/3872589253 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe9780685c0 msgr2=0x7fe9780a5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:14.542 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.518+0000 7fe97659c700 1 --2- 172.21.15.98:0/3872589253 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe9780685c0 0x7fe9780a5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:14.542 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.518+0000 7fe97659c700 1 -- 172.21.15.98:0/3872589253 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe97815d8b0 msgr2=0x7fe97806ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:14.542 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.518+0000 7fe97659c700 1 --2- 172.21.15.98:0/3872589253 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe97815d8b0 0x7fe97806ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:14.542 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.518+0000 7fe97659c700 1 -- 172.21.15.98:0/3872589253 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9680094f0 con 0x7fe9780b1d10 2023-10-06T05:07:14.542 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.518+0000 7fe97659c700 1 --2- 172.21.15.98:0/3872589253 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe9780b1d10 0x7fe9781414a0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe97806d260 tx=0x7fe968017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:14.542 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.518+0000 7f2ae77fe700 1 --2- 172.21.15.98:0/1163746025 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f2ad4066f30 0x7f2ad40693e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2ae80c6cc0 tx=0x7f2adc0091a0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:14.543 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.522+0000 7f744dffb700 1 -- 172.21.15.98:0/241603522 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7454088030 con 0x7f745806d510 2023-10-06T05:07:14.543 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.522+0000 7fc1e4ff9700 1 -- 172.21.15.98:0/3336907641 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1d8014620 con 0x7fc1e816d760 2023-10-06T05:07:14.543 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.522+0000 7fc1e4ff9700 1 -- 172.21.15.98:0/3336907641 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1d800c050 con 0x7fc1e816d760 2023-10-06T05:07:14.543 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.522+0000 7fc1e4ff9700 1 -- 172.21.15.98:0/3336907641 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1d8014990 con 0x7fc1e816d760 2023-10-06T05:07:14.543 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.523+0000 7fc1ef06f700 1 -- 172.21.15.98:0/3336907641 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1e80c02a0 con 0x7fc1e816d760 2023-10-06T05:07:14.543 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.523+0000 7fc1ef06f700 1 -- 172.21.15.98:0/3336907641 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1e8176ce0 con 0x7fc1e816d760 2023-10-06T05:07:14.544 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.522+0000 7fe95effd700 1 -- 172.21.15.98:0/3872589253 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe96800bbe0 con 0x7fe9780b1d10 2023-10-06T05:07:14.544 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.522+0000 7fe95effd700 1 -- 172.21.15.98:0/3872589253 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe968011420 con 0x7fe9780b1d10 2023-10-06T05:07:14.544 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.522+0000 7fe95effd700 1 -- 172.21.15.98:0/3872589253 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9680117c0 con 0x7fe9780b1d10 2023-10-06T05:07:14.544 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.524+0000 7fc1ef06f700 1 -- 172.21.15.98:0/3336907641 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1e80bfb40 con 0x7fc1e816d760 2023-10-06T05:07:14.545 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.525+0000 7fc1e4ff9700 1 -- 172.21.15.98:0/3336907641 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fc1d8014d00 con 0x7fc1e816d760 2023-10-06T05:07:14.545 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.525+0000 7fc1e4ff9700 1 --2- 172.21.15.98:0/3336907641 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fc1d0068c50 0x7fc1d006b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:14.545 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.526+0000 7fc1e4ff9700 1 -- 172.21.15.98:0/3336907641 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fc1d8084f40 con 0x7fc1e816d760 2023-10-06T05:07:14.545 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.526+0000 7f2acffff700 1 -- 172.21.15.98:0/1163746025 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2ad8053180 con 0x7f2ae806c160 2023-10-06T05:07:14.545 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.527+0000 7fc1ece0b700 1 --2- 172.21.15.98:0/3336907641 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fc1d0068c50 0x7fc1d006b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:14.545 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.527+0000 7fc1ece0b700 1 --2- 172.21.15.98:0/3336907641 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fc1d0068c50 0x7fc1d006b100 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc1e00021d0 tx=0x7fc1e0016000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:14.546 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.527+0000 7f745d9a7700 1 --2- 172.21.15.98:0/241603522 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f743c066f30 0x7f743c0693e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f7448003080 tx=0x7f7448018000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:14.546 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.527+0000 7fe97d10b700 1 -- 172.21.15.98:0/3872589253 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9780a9e20 con 0x7fe9780b1d10 2023-10-06T05:07:14.546 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.527+0000 7fe97d10b700 1 -- 172.21.15.98:0/3872589253 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9780aa3a0 con 0x7fe9780b1d10 2023-10-06T05:07:14.546 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.529+0000 7fe97d10b700 1 -- 172.21.15.98:0/3872589253 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe978161e10 con 0x7fe9780b1d10 2023-10-06T05:07:14.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.529+0000 7fc1e4ff9700 1 -- 172.21.15.98:0/3336907641 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc1d8051d40 con 0x7fc1e816d760 2023-10-06T05:07:14.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.531+0000 7fe95effd700 1 -- 172.21.15.98:0/3872589253 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fe968011bf0 con 0x7fe9780b1d10 2023-10-06T05:07:14.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.531+0000 7fe95effd700 1 --2- 172.21.15.98:0/3872589253 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fe964066f30 0x7fe9640693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:14.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.531+0000 7fe95effd700 1 -- 172.21.15.98:0/3872589253 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fe968086a50 con 0x7fe9780b1d10 2023-10-06T05:07:14.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.531+0000 7fe976d9d700 1 --2- 172.21.15.98:0/3872589253 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fe964066f30 0x7fe9640693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:14.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.532+0000 7fe976d9d700 1 --2- 172.21.15.98:0/3872589253 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fe964066f30 0x7fe9640693e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe97000f510 tx=0x7fe97000b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:14.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.533+0000 7fe95effd700 1 -- 172.21.15.98:0/3872589253 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe968053910 con 0x7fe9780b1d10 2023-10-06T05:07:14.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.713+0000 7f2bb3841700 1 -- 172.21.15.98:0/1627614499 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f2b8c005830 con 0x7f2bac0c6ba0 2023-10-06T05:07:14.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.713+0000 7f2ba97fa700 1 -- 172.21.15.98:0/1627614499 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2ba4055d00 con 0x7f2bac0c6ba0 2023-10-06T05:07:14.715 INFO:teuthology.orchestra.run.smithi098.stdout:64424509448 2023-10-06T05:07:14.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.715+0000 7f2bb3841700 1 -- 172.21.15.98:0/1627614499 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f2b98066f30 msgr2=0x7f2b980693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:14.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.715+0000 7f2bb3841700 1 --2- 172.21.15.98:0/1627614499 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f2b98066f30 0x7f2b980693e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f2bac15ba30 tx=0x7f2b9c006c60 comp rx=0 tx=0).stop 2023-10-06T05:07:14.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.715+0000 7f2bb3841700 1 -- 172.21.15.98:0/1627614499 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2bac0c6ba0 msgr2=0x7f2bac0cabe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:14.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.715+0000 7f2bb3841700 1 --2- 172.21.15.98:0/1627614499 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2bac0c6ba0 0x7f2bac0cabe0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f2ba40014b0 tx=0x7f2ba4001a30 comp rx=0 tx=0).stop 2023-10-06T05:07:14.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.715+0000 7f2bb3841700 1 -- 172.21.15.98:0/1627614499 shutdown_connections 2023-10-06T05:07:14.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.715+0000 7f2bb3841700 1 --2- 172.21.15.98:0/1627614499 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f2b98066f30 0x7f2b980693e0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:14.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.715+0000 7f2bb3841700 1 --2- 172.21.15.98:0/1627614499 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2bac0c6ba0 0x7f2bac0cabe0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:14.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.715+0000 7f2bb3841700 1 --2- 172.21.15.98:0/1627614499 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2bac069520 0x7f2bac0aee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:14.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.715+0000 7f2bb3841700 1 --2- 172.21.15.98:0/1627614499 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2bac15acd0 0x7f2bac14b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:14.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.715+0000 7f2bb3841700 1 -- 172.21.15.98:0/1627614499 >> 172.21.15.98:0/1627614499 conn(0x7f2bac068080 msgr2=0x7f2bac06eef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:14.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.715+0000 7f2bb3841700 1 -- 172.21.15.98:0/1627614499 shutdown_connections 2023-10-06T05:07:14.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.715+0000 7f2bb3841700 1 -- 172.21.15.98:0/1627614499 wait complete. 2023-10-06T05:07:14.732 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509449 got 64424509448 for osd.4 2023-10-06T05:07:14.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.838+0000 7fd1ce7fc700 1 -- 172.21.15.98:0/516069118 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fd1c0005830 con 0x7fd1e813fd00 2023-10-06T05:07:14.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.839+0000 7fd1e4ff9700 1 -- 172.21.15.98:0/516069118 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd1e00568e0 con 0x7fd1e813fd00 2023-10-06T05:07:14.841 INFO:teuthology.orchestra.run.smithi098.stdout:64424509448 2023-10-06T05:07:14.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.841+0000 7fd1ce7fc700 1 -- 172.21.15.98:0/516069118 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fd1d4066f30 msgr2=0x7fd1d40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:14.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.841+0000 7fd1ce7fc700 1 --2- 172.21.15.98:0/516069118 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fd1d4066f30 0x7fd1d40693e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd1e806ac10 tx=0x7fd1d8009040 comp rx=0 tx=0).stop 2023-10-06T05:07:14.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.841+0000 7fd1ce7fc700 1 -- 172.21.15.98:0/516069118 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1e813fd00 msgr2=0x7fd1e8140180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:14.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.841+0000 7fd1ce7fc700 1 --2- 172.21.15.98:0/516069118 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1e813fd00 0x7fd1e8140180 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd1e80accf0 tx=0x7fd1e0017000 comp rx=0 tx=0).stop 2023-10-06T05:07:14.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.842+0000 7fd1ce7fc700 1 -- 172.21.15.98:0/516069118 shutdown_connections 2023-10-06T05:07:14.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.842+0000 7fd1ce7fc700 1 --2- 172.21.15.98:0/516069118 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fd1d4066f30 0x7fd1d40693e0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:14.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.842+0000 7fd1ce7fc700 1 --2- 172.21.15.98:0/516069118 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1e813fd00 0x7fd1e8140180 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:14.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.842+0000 7fd1ce7fc700 1 --2- 172.21.15.98:0/516069118 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd1e80707d0 0x7fd1e8154890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:14.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.842+0000 7fd1ce7fc700 1 --2- 172.21.15.98:0/516069118 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd1e8070c50 0x7fd1e80c2280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:14.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.842+0000 7fd1ce7fc700 1 -- 172.21.15.98:0/516069118 >> 172.21.15.98:0/516069118 conn(0x7fd1e806de40 msgr2=0x7fd1e81669d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:14.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.845+0000 7fd1ce7fc700 1 -- 172.21.15.98:0/516069118 shutdown_connections 2023-10-06T05:07:14.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.845+0000 7fd1ce7fc700 1 -- 172.21.15.98:0/516069118 wait complete. 2023-10-06T05:07:14.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.860+0000 7fc9bfe82700 1 -- 172.21.15.98:0/1867976768 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fc9b8064a90 con 0x7fc9b80cbc20 2023-10-06T05:07:14.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.860+0000 7fc9adffb700 1 -- 172.21.15.98:0/1867976768 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc9b0022020 con 0x7fc9b80cbc20 2023-10-06T05:07:14.862 INFO:teuthology.orchestra.run.smithi098.stdout:64424509449 2023-10-06T05:07:14.866 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509448 got 64424509448 for osd.3 2023-10-06T05:07:14.866 DEBUG:teuthology.parallel:result is None 2023-10-06T05:07:14.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.864+0000 7fc9bfe82700 1 -- 172.21.15.98:0/1867976768 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fc99c067080 msgr2=0x7fc99c069530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:14.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.864+0000 7fc9bfe82700 1 --2- 172.21.15.98:0/1867976768 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fc99c067080 0x7fc99c069530 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc9a8007fd0 tx=0x7fc9a8009040 comp rx=0 tx=0).stop 2023-10-06T05:07:14.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.864+0000 7fc9bfe82700 1 -- 172.21.15.98:0/1867976768 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9b80cbc20 msgr2=0x7fc9b81714f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:14.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.864+0000 7fc9bfe82700 1 --2- 172.21.15.98:0/1867976768 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9b80cbc20 0x7fc9b81714f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc9b000bb80 tx=0x7fc9b0009810 comp rx=0 tx=0).stop 2023-10-06T05:07:14.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.865+0000 7fc9bfe82700 1 -- 172.21.15.98:0/1867976768 shutdown_connections 2023-10-06T05:07:14.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.865+0000 7fc9bfe82700 1 --2- 172.21.15.98:0/1867976768 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fc99c067080 0x7fc99c069530 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:14.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.865+0000 7fc9bfe82700 1 --2- 172.21.15.98:0/1867976768 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9b80cbc20 0x7fc9b81714f0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:14.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.865+0000 7fc9bfe82700 1 --2- 172.21.15.98:0/1867976768 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc9b806e3d0 0x7fc9b814ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:14.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.865+0000 7fc9bfe82700 1 --2- 172.21.15.98:0/1867976768 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc9b80c8260 0x7fc9b80bf340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:14.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.865+0000 7fc9bfe82700 1 -- 172.21.15.98:0/1867976768 >> 172.21.15.98:0/1867976768 conn(0x7fc9b8068080 msgr2=0x7fc9b806a380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:14.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.865+0000 7fc9bfe82700 1 -- 172.21.15.98:0/1867976768 shutdown_connections 2023-10-06T05:07:14.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.865+0000 7fc9bfe82700 1 -- 172.21.15.98:0/1867976768 wait complete. 2023-10-06T05:07:14.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.877+0000 7f745fc0b700 1 -- 172.21.15.98:0/241603522 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f7438005830 con 0x7f745806d510 2023-10-06T05:07:14.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.877+0000 7f744dffb700 1 -- 172.21.15.98:0/241603522 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7454055670 con 0x7f745806d510 2023-10-06T05:07:14.879 INFO:teuthology.orchestra.run.smithi098.stdout:60129542153 2023-10-06T05:07:14.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.879+0000 7f745fc0b700 1 -- 172.21.15.98:0/241603522 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f743c066f30 msgr2=0x7f743c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:14.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.879+0000 7f745fc0b700 1 --2- 172.21.15.98:0/241603522 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f743c066f30 0x7f743c0693e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f7448003080 tx=0x7f7448018000 comp rx=0 tx=0).stop 2023-10-06T05:07:14.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.879+0000 7f745fc0b700 1 -- 172.21.15.98:0/241603522 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f745806d510 msgr2=0x7f745817f7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:14.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.879+0000 7f745fc0b700 1 --2- 172.21.15.98:0/241603522 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f745806d510 0x7f745817f7c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f745400dc70 tx=0x7f745400f040 comp rx=0 tx=0).stop 2023-10-06T05:07:14.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.879+0000 7f745fc0b700 1 -- 172.21.15.98:0/241603522 shutdown_connections 2023-10-06T05:07:14.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.879+0000 7f745fc0b700 1 --2- 172.21.15.98:0/241603522 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f74580c82e0 0x7f745817fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:14.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.879+0000 7f745fc0b700 1 --2- 172.21.15.98:0/241603522 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f743c066f30 0x7f743c0693e0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:14.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.879+0000 7f745fc0b700 1 --2- 172.21.15.98:0/241603522 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7458069520 0x7f74580adaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:14.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.879+0000 7f745fc0b700 1 --2- 172.21.15.98:0/241603522 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f745806d510 0x7f745817f7c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:14.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.879+0000 7f745fc0b700 1 -- 172.21.15.98:0/241603522 >> 172.21.15.98:0/241603522 conn(0x7f7458068080 msgr2=0x7f74580c3ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:14.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.879+0000 7f745fc0b700 1 -- 172.21.15.98:0/241603522 shutdown_connections 2023-10-06T05:07:14.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.879+0000 7f745fc0b700 1 -- 172.21.15.98:0/241603522 wait complete. 2023-10-06T05:07:14.884 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509449 got 64424509449 for osd.5 2023-10-06T05:07:14.884 DEBUG:teuthology.parallel:result is None 2023-10-06T05:07:14.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.883+0000 7fc1ef06f700 1 -- 172.21.15.98:0/3336907641 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fc1e80bfb40 con 0x7fc1e816d760 2023-10-06T05:07:14.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.883+0000 7fc1e4ff9700 1 -- 172.21.15.98:0/3336907641 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc1e80bfb40 con 0x7fc1e816d760 2023-10-06T05:07:14.885 INFO:teuthology.orchestra.run.smithi098.stdout:64424509448 2023-10-06T05:07:14.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.885+0000 7fc1ef06f700 1 -- 172.21.15.98:0/3336907641 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fc1d0068c50 msgr2=0x7fc1d006b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:14.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.885+0000 7fc1ef06f700 1 --2- 172.21.15.98:0/3336907641 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fc1d0068c50 0x7fc1d006b100 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc1e00021d0 tx=0x7fc1e0016000 comp rx=0 tx=0).stop 2023-10-06T05:07:14.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.885+0000 7fc1ef06f700 1 -- 172.21.15.98:0/3336907641 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc1e816d760 msgr2=0x7fc1e80cabe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:14.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.885+0000 7fc1ef06f700 1 --2- 172.21.15.98:0/3336907641 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc1e816d760 0x7fc1e80cabe0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc1d80107f0 tx=0x7fc1d8008040 comp rx=0 tx=0).stop 2023-10-06T05:07:14.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.885+0000 7fc1ef06f700 1 -- 172.21.15.98:0/3336907641 shutdown_connections 2023-10-06T05:07:14.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.885+0000 7fc1ef06f700 1 --2- 172.21.15.98:0/3336907641 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fc1d0068c50 0x7fc1d006b100 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:14.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.885+0000 7fc1ef06f700 1 --2- 172.21.15.98:0/3336907641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1e80c07c0 0x7fc1e80cb120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:14.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.885+0000 7fc1ef06f700 1 --2- 172.21.15.98:0/3336907641 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1d0003bf0 0x7fc1e816d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:14.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.885+0000 7fc1ef06f700 1 --2- 172.21.15.98:0/3336907641 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc1e816d760 0x7fc1e80cabe0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:14.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.885+0000 7fc1ef06f700 1 -- 172.21.15.98:0/3336907641 >> 172.21.15.98:0/3336907641 conn(0x7fc1e8068080 msgr2=0x7fc1e80ad370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:14.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.886+0000 7fc1ef06f700 1 -- 172.21.15.98:0/3336907641 shutdown_connections 2023-10-06T05:07:14.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.886+0000 7fc1ef06f700 1 -- 172.21.15.98:0/3336907641 wait complete. 2023-10-06T05:07:14.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.892+0000 7fe97d10b700 1 -- 172.21.15.98:0/3872589253 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fe978064a90 con 0x7fe9780b1d10 2023-10-06T05:07:14.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.892+0000 7fe95effd700 1 -- 172.21.15.98:0/3872589253 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe968056ef0 con 0x7fe9780b1d10 2023-10-06T05:07:14.895 INFO:teuthology.orchestra.run.smithi098.stdout:60129542154 2023-10-06T05:07:14.896 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542153 for osd.1 2023-10-06T05:07:14.897 DEBUG:teuthology.parallel:result is None 2023-10-06T05:07:14.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.894+0000 7fe97d10b700 1 -- 172.21.15.98:0/3872589253 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fe964066f30 msgr2=0x7fe9640693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:14.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.894+0000 7fe97d10b700 1 --2- 172.21.15.98:0/3872589253 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fe964066f30 0x7fe9640693e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe97000f510 tx=0x7fe97000b040 comp rx=0 tx=0).stop 2023-10-06T05:07:14.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.894+0000 7fe97d10b700 1 -- 172.21.15.98:0/3872589253 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe9780b1d10 msgr2=0x7fe9781414a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:14.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.894+0000 7fe97d10b700 1 --2- 172.21.15.98:0/3872589253 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe9780b1d10 0x7fe9781414a0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe97806d260 tx=0x7fe968017000 comp rx=0 tx=0).stop 2023-10-06T05:07:14.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.894+0000 7fe97d10b700 1 -- 172.21.15.98:0/3872589253 shutdown_connections 2023-10-06T05:07:14.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.894+0000 7fe97d10b700 1 --2- 172.21.15.98:0/3872589253 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fe964066f30 0x7fe9640693e0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:14.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.894+0000 7fe97d10b700 1 --2- 172.21.15.98:0/3872589253 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe97815d8b0 0x7fe97806ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:14.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.894+0000 7fe97d10b700 1 --2- 172.21.15.98:0/3872589253 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe9780685c0 0x7fe9780a5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:14.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.894+0000 7fe97d10b700 1 --2- 172.21.15.98:0/3872589253 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe9780b1d10 0x7fe9781414a0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:14.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.894+0000 7fe97d10b700 1 -- 172.21.15.98:0/3872589253 >> 172.21.15.98:0/3872589253 conn(0x7fe97806de40 msgr2=0x7fe97814ef60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:14.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.895+0000 7fe97d10b700 1 -- 172.21.15.98:0/3872589253 shutdown_connections 2023-10-06T05:07:14.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.895+0000 7fe97d10b700 1 -- 172.21.15.98:0/3872589253 wait complete. 2023-10-06T05:07:14.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.897+0000 7f2aee853700 1 -- 172.21.15.98:0/1163746025 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f2ac8005560 con 0x7f2ae806c160 2023-10-06T05:07:14.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.898+0000 7f2acffff700 1 -- 172.21.15.98:0/1163746025 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2ad8015020 con 0x7f2ae806c160 2023-10-06T05:07:14.900 INFO:teuthology.orchestra.run.smithi098.stdout:64424509449 2023-10-06T05:07:14.902 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509448 got 64424509448 for osd.2 2023-10-06T05:07:14.903 DEBUG:teuthology.parallel:result is None 2023-10-06T05:07:14.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.900+0000 7f2aee853700 1 -- 172.21.15.98:0/1163746025 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f2ad4066f30 msgr2=0x7f2ad40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:14.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.900+0000 7f2aee853700 1 --2- 172.21.15.98:0/1163746025 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f2ad4066f30 0x7f2ad40693e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2ae80c6cc0 tx=0x7f2adc0091a0 comp rx=0 tx=0).stop 2023-10-06T05:07:14.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.900+0000 7f2aee853700 1 -- 172.21.15.98:0/1163746025 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ae806c160 msgr2=0x7f2ae8147ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:14.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.900+0000 7f2aee853700 1 --2- 172.21.15.98:0/1163746025 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ae806c160 0x7f2ae8147ad0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f2ae8156c60 tx=0x7f2ad8017000 comp rx=0 tx=0).stop 2023-10-06T05:07:14.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.900+0000 7f2aee853700 1 -- 172.21.15.98:0/1163746025 shutdown_connections 2023-10-06T05:07:14.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.900+0000 7f2aee853700 1 --2- 172.21.15.98:0/1163746025 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2ae8156350 0x7f2ae80b25d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:14.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.900+0000 7f2aee853700 1 --2- 172.21.15.98:0/1163746025 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f2ad4066f30 0x7f2ad40693e0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:14.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.900+0000 7f2aee853700 1 --2- 172.21.15.98:0/1163746025 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ae806c160 0x7f2ae8147ad0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:14.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.900+0000 7f2aee853700 1 --2- 172.21.15.98:0/1163746025 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2ae80cc710 0x7f2ae8148d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:14.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.900+0000 7f2aee853700 1 -- 172.21.15.98:0/1163746025 >> 172.21.15.98:0/1163746025 conn(0x7f2ae80cdeb0 msgr2=0x7f2ae80c9360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:14.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.900+0000 7f2aee853700 1 -- 172.21.15.98:0/1163746025 shutdown_connections 2023-10-06T05:07:14.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:14.900+0000 7f2aee853700 1 -- 172.21.15.98:0/1163746025 wait complete. 2023-10-06T05:07:14.911 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542153 got 60129542154 for osd.0 2023-10-06T05:07:14.911 DEBUG:teuthology.parallel:result is None 2023-10-06T05:07:14.915 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509449 got 64424509449 for osd.6 2023-10-06T05:07:14.915 DEBUG:teuthology.parallel:result is None 2023-10-06T05:07:15.732 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-10-06T05:07:15.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:15.856+0000 7fca94311700 1 Processor -- start 2023-10-06T05:07:15.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:15.856+0000 7fca94311700 1 -- start start 2023-10-06T05:07:15.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:15.857+0000 7fca94311700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fca8c159560 0x7fca8c0aa530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:15.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:15.857+0000 7fca94311700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fca8c0b49a0 0x7fca8c0aaa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:15.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:15.857+0000 7fca94311700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fca8c0b0760 0x7fca8c0b3890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:15.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:15.857+0000 7fca94311700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca8c0b0c60 con 0x7fca8c0b0760 2023-10-06T05:07:15.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:15.857+0000 7fca94311700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca8c0b0f40 con 0x7fca8c159560 2023-10-06T05:07:15.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:15.857+0000 7fca94311700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca8c0b1220 con 0x7fca8c0b49a0 2023-10-06T05:07:15.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:15.857+0000 7fca928ae700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fca8c0b0760 0x7fca8c0b3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:15.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:15.857+0000 7fca928ae700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fca8c0b0760 0x7fca8c0b3890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45998/0 (socket says 172.21.15.98:45998) 2023-10-06T05:07:15.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:15.857+0000 7fca928ae700 1 -- 172.21.15.98:0/3433183006 learned_addr learned my addr 172.21.15.98:0/3433183006 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:15.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:15.857+0000 7fca918ac700 1 --2- 172.21.15.98:0/3433183006 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fca8c0b49a0 0x7fca8c0aaa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:15.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:15.857+0000 7fca920ad700 1 --2- 172.21.15.98:0/3433183006 >> v2:172.21.15.174:3300/0 conn(0x7fca8c159560 0x7fca8c0aa530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:15.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:15.857+0000 7fca928ae700 1 -- 172.21.15.98:0/3433183006 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fca8c0b49a0 msgr2=0x7fca8c0aaa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:15.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:15.857+0000 7fca928ae700 1 --2- 172.21.15.98:0/3433183006 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fca8c0b49a0 0x7fca8c0aaa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:15.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:15.857+0000 7fca928ae700 1 -- 172.21.15.98:0/3433183006 >> v2:172.21.15.174:3300/0 conn(0x7fca8c159560 msgr2=0x7fca8c0aa530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:15.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:15.857+0000 7fca928ae700 1 --2- 172.21.15.98:0/3433183006 >> v2:172.21.15.174:3300/0 conn(0x7fca8c159560 0x7fca8c0aa530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:15.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:15.857+0000 7fca928ae700 1 -- 172.21.15.98:0/3433183006 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca8c0b1500 con 0x7fca8c0b0760 2023-10-06T05:07:15.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:15.857+0000 7fca928ae700 1 --2- 172.21.15.98:0/3433183006 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca8c0b0760 0x7fca8c0b3890 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fca8c088c80 tx=0x7fca84009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3503a69e318ec5b server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:15.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:15.857+0000 7fca910ab700 1 -- 172.21.15.98:0/3433183006 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca8400e070 con 0x7fca8c0b0760 2023-10-06T05:07:15.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:15.857+0000 7fca910ab700 1 -- 172.21.15.98:0/3433183006 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca8400ba50 con 0x7fca8c0b0760 2023-10-06T05:07:15.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:15.857+0000 7fca910ab700 1 -- 172.21.15.98:0/3433183006 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca840092e0 con 0x7fca8c0b0760 2023-10-06T05:07:15.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:15.858+0000 7fca94311700 1 -- 172.21.15.98:0/3433183006 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca8c0b0760 msgr2=0x7fca8c0b3890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:15.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:15.858+0000 7fca94311700 1 --2- 172.21.15.98:0/3433183006 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca8c0b0760 0x7fca8c0b3890 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fca8c088c80 tx=0x7fca84009840 comp rx=0 tx=0).stop 2023-10-06T05:07:15.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:15.858+0000 7fca94311700 1 -- 172.21.15.98:0/3433183006 shutdown_connections 2023-10-06T05:07:15.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:15.858+0000 7fca94311700 1 --2- 172.21.15.98:0/3433183006 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca8c0b0760 0x7fca8c0b3890 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:15.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:15.858+0000 7fca94311700 1 --2- 172.21.15.98:0/3433183006 >> v2:172.21.15.174:3300/0 conn(0x7fca8c159560 0x7fca8c0aa530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:15.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:15.858+0000 7fca94311700 1 --2- 172.21.15.98:0/3433183006 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fca8c0b49a0 0x7fca8c0aaa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:15.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:15.858+0000 7fca94311700 1 -- 172.21.15.98:0/3433183006 >> 172.21.15.98:0/3433183006 conn(0x7fca8c0997f0 msgr2=0x7fca8c06f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:15.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:15.858+0000 7fca94311700 1 -- 172.21.15.98:0/3433183006 shutdown_connections 2023-10-06T05:07:15.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:15.858+0000 7fca94311700 1 -- 172.21.15.98:0/3433183006 wait complete. 2023-10-06T05:07:15.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:15.859+0000 7fca94311700 1 Processor -- start 2023-10-06T05:07:15.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:15.859+0000 7fca94311700 1 -- start start 2023-10-06T05:07:15.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:15.860+0000 7fca94311700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca8c0b0760 0x7fca8c169760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:15.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:15.860+0000 7fca94311700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fca8c0b49a0 0x7fca8c0af080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:15.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:15.860+0000 7fca94311700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fca8c159560 0x7fca8c155fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:15.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:15.860+0000 7fca94311700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca8c169ca0 con 0x7fca8c0b0760 2023-10-06T05:07:15.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:15.860+0000 7fca94311700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca8c0af5c0 con 0x7fca8c159560 2023-10-06T05:07:15.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:15.860+0000 7fca94311700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca8c156c20 con 0x7fca8c0b49a0 2023-10-06T05:07:15.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:15.860+0000 7fca920ad700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca8c0b0760 0x7fca8c169760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:15.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:15.860+0000 7fca920ad700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca8c0b0760 0x7fca8c169760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46014/0 (socket says 172.21.15.98:46014) 2023-10-06T05:07:15.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:15.860+0000 7fca920ad700 1 -- 172.21.15.98:0/2343944376 learned_addr learned my addr 172.21.15.98:0/2343944376 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:15.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:15.860+0000 7fca918ac700 1 --2- 172.21.15.98:0/2343944376 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fca8c0b49a0 0x7fca8c0af080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:15.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:15.860+0000 7fca928ae700 1 --2- 172.21.15.98:0/2343944376 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fca8c159560 0x7fca8c155fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:15.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:15.860+0000 7fca920ad700 1 -- 172.21.15.98:0/2343944376 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fca8c0b49a0 msgr2=0x7fca8c0af080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:15.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:15.860+0000 7fca920ad700 1 --2- 172.21.15.98:0/2343944376 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fca8c0b49a0 0x7fca8c0af080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:15.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:15.860+0000 7fca920ad700 1 -- 172.21.15.98:0/2343944376 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fca8c159560 msgr2=0x7fca8c155fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:15.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:15.860+0000 7fca920ad700 1 --2- 172.21.15.98:0/2343944376 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fca8c159560 0x7fca8c155fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:15.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:15.860+0000 7fca920ad700 1 -- 172.21.15.98:0/2343944376 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca84009ae0 con 0x7fca8c0b0760 2023-10-06T05:07:15.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:15.860+0000 7fca920ad700 1 --2- 172.21.15.98:0/2343944376 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca8c0b0760 0x7fca8c169760 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fca8800d870 tx=0x7fca88008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:15.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:15.860+0000 7fca827fc700 1 -- 172.21.15.98:0/2343944376 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca88014620 con 0x7fca8c0b0760 2023-10-06T05:07:15.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:15.860+0000 7fca827fc700 1 -- 172.21.15.98:0/2343944376 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca88014780 con 0x7fca8c0b0760 2023-10-06T05:07:15.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:15.860+0000 7fca94311700 1 -- 172.21.15.98:0/2343944376 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca8c156940 con 0x7fca8c0b0760 2023-10-06T05:07:15.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:15.860+0000 7fca94311700 1 -- 172.21.15.98:0/2343944376 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca8c15d900 con 0x7fca8c0b0760 2023-10-06T05:07:15.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:15.860+0000 7fca827fc700 1 -- 172.21.15.98:0/2343944376 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca88014b40 con 0x7fca8c0b0760 2023-10-06T05:07:15.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:15.861+0000 7fca94311700 1 -- 172.21.15.98:0/2343944376 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca8c064a90 con 0x7fca8c0b0760 2023-10-06T05:07:15.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:15.862+0000 7fca827fc700 1 -- 172.21.15.98:0/2343944376 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fca8801d400 con 0x7fca8c0b0760 2023-10-06T05:07:15.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:15.862+0000 7fca827fc700 1 --2- 172.21.15.98:0/2343944376 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fca70066f30 0x7fca700693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:15.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:15.862+0000 7fca827fc700 1 -- 172.21.15.98:0/2343944376 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fca88085120 con 0x7fca8c0b0760 2023-10-06T05:07:15.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:15.862+0000 7fca918ac700 1 --2- 172.21.15.98:0/2343944376 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fca70066f30 0x7fca700693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:15.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:15.863+0000 7fca918ac700 1 --2- 172.21.15.98:0/2343944376 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fca70066f30 0x7fca700693e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fca8c154970 tx=0x7fca7c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:15.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:15.866+0000 7fca827fc700 1 -- 172.21.15.98:0/2343944376 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fca88051f40 con 0x7fca8c0b0760 2023-10-06T05:07:16.057 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.055+0000 7fca94311700 1 -- 172.21.15.98:0/2343944376 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fca8c0b17e0 con 0x7fca8c0b0760 2023-10-06T05:07:16.057 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.055+0000 7fca827fc700 1 -- 172.21.15.98:0/2343944376 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fca88055560 con 0x7fca8c0b0760 2023-10-06T05:07:16.057 INFO:teuthology.orchestra.run.smithi098.stdout:64424509449 2023-10-06T05:07:16.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.057+0000 7fca94311700 1 -- 172.21.15.98:0/2343944376 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fca70066f30 msgr2=0x7fca700693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:16.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.057+0000 7fca94311700 1 --2- 172.21.15.98:0/2343944376 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fca70066f30 0x7fca700693e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fca8c154970 tx=0x7fca7c008040 comp rx=0 tx=0).stop 2023-10-06T05:07:16.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.057+0000 7fca94311700 1 -- 172.21.15.98:0/2343944376 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca8c0b0760 msgr2=0x7fca8c169760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:16.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.057+0000 7fca94311700 1 --2- 172.21.15.98:0/2343944376 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca8c0b0760 0x7fca8c169760 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fca8800d870 tx=0x7fca88008040 comp rx=0 tx=0).stop 2023-10-06T05:07:16.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.057+0000 7fca94311700 1 -- 172.21.15.98:0/2343944376 shutdown_connections 2023-10-06T05:07:16.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.057+0000 7fca94311700 1 --2- 172.21.15.98:0/2343944376 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fca8c159560 0x7fca8c155fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:16.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.057+0000 7fca94311700 1 --2- 172.21.15.98:0/2343944376 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fca70066f30 0x7fca700693e0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:16.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.057+0000 7fca94311700 1 --2- 172.21.15.98:0/2343944376 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca8c0b0760 0x7fca8c169760 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:16.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.057+0000 7fca94311700 1 --2- 172.21.15.98:0/2343944376 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fca8c0b49a0 0x7fca8c0af080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:16.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.057+0000 7fca94311700 1 -- 172.21.15.98:0/2343944376 >> 172.21.15.98:0/2343944376 conn(0x7fca8c0997f0 msgr2=0x7fca8c0b3230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:16.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.057+0000 7fca94311700 1 -- 172.21.15.98:0/2343944376 shutdown_connections 2023-10-06T05:07:16.061 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.057+0000 7fca94311700 1 -- 172.21.15.98:0/2343944376 wait complete. 2023-10-06T05:07:16.071 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509449 got 64424509449 for osd.4 2023-10-06T05:07:16.071 DEBUG:teuthology.parallel:result is None 2023-10-06T05:07:16.072 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-06T05:07:16.072 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-06T05:07:16.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.183+0000 7fb693fef700 1 Processor -- start 2023-10-06T05:07:16.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.183+0000 7fb693fef700 1 -- start start 2023-10-06T05:07:16.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.184+0000 7fb693fef700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fb68c0b49a0 0x7fb68c0aac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:16.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.184+0000 7fb693fef700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb68c0b0c60 con 0x7fb68c0b0760 2023-10-06T05:07:16.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.184+0000 7fb693fef700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb68c0b0f40 con 0x7fb68c159560 2023-10-06T05:07:16.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.184+0000 7fb693fef700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb68c0b1220 con 0x7fb68c0b49a0 2023-10-06T05:07:16.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.184+0000 7fb69258c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fb68c0b0760 0x7fb68c0b1bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:41252/0 (socket says 172.21.15.98:41252) 2023-10-06T05:07:16.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.184+0000 7fb69258c700 1 -- 172.21.15.98:0/3803105266 learned_addr learned my addr 172.21.15.98:0/3803105266 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:07:16.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.184+0000 7fb69158a700 1 --2- 172.21.15.98:0/3803105266 >> v2:172.21.15.174:3300/0 conn(0x7fb68c0b49a0 0x7fb68c0aac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:16.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.184+0000 7fb690d89700 1 -- 172.21.15.98:0/3803105266 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1771286634 0 0) 0x7fb68c0b0c60 con 0x7fb68c0b0760 2023-10-06T05:07:16.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.185+0000 7fb690d89700 1 -- 172.21.15.98:0/3803105266 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb678003570 con 0x7fb68c0b0760 2023-10-06T05:07:16.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.185+0000 7fb69158a700 1 -- 172.21.15.98:0/3803105266 >> v1:172.21.15.174:6789/0 conn(0x7fb68c159560 legacy=0x7fb68c0aa530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:16.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.185+0000 7fb69158a700 1 -- 172.21.15.98:0/3803105266 >> v1:172.21.15.98:6789/0 conn(0x7fb68c0b0760 legacy=0x7fb68c0b1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:16.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.185+0000 7fb69158a700 1 -- 172.21.15.98:0/3803105266 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb68c0b1500 con 0x7fb68c0b49a0 2023-10-06T05:07:16.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.185+0000 7fb69158a700 1 --2- 172.21.15.98:0/3803105266 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb68c0b49a0 0x7fb68c0aac40 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fb67c009910 tx=0x7fb67c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d438ca69e8022ad1 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:16.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.185+0000 7fb690d89700 1 -- 172.21.15.98:0/3803105266 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb67c00b640 con 0x7fb68c0b49a0 2023-10-06T05:07:16.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.185+0000 7fb690d89700 1 -- 172.21.15.98:0/3803105266 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb67c00b7a0 con 0x7fb68c0b49a0 2023-10-06T05:07:16.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.185+0000 7fb690d89700 1 -- 172.21.15.98:0/3803105266 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb67c00bb20 con 0x7fb68c0b49a0 2023-10-06T05:07:16.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.185+0000 7fb693fef700 1 -- 172.21.15.98:0/3803105266 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb68c0b49a0 msgr2=0x7fb68c0aac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:16.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.185+0000 7fb693fef700 1 --2- 172.21.15.98:0/3803105266 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb68c0b49a0 0x7fb68c0aac40 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fb67c009910 tx=0x7fb67c0091a0 comp rx=0 tx=0).stop 2023-10-06T05:07:16.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.186+0000 7fb693fef700 1 -- 172.21.15.98:0/3803105266 shutdown_connections 2023-10-06T05:07:16.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.186+0000 7fb693fef700 1 --2- 172.21.15.98:0/3803105266 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb68c0b49a0 0x7fb68c0aac40 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:16.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.186+0000 7fb693fef700 1 -- 172.21.15.98:0/3803105266 >> 172.21.15.98:0/3803105266 conn(0x7fb68c0997f0 msgr2=0x7fb68c06f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:16.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.186+0000 7fb693fef700 1 -- 172.21.15.98:0/3803105266 shutdown_connections 2023-10-06T05:07:16.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.186+0000 7fb693fef700 1 -- 172.21.15.98:0/3803105266 wait complete. 2023-10-06T05:07:16.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.187+0000 7fb693fef700 1 Processor -- start 2023-10-06T05:07:16.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.187+0000 7fb693fef700 1 -- start start 2023-10-06T05:07:16.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.187+0000 7fb693fef700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb68c0b0760 0x7fb68c15cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:16.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.187+0000 7fb693fef700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb68c0b49a0 0x7fb68c09d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:16.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.187+0000 7fb693fef700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb68c159560 0x7fb68c09c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:16.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.187+0000 7fb693fef700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb68c09dd40 con 0x7fb68c0b0760 2023-10-06T05:07:16.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.187+0000 7fb693fef700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb68c16dc80 con 0x7fb68c0b49a0 2023-10-06T05:07:16.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.187+0000 7fb693fef700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb68c09da60 con 0x7fb68c159560 2023-10-06T05:07:16.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.187+0000 7fb691d8b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb68c0b0760 0x7fb68c15cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:16.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.187+0000 7fb691d8b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb68c0b0760 0x7fb68c15cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46016/0 (socket says 172.21.15.98:46016) 2023-10-06T05:07:16.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.187+0000 7fb691d8b700 1 -- 172.21.15.98:0/1648993415 learned_addr learned my addr 172.21.15.98:0/1648993415 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:16.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.188+0000 7fb69158a700 1 --2- 172.21.15.98:0/1648993415 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb68c0b49a0 0x7fb68c09d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:16.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.188+0000 7fb69258c700 1 --2- 172.21.15.98:0/1648993415 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb68c159560 0x7fb68c09c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:16.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.188+0000 7fb691d8b700 1 -- 172.21.15.98:0/1648993415 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb68c159560 msgr2=0x7fb68c09c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:16.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.188+0000 7fb691d8b700 1 --2- 172.21.15.98:0/1648993415 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb68c159560 0x7fb68c09c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:16.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.188+0000 7fb691d8b700 1 -- 172.21.15.98:0/1648993415 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb68c0b49a0 msgr2=0x7fb68c09d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:16.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.188+0000 7fb691d8b700 1 --2- 172.21.15.98:0/1648993415 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb68c0b49a0 0x7fb68c09d230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:16.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.188+0000 7fb691d8b700 1 -- 172.21.15.98:0/1648993415 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb67c009420 con 0x7fb68c0b0760 2023-10-06T05:07:16.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.188+0000 7fb69158a700 1 --2- 172.21.15.98:0/1648993415 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb68c0b49a0 0x7fb68c09d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:07:16.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.188+0000 7fb691d8b700 1 --2- 172.21.15.98:0/1648993415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb68c0b0760 0x7fb68c15cd10 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fb68800cc70 tx=0x7fb688001af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:16.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.188+0000 7fb681ffb700 1 -- 172.21.15.98:0/1648993415 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb68800ce80 con 0x7fb68c0b0760 2023-10-06T05:07:16.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.188+0000 7fb681ffb700 1 -- 172.21.15.98:0/1648993415 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb68800ab70 con 0x7fb68c0b0760 2023-10-06T05:07:16.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.188+0000 7fb693fef700 1 -- 172.21.15.98:0/1648993415 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb68c16d630 con 0x7fb68c0b0760 2023-10-06T05:07:16.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.188+0000 7fb681ffb700 1 -- 172.21.15.98:0/1648993415 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb688012420 con 0x7fb68c0b0760 2023-10-06T05:07:16.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.188+0000 7fb693fef700 1 -- 172.21.15.98:0/1648993415 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb68c09d770 con 0x7fb68c0b0760 2023-10-06T05:07:16.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.189+0000 7fb693fef700 1 -- 172.21.15.98:0/1648993415 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb68c064a90 con 0x7fb68c0b0760 2023-10-06T05:07:16.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.189+0000 7fb681ffb700 1 -- 172.21.15.98:0/1648993415 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fb688012790 con 0x7fb68c0b0760 2023-10-06T05:07:16.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.190+0000 7fb681ffb700 1 --2- 172.21.15.98:0/1648993415 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fb670066f30 0x7fb6700693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:16.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.190+0000 7fb681ffb700 1 -- 172.21.15.98:0/1648993415 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fb6880850f0 con 0x7fb68c0b0760 2023-10-06T05:07:16.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.190+0000 7fb69158a700 1 --2- 172.21.15.98:0/1648993415 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fb670066f30 0x7fb6700693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:16.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.190+0000 7fb69158a700 1 --2- 172.21.15.98:0/1648993415 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fb670066f30 0x7fb6700693e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb67c009910 tx=0x7fb67c017000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:16.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.194+0000 7fb681ffb700 1 -- 172.21.15.98:0/1648993415 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb688051f10 con 0x7fb68c0b0760 2023-10-06T05:07:16.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.370+0000 7fb693fef700 1 -- 172.21.15.98:0/1648993415 --> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb68c16ddc0 con 0x7fb670066f30 2023-10-06T05:07:16.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.380+0000 7fb681ffb700 1 -- 172.21.15.98:0/1648993415 <== mgr.4104 v2:172.21.15.98:6832/90397 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+367356 (secure 0 0 0) 0x7fb67c000bc0 con 0x7fb670066f30 2023-10-06T05:07:16.387 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:07:16.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.384+0000 7fb693fef700 1 -- 172.21.15.98:0/1648993415 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fb670066f30 msgr2=0x7fb6700693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:16.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.384+0000 7fb693fef700 1 --2- 172.21.15.98:0/1648993415 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fb670066f30 0x7fb6700693e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb67c009910 tx=0x7fb67c017000 comp rx=0 tx=0).stop 2023-10-06T05:07:16.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.384+0000 7fb693fef700 1 -- 172.21.15.98:0/1648993415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb68c0b0760 msgr2=0x7fb68c15cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:16.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.384+0000 7fb693fef700 1 --2- 172.21.15.98:0/1648993415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb68c0b0760 0x7fb68c15cd10 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fb68800cc70 tx=0x7fb688001af0 comp rx=0 tx=0).stop 2023-10-06T05:07:16.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.384+0000 7fb693fef700 1 -- 172.21.15.98:0/1648993415 shutdown_connections 2023-10-06T05:07:16.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.384+0000 7fb693fef700 1 --2- 172.21.15.98:0/1648993415 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb68c159560 0x7fb68c09c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:16.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.384+0000 7fb693fef700 1 --2- 172.21.15.98:0/1648993415 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fb670066f30 0x7fb6700693e0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:16.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.384+0000 7fb693fef700 1 --2- 172.21.15.98:0/1648993415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb68c0b0760 0x7fb68c15cd10 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:16.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.384+0000 7fb693fef700 1 --2- 172.21.15.98:0/1648993415 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb68c0b49a0 0x7fb68c09d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:16.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.384+0000 7fb693fef700 1 -- 172.21.15.98:0/1648993415 >> 172.21.15.98:0/1648993415 conn(0x7fb68c0997f0 msgr2=0x7fb68c0cf050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:16.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.384+0000 7fb693fef700 1 -- 172.21.15.98:0/1648993415 shutdown_connections 2023-10-06T05:07:16.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.384+0000 7fb693fef700 1 -- 172.21.15.98:0/1648993415 wait complete. 2023-10-06T05:07:16.391 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2023-10-06T05:07:16.398 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":38,"stamp":"2023-10-06T05:07:14.523063+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594852,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":338,"acting":338,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":338,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2381928,"kb_used_data":3624,"kb_used_omap":0,"kb_used_meta":2378240,"kb_avail":752592792,"statfs":{"total":773094113280,"available":770655019008,"internally_reserved":0,"allocated":3710976,"data_stored":2198496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2435317760},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2178,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.794615"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.753513+0000","last_change":"2023-10-06T05:06:57.619460+0000","last_active":"2023-10-06T05:07:01.753513+0000","last_peered":"2023-10-06T05:07:01.753513+0000","last_clean":"2023-10-06T05:07:01.753513+0000","last_became_active":"2023-10-06T05:06:57.619199+0000","last_became_peered":"2023-10-06T05:06:57.619199+0000","last_unstale":"2023-10-06T05:07:01.753513+0000","last_undegraded":"2023-10-06T05:07:01.753513+0000","last_fullsized":"2023-10-06T05:07:01.753513+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:49:28.024174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.747458+0000","last_change":"2023-10-06T05:06:57.610280+0000","last_active":"2023-10-06T05:07:01.747458+0000","last_peered":"2023-10-06T05:07:01.747458+0000","last_clean":"2023-10-06T05:07:01.747458+0000","last_became_active":"2023-10-06T05:06:57.609973+0000","last_became_peered":"2023-10-06T05:06:57.609973+0000","last_unstale":"2023-10-06T05:07:01.747458+0000","last_undegraded":"2023-10-06T05:07:01.747458+0000","last_fullsized":"2023-10-06T05:07:01.747458+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:22:11.051575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.762104+0000","last_change":"2023-10-06T05:06:57.615468+0000","last_active":"2023-10-06T05:07:01.762104+0000","last_peered":"2023-10-06T05:07:01.762104+0000","last_clean":"2023-10-06T05:07:01.762104+0000","last_became_active":"2023-10-06T05:06:57.615221+0000","last_became_peered":"2023-10-06T05:06:57.615221+0000","last_unstale":"2023-10-06T05:07:01.762104+0000","last_undegraded":"2023-10-06T05:07:01.762104+0000","last_fullsized":"2023-10-06T05:07:01.762104+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:49:48.309158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.754474+0000","last_change":"2023-10-06T05:06:57.614470+0000","last_active":"2023-10-06T05:07:01.754474+0000","last_peered":"2023-10-06T05:07:01.754474+0000","last_clean":"2023-10-06T05:07:01.754474+0000","last_became_active":"2023-10-06T05:06:57.614169+0000","last_became_peered":"2023-10-06T05:06:57.614169+0000","last_unstale":"2023-10-06T05:07:01.754474+0000","last_undegraded":"2023-10-06T05:07:01.754474+0000","last_fullsized":"2023-10-06T05:07:01.754474+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:23:03.302621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750277+0000","last_change":"2023-10-06T05:06:57.622050+0000","last_active":"2023-10-06T05:07:01.750277+0000","last_peered":"2023-10-06T05:07:01.750277+0000","last_clean":"2023-10-06T05:07:01.750277+0000","last_became_active":"2023-10-06T05:06:57.621777+0000","last_became_peered":"2023-10-06T05:06:57.621777+0000","last_unstale":"2023-10-06T05:07:01.750277+0000","last_undegraded":"2023-10-06T05:07:01.750277+0000","last_fullsized":"2023-10-06T05:07:01.750277+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:04:40.577463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.761680+0000","last_change":"2023-10-06T05:06:57.611449+0000","last_active":"2023-10-06T05:07:01.761680+0000","last_peered":"2023-10-06T05:07:01.761680+0000","last_clean":"2023-10-06T05:07:01.761680+0000","last_became_active":"2023-10-06T05:06:57.610301+0000","last_became_peered":"2023-10-06T05:06:57.610301+0000","last_unstale":"2023-10-06T05:07:01.761680+0000","last_undegraded":"2023-10-06T05:07:01.761680+0000","last_fullsized":"2023-10-06T05:07:01.761680+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:43:21.654853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.755118+0000","last_change":"2023-10-06T05:06:57.610678+0000","last_active":"2023-10-06T05:07:01.755118+0000","last_peered":"2023-10-06T05:07:01.755118+0000","last_clean":"2023-10-06T05:07:01.755118+0000","last_became_active":"2023-10-06T05:06:57.609746+0000","last_became_peered":"2023-10-06T05:06:57.609746+0000","last_unstale":"2023-10-06T05:07:01.755118+0000","last_undegraded":"2023-10-06T05:07:01.755118+0000","last_fullsized":"2023-10-06T05:07:01.755118+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:11:00.525585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.745457+0000","last_change":"2023-10-06T05:06:57.612559+0000","last_active":"2023-10-06T05:07:01.745457+0000","last_peered":"2023-10-06T05:07:01.745457+0000","last_clean":"2023-10-06T05:07:01.745457+0000","last_became_active":"2023-10-06T05:06:57.612199+0000","last_became_peered":"2023-10-06T05:06:57.612199+0000","last_unstale":"2023-10-06T05:07:01.745457+0000","last_undegraded":"2023-10-06T05:07:01.745457+0000","last_fullsized":"2023-10-06T05:07:01.745457+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:01:11.983432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.751122+0000","last_change":"2023-10-06T05:06:57.618496+0000","last_active":"2023-10-06T05:07:01.751122+0000","last_peered":"2023-10-06T05:07:01.751122+0000","last_clean":"2023-10-06T05:07:01.751122+0000","last_became_active":"2023-10-06T05:06:57.618261+0000","last_became_peered":"2023-10-06T05:06:57.618261+0000","last_unstale":"2023-10-06T05:07:01.751122+0000","last_undegraded":"2023-10-06T05:07:01.751122+0000","last_fullsized":"2023-10-06T05:07:01.751122+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:24:33.077189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.747636+0000","last_change":"2023-10-06T05:06:57.614417+0000","last_active":"2023-10-06T05:07:01.747636+0000","last_peered":"2023-10-06T05:07:01.747636+0000","last_clean":"2023-10-06T05:07:01.747636+0000","last_became_active":"2023-10-06T05:06:57.614144+0000","last_became_peered":"2023-10-06T05:06:57.614144+0000","last_unstale":"2023-10-06T05:07:01.747636+0000","last_undegraded":"2023-10-06T05:07:01.747636+0000","last_fullsized":"2023-10-06T05:07:01.747636+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:32:33.731449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.762013+0000","last_change":"2023-10-06T05:06:57.610272+0000","last_active":"2023-10-06T05:07:01.762013+0000","last_peered":"2023-10-06T05:07:01.762013+0000","last_clean":"2023-10-06T05:07:01.762013+0000","last_became_active":"2023-10-06T05:06:57.609985+0000","last_became_peered":"2023-10-06T05:06:57.609985+0000","last_unstale":"2023-10-06T05:07:01.762013+0000","last_undegraded":"2023-10-06T05:07:01.762013+0000","last_fullsized":"2023-10-06T05:07:01.762013+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:20:04.673613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.745990+0000","last_change":"2023-10-06T05:06:57.606852+0000","last_active":"2023-10-06T05:07:01.745990+0000","last_peered":"2023-10-06T05:07:01.745990+0000","last_clean":"2023-10-06T05:07:01.745990+0000","last_became_active":"2023-10-06T05:06:57.606582+0000","last_became_peered":"2023-10-06T05:06:57.606582+0000","last_unstale":"2023-10-06T05:07:01.745990+0000","last_undegraded":"2023-10-06T05:07:01.745990+0000","last_fullsized":"2023-10-06T05:07:01.745990+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:23:03.302621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.753767+0000","last_change":"2023-10-06T05:06:57.615706+0000","last_active":"2023-10-06T05:07:01.753767+0000","last_peered":"2023-10-06T05:07:01.753767+0000","last_clean":"2023-10-06T05:07:01.753767+0000","last_became_active":"2023-10-06T05:06:57.615408+0000","last_became_peered":"2023-10-06T05:06:57.615408+0000","last_unstale":"2023-10-06T05:07:01.753767+0000","last_undegraded":"2023-10-06T05:07:01.753767+0000","last_fullsized":"2023-10-06T05:07:01.753767+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:38:51.879514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.744412+0000","last_change":"2023-10-06T05:06:57.613222+0000","last_active":"2023-10-06T05:07:01.744412+0000","last_peered":"2023-10-06T05:07:01.744412+0000","last_clean":"2023-10-06T05:07:01.744412+0000","last_became_active":"2023-10-06T05:06:57.612924+0000","last_became_peered":"2023-10-06T05:06:57.612924+0000","last_unstale":"2023-10-06T05:07:01.744412+0000","last_undegraded":"2023-10-06T05:07:01.744412+0000","last_fullsized":"2023-10-06T05:07:01.744412+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:29:19.951056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.754947+0000","last_change":"2023-10-06T05:06:57.610023+0000","last_active":"2023-10-06T05:07:01.754947+0000","last_peered":"2023-10-06T05:07:01.754947+0000","last_clean":"2023-10-06T05:07:01.754947+0000","last_became_active":"2023-10-06T05:06:57.609367+0000","last_became_peered":"2023-10-06T05:06:57.609367+0000","last_unstale":"2023-10-06T05:07:01.754947+0000","last_undegraded":"2023-10-06T05:07:01.754947+0000","last_fullsized":"2023-10-06T05:07:01.754947+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:34:55.987076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.758813+0000","last_change":"2023-10-06T05:06:57.610369+0000","last_active":"2023-10-06T05:07:01.758813+0000","last_peered":"2023-10-06T05:07:01.758813+0000","last_clean":"2023-10-06T05:07:01.758813+0000","last_became_active":"2023-10-06T05:06:57.610044+0000","last_became_peered":"2023-10-06T05:06:57.610044+0000","last_unstale":"2023-10-06T05:07:01.758813+0000","last_undegraded":"2023-10-06T05:07:01.758813+0000","last_fullsized":"2023-10-06T05:07:01.758813+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:10:12.061112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.751198+0000","last_change":"2023-10-06T05:06:57.619477+0000","last_active":"2023-10-06T05:07:01.751198+0000","last_peered":"2023-10-06T05:07:01.751198+0000","last_clean":"2023-10-06T05:07:01.751198+0000","last_became_active":"2023-10-06T05:06:57.618021+0000","last_became_peered":"2023-10-06T05:06:57.618021+0000","last_unstale":"2023-10-06T05:07:01.751198+0000","last_undegraded":"2023-10-06T05:07:01.751198+0000","last_fullsized":"2023-10-06T05:07:01.751198+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:58:02.572423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.761093+0000","last_change":"2023-10-06T05:06:57.602802+0000","last_active":"2023-10-06T05:07:01.761093+0000","last_peered":"2023-10-06T05:07:01.761093+0000","last_clean":"2023-10-06T05:07:01.761093+0000","last_became_active":"2023-10-06T05:06:57.602513+0000","last_became_peered":"2023-10-06T05:06:57.602513+0000","last_unstale":"2023-10-06T05:07:01.761093+0000","last_undegraded":"2023-10-06T05:07:01.761093+0000","last_fullsized":"2023-10-06T05:07:01.761093+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T16:34:15.990651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.747139+0000","last_change":"2023-10-06T05:06:57.608190+0000","last_active":"2023-10-06T05:07:01.747139+0000","last_peered":"2023-10-06T05:07:01.747139+0000","last_clean":"2023-10-06T05:07:01.747139+0000","last_became_active":"2023-10-06T05:06:57.607936+0000","last_became_peered":"2023-10-06T05:06:57.607936+0000","last_unstale":"2023-10-06T05:07:01.747139+0000","last_undegraded":"2023-10-06T05:07:01.747139+0000","last_fullsized":"2023-10-06T05:07:01.747139+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:00:30.035770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.761424+0000","last_change":"2023-10-06T05:06:57.611226+0000","last_active":"2023-10-06T05:07:01.761424+0000","last_peered":"2023-10-06T05:07:01.761424+0000","last_clean":"2023-10-06T05:07:01.761424+0000","last_became_active":"2023-10-06T05:06:57.610931+0000","last_became_peered":"2023-10-06T05:06:57.610931+0000","last_unstale":"2023-10-06T05:07:01.761424+0000","last_undegraded":"2023-10-06T05:07:01.761424+0000","last_fullsized":"2023-10-06T05:07:01.761424+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:52:14.772801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.749471+0000","last_change":"2023-10-06T05:06:57.615682+0000","last_active":"2023-10-06T05:07:01.749471+0000","last_peered":"2023-10-06T05:07:01.749471+0000","last_clean":"2023-10-06T05:07:01.749471+0000","last_became_active":"2023-10-06T05:06:57.615408+0000","last_became_peered":"2023-10-06T05:06:57.615408+0000","last_unstale":"2023-10-06T05:07:01.749471+0000","last_undegraded":"2023-10-06T05:07:01.749471+0000","last_fullsized":"2023-10-06T05:07:01.749471+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:58:02.572423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.749868+0000","last_change":"2023-10-06T05:06:57.619795+0000","last_active":"2023-10-06T05:07:01.749868+0000","last_peered":"2023-10-06T05:07:01.749868+0000","last_clean":"2023-10-06T05:07:01.749868+0000","last_became_active":"2023-10-06T05:06:57.619533+0000","last_became_peered":"2023-10-06T05:06:57.619533+0000","last_unstale":"2023-10-06T05:07:01.749868+0000","last_undegraded":"2023-10-06T05:07:01.749868+0000","last_fullsized":"2023-10-06T05:07:01.749868+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:11:00.525585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750332+0000","last_change":"2023-10-06T05:06:57.617926+0000","last_active":"2023-10-06T05:07:01.750332+0000","last_peered":"2023-10-06T05:07:01.750332+0000","last_clean":"2023-10-06T05:07:01.750332+0000","last_became_active":"2023-10-06T05:06:57.617128+0000","last_became_peered":"2023-10-06T05:06:57.617128+0000","last_unstale":"2023-10-06T05:07:01.750332+0000","last_undegraded":"2023-10-06T05:07:01.750332+0000","last_fullsized":"2023-10-06T05:07:01.750332+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:38:16.871694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.748479+0000","last_change":"2023-10-06T05:06:57.620090+0000","last_active":"2023-10-06T05:07:01.748479+0000","last_peered":"2023-10-06T05:07:01.748479+0000","last_clean":"2023-10-06T05:07:01.748479+0000","last_became_active":"2023-10-06T05:06:57.619892+0000","last_became_peered":"2023-10-06T05:06:57.619892+0000","last_unstale":"2023-10-06T05:07:01.748479+0000","last_undegraded":"2023-10-06T05:07:01.748479+0000","last_fullsized":"2023-10-06T05:07:01.748479+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:45:23.272110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750371+0000","last_change":"2023-10-06T05:06:57.613844+0000","last_active":"2023-10-06T05:07:01.750371+0000","last_peered":"2023-10-06T05:07:01.750371+0000","last_clean":"2023-10-06T05:07:01.750371+0000","last_became_active":"2023-10-06T05:06:57.613513+0000","last_became_peered":"2023-10-06T05:06:57.613513+0000","last_unstale":"2023-10-06T05:07:01.750371+0000","last_undegraded":"2023-10-06T05:07:01.750371+0000","last_fullsized":"2023-10-06T05:07:01.750371+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:53:03.171108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.748759+0000","last_change":"2023-10-06T05:06:57.612464+0000","last_active":"2023-10-06T05:07:01.748759+0000","last_peered":"2023-10-06T05:07:01.748759+0000","last_clean":"2023-10-06T05:07:01.748759+0000","last_became_active":"2023-10-06T05:06:57.612179+0000","last_became_peered":"2023-10-06T05:06:57.612179+0000","last_unstale":"2023-10-06T05:07:01.748759+0000","last_undegraded":"2023-10-06T05:07:01.748759+0000","last_fullsized":"2023-10-06T05:07:01.748759+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:12:34.550298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.744832+0000","last_change":"2023-10-06T05:06:57.612625+0000","last_active":"2023-10-06T05:07:01.744832+0000","last_peered":"2023-10-06T05:07:01.744832+0000","last_clean":"2023-10-06T05:07:01.744832+0000","last_became_active":"2023-10-06T05:06:57.611582+0000","last_became_peered":"2023-10-06T05:06:57.611582+0000","last_unstale":"2023-10-06T05:07:01.744832+0000","last_undegraded":"2023-10-06T05:07:01.744832+0000","last_fullsized":"2023-10-06T05:07:01.744832+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:59:39.994965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.768197+0000","last_change":"2023-10-06T05:07:01.765965+0000","last_active":"2023-10-06T05:07:01.768197+0000","last_peered":"2023-10-06T05:07:01.768197+0000","last_clean":"2023-10-06T05:07:01.768197+0000","last_became_active":"2023-10-06T05:07:01.764635+0000","last_became_peered":"2023-10-06T05:07:01.764635+0000","last_unstale":"2023-10-06T05:07:01.768197+0000","last_undegraded":"2023-10-06T05:07:01.768197+0000","last_fullsized":"2023-10-06T05:07:01.768197+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:18:43.042335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2,0],"acting":[4,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"24'1","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.760983+0000","last_change":"2023-10-06T05:06:56.716498+0000","last_active":"2023-10-06T05:07:01.760983+0000","last_peered":"2023-10-06T05:07:01.760983+0000","last_clean":"2023-10-06T05:07:01.760983+0000","last_became_active":"2023-10-06T05:06:56.714771+0000","last_became_peered":"2023-10-06T05:06:56.714771+0000","last_unstale":"2023-10-06T05:07:01.760983+0000","last_undegraded":"2023-10-06T05:07:01.760983+0000","last_fullsized":"2023-10-06T05:07:01.760983+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:53:56.989908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.752793+0000","last_change":"2023-10-06T05:06:57.617712+0000","last_active":"2023-10-06T05:07:01.752793+0000","last_peered":"2023-10-06T05:07:01.752793+0000","last_clean":"2023-10-06T05:07:01.752793+0000","last_became_active":"2023-10-06T05:06:57.616946+0000","last_became_peered":"2023-10-06T05:06:57.616946+0000","last_unstale":"2023-10-06T05:07:01.752793+0000","last_undegraded":"2023-10-06T05:07:01.752793+0000","last_fullsized":"2023-10-06T05:07:01.752793+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:21:13.192156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.767576+0000","last_change":"2023-10-06T05:07:01.763425+0000","last_active":"2023-10-06T05:07:01.767576+0000","last_peered":"2023-10-06T05:07:01.767576+0000","last_clean":"2023-10-06T05:07:01.767576+0000","last_became_active":"2023-10-06T05:07:01.763035+0000","last_became_peered":"2023-10-06T05:07:01.763035+0000","last_unstale":"2023-10-06T05:07:01.767576+0000","last_undegraded":"2023-10-06T05:07:01.767576+0000","last_fullsized":"2023-10-06T05:07:01.767576+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:51:19.615655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3,7],"acting":[6,4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.745972+0000","last_change":"2023-10-06T05:06:56.714205+0000","last_active":"2023-10-06T05:07:01.745972+0000","last_peered":"2023-10-06T05:07:01.745972+0000","last_clean":"2023-10-06T05:07:01.745972+0000","last_became_active":"2023-10-06T05:06:56.713947+0000","last_became_peered":"2023-10-06T05:06:56.713947+0000","last_unstale":"2023-10-06T05:07:01.745972+0000","last_undegraded":"2023-10-06T05:07:01.745972+0000","last_fullsized":"2023-10-06T05:07:01.745972+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:54:07.584005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750046+0000","last_change":"2023-10-06T05:06:57.616019+0000","last_active":"2023-10-06T05:07:01.750046+0000","last_peered":"2023-10-06T05:07:01.750046+0000","last_clean":"2023-10-06T05:07:01.750046+0000","last_became_active":"2023-10-06T05:06:57.615747+0000","last_became_peered":"2023-10-06T05:06:57.615747+0000","last_unstale":"2023-10-06T05:07:01.750046+0000","last_undegraded":"2023-10-06T05:07:01.750046+0000","last_fullsized":"2023-10-06T05:07:01.750046+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:07:52.020683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.770991+0000","last_change":"2023-10-06T05:07:01.767383+0000","last_active":"2023-10-06T05:07:01.770991+0000","last_peered":"2023-10-06T05:07:01.770991+0000","last_clean":"2023-10-06T05:07:01.770991+0000","last_became_active":"2023-10-06T05:07:01.766997+0000","last_became_peered":"2023-10-06T05:07:01.766997+0000","last_unstale":"2023-10-06T05:07:01.770991+0000","last_undegraded":"2023-10-06T05:07:01.770991+0000","last_fullsized":"2023-10-06T05:07:01.770991+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:45:03.209108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1,2],"acting":[0,7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.749601+0000","last_change":"2023-10-06T05:06:56.706133+0000","last_active":"2023-10-06T05:07:01.749601+0000","last_peered":"2023-10-06T05:07:01.749601+0000","last_clean":"2023-10-06T05:07:01.749601+0000","last_became_active":"2023-10-06T05:06:56.705867+0000","last_became_peered":"2023-10-06T05:06:56.705867+0000","last_unstale":"2023-10-06T05:07:01.749601+0000","last_undegraded":"2023-10-06T05:07:01.749601+0000","last_fullsized":"2023-10-06T05:07:01.749601+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:55:08.124129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.751243+0000","last_change":"2023-10-06T05:06:57.608813+0000","last_active":"2023-10-06T05:07:01.751243+0000","last_peered":"2023-10-06T05:07:01.751243+0000","last_clean":"2023-10-06T05:07:01.751243+0000","last_became_active":"2023-10-06T05:06:57.608497+0000","last_became_peered":"2023-10-06T05:06:57.608497+0000","last_unstale":"2023-10-06T05:07:01.751243+0000","last_undegraded":"2023-10-06T05:07:01.751243+0000","last_fullsized":"2023-10-06T05:07:01.751243+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:30:19.509164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.770179+0000","last_change":"2023-10-06T05:07:01.765801+0000","last_active":"2023-10-06T05:07:01.770179+0000","last_peered":"2023-10-06T05:07:01.770179+0000","last_clean":"2023-10-06T05:07:01.770179+0000","last_became_active":"2023-10-06T05:07:01.765422+0000","last_became_peered":"2023-10-06T05:07:01.765422+0000","last_unstale":"2023-10-06T05:07:01.770179+0000","last_undegraded":"2023-10-06T05:07:01.770179+0000","last_fullsized":"2023-10-06T05:07:01.770179+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:20:08.719247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7,1],"acting":[6,4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.751276+0000","last_change":"2023-10-06T05:06:56.714826+0000","last_active":"2023-10-06T05:07:01.751276+0000","last_peered":"2023-10-06T05:07:01.751276+0000","last_clean":"2023-10-06T05:07:01.751276+0000","last_became_active":"2023-10-06T05:06:56.714564+0000","last_became_peered":"2023-10-06T05:06:56.714564+0000","last_unstale":"2023-10-06T05:07:01.751276+0000","last_undegraded":"2023-10-06T05:07:01.751276+0000","last_fullsized":"2023-10-06T05:07:01.751276+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:19:13.930707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.749520+0000","last_change":"2023-10-06T05:06:57.618934+0000","last_active":"2023-10-06T05:07:01.749520+0000","last_peered":"2023-10-06T05:07:01.749520+0000","last_clean":"2023-10-06T05:07:01.749520+0000","last_became_active":"2023-10-06T05:06:57.618410+0000","last_became_peered":"2023-10-06T05:06:57.618410+0000","last_unstale":"2023-10-06T05:07:01.749520+0000","last_undegraded":"2023-10-06T05:07:01.749520+0000","last_fullsized":"2023-10-06T05:07:01.749520+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:51:15.570020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.775608+0000","last_change":"2023-10-06T05:07:01.766370+0000","last_active":"2023-10-06T05:07:01.775608+0000","last_peered":"2023-10-06T05:07:01.775608+0000","last_clean":"2023-10-06T05:07:01.775608+0000","last_became_active":"2023-10-06T05:07:01.765994+0000","last_became_peered":"2023-10-06T05:07:01.765994+0000","last_unstale":"2023-10-06T05:07:01.775608+0000","last_undegraded":"2023-10-06T05:07:01.775608+0000","last_fullsized":"2023-10-06T05:07:01.775608+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:20:46.564009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6,0],"acting":[7,4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.747549+0000","last_change":"2023-10-06T05:06:56.708479+0000","last_active":"2023-10-06T05:07:01.747549+0000","last_peered":"2023-10-06T05:07:01.747549+0000","last_clean":"2023-10-06T05:07:01.747549+0000","last_became_active":"2023-10-06T05:06:56.708155+0000","last_became_peered":"2023-10-06T05:06:56.708155+0000","last_unstale":"2023-10-06T05:07:01.747549+0000","last_undegraded":"2023-10-06T05:07:01.747549+0000","last_fullsized":"2023-10-06T05:07:01.747549+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:30:18.507322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750417+0000","last_change":"2023-10-06T05:06:57.617995+0000","last_active":"2023-10-06T05:07:01.750417+0000","last_peered":"2023-10-06T05:07:01.750417+0000","last_clean":"2023-10-06T05:07:01.750417+0000","last_became_active":"2023-10-06T05:06:57.617577+0000","last_became_peered":"2023-10-06T05:06:57.617577+0000","last_unstale":"2023-10-06T05:07:01.750417+0000","last_undegraded":"2023-10-06T05:07:01.750417+0000","last_fullsized":"2023-10-06T05:07:01.750417+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:01:11.983432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.763161+0000","last_change":"2023-10-06T05:07:01.761436+0000","last_active":"2023-10-06T05:07:01.763161+0000","last_peered":"2023-10-06T05:07:01.763161+0000","last_clean":"2023-10-06T05:07:01.763161+0000","last_became_active":"2023-10-06T05:07:01.761049+0000","last_became_peered":"2023-10-06T05:07:01.761049+0000","last_unstale":"2023-10-06T05:07:01.763161+0000","last_undegraded":"2023-10-06T05:07:01.763161+0000","last_fullsized":"2023-10-06T05:07:01.763161+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:09:13.387478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,5],"acting":[3,0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"24'3","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.753446+0000","last_change":"2023-10-06T05:06:56.717320+0000","last_active":"2023-10-06T05:07:01.753446+0000","last_peered":"2023-10-06T05:07:01.753446+0000","last_clean":"2023-10-06T05:07:01.753446+0000","last_became_active":"2023-10-06T05:06:56.716985+0000","last_became_peered":"2023-10-06T05:06:56.716985+0000","last_unstale":"2023-10-06T05:07:01.753446+0000","last_undegraded":"2023-10-06T05:07:01.753446+0000","last_fullsized":"2023-10-06T05:07:01.753446+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:53:02.169267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.747499+0000","last_change":"2023-10-06T05:06:57.613655+0000","last_active":"2023-10-06T05:07:01.747499+0000","last_peered":"2023-10-06T05:07:01.747499+0000","last_clean":"2023-10-06T05:07:01.747499+0000","last_became_active":"2023-10-06T05:06:57.613378+0000","last_became_peered":"2023-10-06T05:06:57.613378+0000","last_unstale":"2023-10-06T05:07:01.747499+0000","last_undegraded":"2023-10-06T05:07:01.747499+0000","last_fullsized":"2023-10-06T05:07:01.747499+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:23:03.919621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.772389+0000","last_change":"2023-10-06T05:07:01.766400+0000","last_active":"2023-10-06T05:07:01.772389+0000","last_peered":"2023-10-06T05:07:01.772389+0000","last_clean":"2023-10-06T05:07:01.772389+0000","last_became_active":"2023-10-06T05:07:01.766034+0000","last_became_peered":"2023-10-06T05:07:01.766034+0000","last_unstale":"2023-10-06T05:07:01.772389+0000","last_undegraded":"2023-10-06T05:07:01.772389+0000","last_fullsized":"2023-10-06T05:07:01.772389+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:04:14.761320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3,7],"acting":[1,5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"29'9","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-06T05:07:07.500759+0000","last_change":"2023-10-06T05:06:56.716406+0000","last_active":"2023-10-06T05:07:07.500759+0000","last_peered":"2023-10-06T05:07:07.500759+0000","last_clean":"2023-10-06T05:07:07.500759+0000","last_became_active":"2023-10-06T05:06:56.716075+0000","last_became_peered":"2023-10-06T05:06:56.716075+0000","last_unstale":"2023-10-06T05:07:07.500759+0000","last_undegraded":"2023-10-06T05:07:07.500759+0000","last_fullsized":"2023-10-06T05:07:07.500759+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:36:39.323759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3714,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.761355+0000","last_change":"2023-10-06T05:06:57.605798+0000","last_active":"2023-10-06T05:07:01.761355+0000","last_peered":"2023-10-06T05:07:01.761355+0000","last_clean":"2023-10-06T05:07:01.761355+0000","last_became_active":"2023-10-06T05:06:57.605523+0000","last_became_peered":"2023-10-06T05:06:57.605523+0000","last_unstale":"2023-10-06T05:07:01.761355+0000","last_undegraded":"2023-10-06T05:07:01.761355+0000","last_fullsized":"2023-10-06T05:07:01.761355+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:32:33.731449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.765132+0000","last_change":"2023-10-06T05:07:01.760576+0000","last_active":"2023-10-06T05:07:01.765132+0000","last_peered":"2023-10-06T05:07:01.765132+0000","last_clean":"2023-10-06T05:07:01.765132+0000","last_became_active":"2023-10-06T05:07:01.760146+0000","last_became_peered":"2023-10-06T05:07:01.760146+0000","last_unstale":"2023-10-06T05:07:01.765132+0000","last_undegraded":"2023-10-06T05:07:01.765132+0000","last_fullsized":"2023-10-06T05:07:01.765132+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:30:21.024442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3,0],"acting":[5,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"24'2","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.748025+0000","last_change":"2023-10-06T05:06:56.722076+0000","last_active":"2023-10-06T05:07:01.748025+0000","last_peered":"2023-10-06T05:07:01.748025+0000","last_clean":"2023-10-06T05:07:01.748025+0000","last_became_active":"2023-10-06T05:06:56.721786+0000","last_became_peered":"2023-10-06T05:06:56.721786+0000","last_unstale":"2023-10-06T05:07:01.748025+0000","last_undegraded":"2023-10-06T05:07:01.748025+0000","last_fullsized":"2023-10-06T05:07:01.748025+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:15:54.616807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.749669+0000","last_change":"2023-10-06T05:06:57.620208+0000","last_active":"2023-10-06T05:07:01.749669+0000","last_peered":"2023-10-06T05:07:01.749669+0000","last_clean":"2023-10-06T05:07:01.749669+0000","last_became_active":"2023-10-06T05:06:57.619747+0000","last_became_peered":"2023-10-06T05:06:57.619747+0000","last_unstale":"2023-10-06T05:07:01.749669+0000","last_undegraded":"2023-10-06T05:07:01.749669+0000","last_fullsized":"2023-10-06T05:07:01.749669+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:52:14.772801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.764815+0000","last_change":"2023-10-06T05:07:01.762230+0000","last_active":"2023-10-06T05:07:01.764815+0000","last_peered":"2023-10-06T05:07:01.764815+0000","last_clean":"2023-10-06T05:07:01.764815+0000","last_became_active":"2023-10-06T05:07:01.761878+0000","last_became_peered":"2023-10-06T05:07:01.761878+0000","last_unstale":"2023-10-06T05:07:01.764815+0000","last_undegraded":"2023-10-06T05:07:01.764815+0000","last_fullsized":"2023-10-06T05:07:01.764815+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:18:24.586582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2,1],"acting":[3,7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.749791+0000","last_change":"2023-10-06T05:06:56.715016+0000","last_active":"2023-10-06T05:07:01.749791+0000","last_peered":"2023-10-06T05:07:01.749791+0000","last_clean":"2023-10-06T05:07:01.749791+0000","last_became_active":"2023-10-06T05:06:56.714513+0000","last_became_peered":"2023-10-06T05:06:56.714513+0000","last_unstale":"2023-10-06T05:07:01.749791+0000","last_undegraded":"2023-10-06T05:07:01.749791+0000","last_fullsized":"2023-10-06T05:07:01.749791+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T16:11:31.404755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.744725+0000","last_change":"2023-10-06T05:06:57.613062+0000","last_active":"2023-10-06T05:07:01.744725+0000","last_peered":"2023-10-06T05:07:01.744725+0000","last_clean":"2023-10-06T05:07:01.744725+0000","last_became_active":"2023-10-06T05:06:57.611524+0000","last_became_peered":"2023-10-06T05:06:57.611524+0000","last_unstale":"2023-10-06T05:07:01.744725+0000","last_undegraded":"2023-10-06T05:07:01.744725+0000","last_fullsized":"2023-10-06T05:07:01.744725+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:01:11.480811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.770644+0000","last_change":"2023-10-06T05:07:01.764987+0000","last_active":"2023-10-06T05:07:01.770644+0000","last_peered":"2023-10-06T05:07:01.770644+0000","last_clean":"2023-10-06T05:07:01.770644+0000","last_became_active":"2023-10-06T05:07:01.764475+0000","last_became_peered":"2023-10-06T05:07:01.764475+0000","last_unstale":"2023-10-06T05:07:01.770644+0000","last_undegraded":"2023-10-06T05:07:01.770644+0000","last_fullsized":"2023-10-06T05:07:01.770644+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:25:41.281713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,0],"acting":[3,1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.755037+0000","last_change":"2023-10-06T05:06:56.719174+0000","last_active":"2023-10-06T05:07:01.755037+0000","last_peered":"2023-10-06T05:07:01.755037+0000","last_clean":"2023-10-06T05:07:01.755037+0000","last_became_active":"2023-10-06T05:06:56.718930+0000","last_became_peered":"2023-10-06T05:06:56.718930+0000","last_unstale":"2023-10-06T05:07:01.755037+0000","last_undegraded":"2023-10-06T05:07:01.755037+0000","last_fullsized":"2023-10-06T05:07:01.755037+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:51:30.871635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750114+0000","last_change":"2023-10-06T05:06:57.617389+0000","last_active":"2023-10-06T05:07:01.750114+0000","last_peered":"2023-10-06T05:07:01.750114+0000","last_clean":"2023-10-06T05:07:01.750114+0000","last_became_active":"2023-10-06T05:06:57.617099+0000","last_became_peered":"2023-10-06T05:06:57.617099+0000","last_unstale":"2023-10-06T05:07:01.750114+0000","last_undegraded":"2023-10-06T05:07:01.750114+0000","last_fullsized":"2023-10-06T05:07:01.750114+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:51:15.929322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.759618+0000","last_change":"2023-10-06T05:07:01.756721+0000","last_active":"2023-10-06T05:07:01.759618+0000","last_peered":"2023-10-06T05:07:01.759618+0000","last_clean":"2023-10-06T05:07:01.759618+0000","last_became_active":"2023-10-06T05:07:01.756335+0000","last_became_peered":"2023-10-06T05:07:01.756335+0000","last_unstale":"2023-10-06T05:07:01.759618+0000","last_undegraded":"2023-10-06T05:07:01.759618+0000","last_fullsized":"2023-10-06T05:07:01.759618+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:20:29.759167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1,6],"acting":[5,3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"24'8","reported_seq":22,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750615+0000","last_change":"2023-10-06T05:06:56.713625+0000","last_active":"2023-10-06T05:07:01.750615+0000","last_peered":"2023-10-06T05:07:01.750615+0000","last_clean":"2023-10-06T05:07:01.750615+0000","last_became_active":"2023-10-06T05:06:56.713346+0000","last_became_peered":"2023-10-06T05:06:56.713346+0000","last_unstale":"2023-10-06T05:07:01.750615+0000","last_undegraded":"2023-10-06T05:07:01.750615+0000","last_fullsized":"2023-10-06T05:07:01.750615+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:57:59.037083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750303+0000","last_change":"2023-10-06T05:06:57.617670+0000","last_active":"2023-10-06T05:07:01.750303+0000","last_peered":"2023-10-06T05:07:01.750303+0000","last_clean":"2023-10-06T05:07:01.750303+0000","last_became_active":"2023-10-06T05:06:57.617259+0000","last_became_peered":"2023-10-06T05:06:57.617259+0000","last_unstale":"2023-10-06T05:07:01.750303+0000","last_undegraded":"2023-10-06T05:07:01.750303+0000","last_fullsized":"2023-10-06T05:07:01.750303+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:22:11.051575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.775868+0000","last_change":"2023-10-06T05:07:01.771047+0000","last_active":"2023-10-06T05:07:01.775868+0000","last_peered":"2023-10-06T05:07:01.775868+0000","last_clean":"2023-10-06T05:07:01.775868+0000","last_became_active":"2023-10-06T05:07:01.770692+0000","last_became_peered":"2023-10-06T05:07:01.770692+0000","last_unstale":"2023-10-06T05:07:01.775868+0000","last_undegraded":"2023-10-06T05:07:01.775868+0000","last_fullsized":"2023-10-06T05:07:01.775868+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:36:04.280555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3,4],"acting":[7,6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"24'1","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.752603+0000","last_change":"2023-10-06T05:06:56.722082+0000","last_active":"2023-10-06T05:07:01.752603+0000","last_peered":"2023-10-06T05:07:01.752603+0000","last_clean":"2023-10-06T05:07:01.752603+0000","last_became_active":"2023-10-06T05:06:56.721758+0000","last_became_peered":"2023-10-06T05:06:56.721758+0000","last_unstale":"2023-10-06T05:07:01.752603+0000","last_undegraded":"2023-10-06T05:07:01.752603+0000","last_fullsized":"2023-10-06T05:07:01.752603+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:10:26.413910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.745354+0000","last_change":"2023-10-06T05:06:57.611665+0000","last_active":"2023-10-06T05:07:01.745354+0000","last_peered":"2023-10-06T05:07:01.745354+0000","last_clean":"2023-10-06T05:07:01.745354+0000","last_became_active":"2023-10-06T05:06:57.611226+0000","last_became_peered":"2023-10-06T05:06:57.611226+0000","last_unstale":"2023-10-06T05:07:01.745354+0000","last_undegraded":"2023-10-06T05:07:01.745354+0000","last_fullsized":"2023-10-06T05:07:01.745354+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:26:29.348498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.761343+0000","last_change":"2023-10-06T05:07:01.760161+0000","last_active":"2023-10-06T05:07:01.761343+0000","last_peered":"2023-10-06T05:07:01.761343+0000","last_clean":"2023-10-06T05:07:01.761343+0000","last_became_active":"2023-10-06T05:07:01.759742+0000","last_became_peered":"2023-10-06T05:07:01.759742+0000","last_unstale":"2023-10-06T05:07:01.761343+0000","last_undegraded":"2023-10-06T05:07:01.761343+0000","last_fullsized":"2023-10-06T05:07:01.761343+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:10:16.106747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1,5],"acting":[2,0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"24'2","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.751770+0000","last_change":"2023-10-06T05:06:56.722004+0000","last_active":"2023-10-06T05:07:01.751770+0000","last_peered":"2023-10-06T05:07:01.751770+0000","last_clean":"2023-10-06T05:07:01.751770+0000","last_became_active":"2023-10-06T05:06:56.721717+0000","last_became_peered":"2023-10-06T05:06:56.721717+0000","last_unstale":"2023-10-06T05:07:01.751770+0000","last_undegraded":"2023-10-06T05:07:01.751770+0000","last_fullsized":"2023-10-06T05:07:01.751770+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:50:35.786032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.761552+0000","last_change":"2023-10-06T05:06:57.609279+0000","last_active":"2023-10-06T05:07:01.761552+0000","last_peered":"2023-10-06T05:07:01.761552+0000","last_clean":"2023-10-06T05:07:01.761552+0000","last_became_active":"2023-10-06T05:06:57.608982+0000","last_became_peered":"2023-10-06T05:06:57.608982+0000","last_unstale":"2023-10-06T05:07:01.761552+0000","last_undegraded":"2023-10-06T05:07:01.761552+0000","last_fullsized":"2023-10-06T05:07:01.761552+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:38:16.871694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.765310+0000","last_change":"2023-10-06T05:07:01.762960+0000","last_active":"2023-10-06T05:07:01.765310+0000","last_peered":"2023-10-06T05:07:01.765310+0000","last_clean":"2023-10-06T05:07:01.765310+0000","last_became_active":"2023-10-06T05:07:01.762308+0000","last_became_peered":"2023-10-06T05:07:01.762308+0000","last_unstale":"2023-10-06T05:07:01.765310+0000","last_undegraded":"2023-10-06T05:07:01.765310+0000","last_fullsized":"2023-10-06T05:07:01.765310+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:49:52.354792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5,6],"acting":[2,0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"24'1","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.754352+0000","last_change":"2023-10-06T05:06:56.710907+0000","last_active":"2023-10-06T05:07:01.754352+0000","last_peered":"2023-10-06T05:07:01.754352+0000","last_clean":"2023-10-06T05:07:01.754352+0000","last_became_active":"2023-10-06T05:06:56.710595+0000","last_became_peered":"2023-10-06T05:06:56.710595+0000","last_unstale":"2023-10-06T05:07:01.754352+0000","last_undegraded":"2023-10-06T05:07:01.754352+0000","last_fullsized":"2023-10-06T05:07:01.754352+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:17:31.213115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.751968+0000","last_change":"2023-10-06T05:06:57.611226+0000","last_active":"2023-10-06T05:07:01.751968+0000","last_peered":"2023-10-06T05:07:01.751968+0000","last_clean":"2023-10-06T05:07:01.751968+0000","last_became_active":"2023-10-06T05:06:57.610966+0000","last_became_peered":"2023-10-06T05:06:57.610966+0000","last_unstale":"2023-10-06T05:07:01.751968+0000","last_undegraded":"2023-10-06T05:07:01.751968+0000","last_fullsized":"2023-10-06T05:07:01.751968+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:15:48.746279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.768101+0000","last_change":"2023-10-06T05:07:01.764757+0000","last_active":"2023-10-06T05:07:01.768101+0000","last_peered":"2023-10-06T05:07:01.768101+0000","last_clean":"2023-10-06T05:07:01.768101+0000","last_became_active":"2023-10-06T05:07:01.762819+0000","last_became_peered":"2023-10-06T05:07:01.762819+0000","last_unstale":"2023-10-06T05:07:01.768101+0000","last_undegraded":"2023-10-06T05:07:01.768101+0000","last_fullsized":"2023-10-06T05:07:01.768101+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:21:17.237790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3,6],"acting":[2,4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.745471+0000","last_change":"2023-10-06T05:06:56.704818+0000","last_active":"2023-10-06T05:07:01.745471+0000","last_peered":"2023-10-06T05:07:01.745471+0000","last_clean":"2023-10-06T05:07:01.745471+0000","last_became_active":"2023-10-06T05:06:56.704596+0000","last_became_peered":"2023-10-06T05:06:56.704596+0000","last_unstale":"2023-10-06T05:07:01.745471+0000","last_undegraded":"2023-10-06T05:07:01.745471+0000","last_fullsized":"2023-10-06T05:07:01.745471+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:23:02.300780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750614+0000","last_change":"2023-10-06T05:06:57.615733+0000","last_active":"2023-10-06T05:07:01.750614+0000","last_peered":"2023-10-06T05:07:01.750614+0000","last_clean":"2023-10-06T05:07:01.750614+0000","last_became_active":"2023-10-06T05:06:57.615061+0000","last_became_peered":"2023-10-06T05:06:57.615061+0000","last_unstale":"2023-10-06T05:07:01.750614+0000","last_undegraded":"2023-10-06T05:07:01.750614+0000","last_fullsized":"2023-10-06T05:07:01.750614+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:45:23.272110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.768073+0000","last_change":"2023-10-06T05:07:01.763754+0000","last_active":"2023-10-06T05:07:01.768073+0000","last_peered":"2023-10-06T05:07:01.768073+0000","last_clean":"2023-10-06T05:07:01.768073+0000","last_became_active":"2023-10-06T05:07:01.763392+0000","last_became_peered":"2023-10-06T05:07:01.763392+0000","last_unstale":"2023-10-06T05:07:01.768073+0000","last_undegraded":"2023-10-06T05:07:01.768073+0000","last_fullsized":"2023-10-06T05:07:01.768073+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:38:20.917328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5,6],"acting":[2,7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"24'2","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750935+0000","last_change":"2023-10-06T05:06:56.717936+0000","last_active":"2023-10-06T05:07:01.750935+0000","last_peered":"2023-10-06T05:07:01.750935+0000","last_clean":"2023-10-06T05:07:01.750935+0000","last_became_active":"2023-10-06T05:06:56.717683+0000","last_became_peered":"2023-10-06T05:06:56.717683+0000","last_unstale":"2023-10-06T05:07:01.750935+0000","last_undegraded":"2023-10-06T05:07:01.750935+0000","last_fullsized":"2023-10-06T05:07:01.750935+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:23:02.300780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.760808+0000","last_change":"2023-10-06T05:06:57.617318+0000","last_active":"2023-10-06T05:07:01.760808+0000","last_peered":"2023-10-06T05:07:01.760808+0000","last_clean":"2023-10-06T05:07:01.760808+0000","last_became_active":"2023-10-06T05:06:57.617044+0000","last_became_peered":"2023-10-06T05:06:57.617044+0000","last_unstale":"2023-10-06T05:07:01.760808+0000","last_undegraded":"2023-10-06T05:07:01.760808+0000","last_fullsized":"2023-10-06T05:07:01.760808+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:51:15.570020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.772045+0000","last_change":"2023-10-06T05:07:01.762386+0000","last_active":"2023-10-06T05:07:01.772045+0000","last_peered":"2023-10-06T05:07:01.772045+0000","last_clean":"2023-10-06T05:07:01.772045+0000","last_became_active":"2023-10-06T05:07:01.761040+0000","last_became_peered":"2023-10-06T05:07:01.761040+0000","last_unstale":"2023-10-06T05:07:01.772045+0000","last_undegraded":"2023-10-06T05:07:01.772045+0000","last_fullsized":"2023-10-06T05:07:01.772045+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:11:22.468821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0,6],"acting":[1,4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.752055+0000","last_change":"2023-10-06T05:06:56.723332+0000","last_active":"2023-10-06T05:07:01.752055+0000","last_peered":"2023-10-06T05:07:01.752055+0000","last_clean":"2023-10-06T05:07:01.752055+0000","last_became_active":"2023-10-06T05:06:56.723023+0000","last_became_peered":"2023-10-06T05:06:56.723023+0000","last_unstale":"2023-10-06T05:07:01.752055+0000","last_undegraded":"2023-10-06T05:07:01.752055+0000","last_fullsized":"2023-10-06T05:07:01.752055+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:36:39.323759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.761806+0000","last_change":"2023-10-06T05:06:57.609386+0000","last_active":"2023-10-06T05:07:01.761806+0000","last_peered":"2023-10-06T05:07:01.761806+0000","last_clean":"2023-10-06T05:07:01.761806+0000","last_became_active":"2023-10-06T05:06:57.609019+0000","last_became_peered":"2023-10-06T05:06:57.609019+0000","last_unstale":"2023-10-06T05:07:01.761806+0000","last_undegraded":"2023-10-06T05:07:01.761806+0000","last_fullsized":"2023-10-06T05:07:01.761806+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:20:18.722541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.767691+0000","last_change":"2023-10-06T05:07:01.763186+0000","last_active":"2023-10-06T05:07:01.767691+0000","last_peered":"2023-10-06T05:07:01.767691+0000","last_clean":"2023-10-06T05:07:01.767691+0000","last_became_active":"2023-10-06T05:07:01.762259+0000","last_became_peered":"2023-10-06T05:07:01.762259+0000","last_unstale":"2023-10-06T05:07:01.767691+0000","last_undegraded":"2023-10-06T05:07:01.767691+0000","last_fullsized":"2023-10-06T05:07:01.767691+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:46:13.640987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6,2],"acting":[5,4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.761404+0000","last_change":"2023-10-06T05:06:56.722507+0000","last_active":"2023-10-06T05:07:01.761404+0000","last_peered":"2023-10-06T05:07:01.761404+0000","last_clean":"2023-10-06T05:07:01.761404+0000","last_became_active":"2023-10-06T05:06:56.722243+0000","last_became_peered":"2023-10-06T05:06:56.722243+0000","last_unstale":"2023-10-06T05:07:01.761404+0000","last_undegraded":"2023-10-06T05:07:01.761404+0000","last_fullsized":"2023-10-06T05:07:01.761404+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:57:59.037083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.761170+0000","last_change":"2023-10-06T05:06:57.606979+0000","last_active":"2023-10-06T05:07:01.761170+0000","last_peered":"2023-10-06T05:07:01.761170+0000","last_clean":"2023-10-06T05:07:01.761170+0000","last_became_active":"2023-10-06T05:06:57.606633+0000","last_became_peered":"2023-10-06T05:06:57.606633+0000","last_unstale":"2023-10-06T05:07:01.761170+0000","last_undegraded":"2023-10-06T05:07:01.761170+0000","last_fullsized":"2023-10-06T05:07:01.761170+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:21:13.192156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.768126+0000","last_change":"2023-10-06T05:07:01.765104+0000","last_active":"2023-10-06T05:07:01.768126+0000","last_peered":"2023-10-06T05:07:01.768126+0000","last_clean":"2023-10-06T05:07:01.768126+0000","last_became_active":"2023-10-06T05:07:01.764692+0000","last_became_peered":"2023-10-06T05:07:01.764692+0000","last_unstale":"2023-10-06T05:07:01.768126+0000","last_undegraded":"2023-10-06T05:07:01.768126+0000","last_fullsized":"2023-10-06T05:07:01.768126+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:40:47.847563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0,2],"acting":[4,5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.749439+0000","last_change":"2023-10-06T05:06:56.708403+0000","last_active":"2023-10-06T05:07:01.749439+0000","last_peered":"2023-10-06T05:07:01.749439+0000","last_clean":"2023-10-06T05:07:01.749439+0000","last_became_active":"2023-10-06T05:06:56.708144+0000","last_became_peered":"2023-10-06T05:06:56.708144+0000","last_unstale":"2023-10-06T05:07:01.749439+0000","last_undegraded":"2023-10-06T05:07:01.749439+0000","last_fullsized":"2023-10-06T05:07:01.749439+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:17:02.349663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750522+0000","last_change":"2023-10-06T05:06:57.620557+0000","last_active":"2023-10-06T05:07:01.750522+0000","last_peered":"2023-10-06T05:07:01.750522+0000","last_clean":"2023-10-06T05:07:01.750522+0000","last_became_active":"2023-10-06T05:06:57.620286+0000","last_became_peered":"2023-10-06T05:06:57.620286+0000","last_unstale":"2023-10-06T05:07:01.750522+0000","last_undegraded":"2023-10-06T05:07:01.750522+0000","last_fullsized":"2023-10-06T05:07:01.750522+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T16:14:57.399087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.765266+0000","last_change":"2023-10-06T05:07:01.762346+0000","last_active":"2023-10-06T05:07:01.765266+0000","last_peered":"2023-10-06T05:07:01.765266+0000","last_clean":"2023-10-06T05:07:01.765266+0000","last_became_active":"2023-10-06T05:07:01.761861+0000","last_became_peered":"2023-10-06T05:07:01.761861+0000","last_unstale":"2023-10-06T05:07:01.765266+0000","last_undegraded":"2023-10-06T05:07:01.765266+0000","last_fullsized":"2023-10-06T05:07:01.765266+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:41:09.047631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6,5],"acting":[0,1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.748453+0000","last_change":"2023-10-06T05:06:52.627232+0000","last_active":"2023-10-06T05:07:01.748453+0000","last_peered":"2023-10-06T05:07:01.748453+0000","last_clean":"2023-10-06T05:07:01.748453+0000","last_became_active":"2023-10-06T05:06:52.626936+0000","last_became_peered":"2023-10-06T05:06:52.626936+0000","last_unstale":"2023-10-06T05:07:01.748453+0000","last_undegraded":"2023-10-06T05:07:01.748453+0000","last_fullsized":"2023-10-06T05:07:01.748453+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:20:13.925170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.744621+0000","last_change":"2023-10-06T05:06:56.712027+0000","last_active":"2023-10-06T05:07:01.744621+0000","last_peered":"2023-10-06T05:07:01.744621+0000","last_clean":"2023-10-06T05:07:01.744621+0000","last_became_active":"2023-10-06T05:06:56.711779+0000","last_became_peered":"2023-10-06T05:06:56.711779+0000","last_unstale":"2023-10-06T05:07:01.744621+0000","last_undegraded":"2023-10-06T05:07:01.744621+0000","last_fullsized":"2023-10-06T05:07:01.744621+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:49:27.022333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.747306+0000","last_change":"2023-10-06T05:06:57.613847+0000","last_active":"2023-10-06T05:07:01.747306+0000","last_peered":"2023-10-06T05:07:01.747306+0000","last_clean":"2023-10-06T05:07:01.747306+0000","last_became_active":"2023-10-06T05:06:57.613560+0000","last_became_peered":"2023-10-06T05:06:57.613560+0000","last_unstale":"2023-10-06T05:07:01.747306+0000","last_undegraded":"2023-10-06T05:07:01.747306+0000","last_fullsized":"2023-10-06T05:07:01.747306+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:50:36.787873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.766219+0000","last_change":"2023-10-06T05:07:01.764522+0000","last_active":"2023-10-06T05:07:01.766219+0000","last_peered":"2023-10-06T05:07:01.766219+0000","last_clean":"2023-10-06T05:07:01.766219+0000","last_became_active":"2023-10-06T05:07:01.764199+0000","last_became_peered":"2023-10-06T05:07:01.764199+0000","last_unstale":"2023-10-06T05:07:01.766219+0000","last_undegraded":"2023-10-06T05:07:01.766219+0000","last_fullsized":"2023-10-06T05:07:01.766219+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:13:19.947679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,6],"acting":[0,1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.744573+0000","last_change":"2023-10-06T05:06:52.587795+0000","last_active":"2023-10-06T05:07:01.744573+0000","last_peered":"2023-10-06T05:07:01.744573+0000","last_clean":"2023-10-06T05:07:01.744573+0000","last_became_active":"2023-10-06T05:06:52.587407+0000","last_became_peered":"2023-10-06T05:06:52.587407+0000","last_unstale":"2023-10-06T05:07:01.744573+0000","last_undegraded":"2023-10-06T05:07:01.744573+0000","last_fullsized":"2023-10-06T05:07:01.744573+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:36:31.526532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.761857+0000","last_change":"2023-10-06T05:06:56.721976+0000","last_active":"2023-10-06T05:07:01.761857+0000","last_peered":"2023-10-06T05:07:01.761857+0000","last_clean":"2023-10-06T05:07:01.761857+0000","last_became_active":"2023-10-06T05:06:56.721679+0000","last_became_peered":"2023-10-06T05:06:56.721679+0000","last_unstale":"2023-10-06T05:07:01.761857+0000","last_undegraded":"2023-10-06T05:07:01.761857+0000","last_fullsized":"2023-10-06T05:07:01.761857+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:38:50.877673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.748383+0000","last_change":"2023-10-06T05:06:57.612621+0000","last_active":"2023-10-06T05:07:01.748383+0000","last_peered":"2023-10-06T05:07:01.748383+0000","last_clean":"2023-10-06T05:07:01.748383+0000","last_became_active":"2023-10-06T05:06:57.612305+0000","last_became_peered":"2023-10-06T05:06:57.612305+0000","last_unstale":"2023-10-06T05:07:01.748383+0000","last_undegraded":"2023-10-06T05:07:01.748383+0000","last_fullsized":"2023-10-06T05:07:01.748383+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:35:18.828024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.768737+0000","last_change":"2023-10-06T05:07:01.764514+0000","last_active":"2023-10-06T05:07:01.768737+0000","last_peered":"2023-10-06T05:07:01.768737+0000","last_clean":"2023-10-06T05:07:01.768737+0000","last_became_active":"2023-10-06T05:07:01.764176+0000","last_became_peered":"2023-10-06T05:07:01.764176+0000","last_unstale":"2023-10-06T05:07:01.768737+0000","last_undegraded":"2023-10-06T05:07:01.768737+0000","last_fullsized":"2023-10-06T05:07:01.768737+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:20:08.719247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1,7],"acting":[2,5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.746279+0000","last_change":"2023-10-06T05:06:52.675238+0000","last_active":"2023-10-06T05:07:01.746279+0000","last_peered":"2023-10-06T05:07:01.746279+0000","last_clean":"2023-10-06T05:07:01.746279+0000","last_became_active":"2023-10-06T05:06:52.674952+0000","last_became_peered":"2023-10-06T05:06:52.674952+0000","last_unstale":"2023-10-06T05:07:01.746279+0000","last_undegraded":"2023-10-06T05:07:01.746279+0000","last_fullsized":"2023-10-06T05:07:01.746279+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:27:51.141184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.746657+0000","last_change":"2023-10-06T05:06:56.711280+0000","last_active":"2023-10-06T05:07:01.746657+0000","last_peered":"2023-10-06T05:07:01.746657+0000","last_clean":"2023-10-06T05:07:01.746657+0000","last_became_active":"2023-10-06T05:06:56.711033+0000","last_became_peered":"2023-10-06T05:06:56.711033+0000","last_unstale":"2023-10-06T05:07:01.746657+0000","last_undegraded":"2023-10-06T05:07:01.746657+0000","last_fullsized":"2023-10-06T05:07:01.746657+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:38:12.815936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750448+0000","last_change":"2023-10-06T05:06:57.614460+0000","last_active":"2023-10-06T05:07:01.750448+0000","last_peered":"2023-10-06T05:07:01.750448+0000","last_clean":"2023-10-06T05:07:01.750448+0000","last_became_active":"2023-10-06T05:06:57.614245+0000","last_became_peered":"2023-10-06T05:06:57.614245+0000","last_unstale":"2023-10-06T05:07:01.750448+0000","last_undegraded":"2023-10-06T05:07:01.750448+0000","last_fullsized":"2023-10-06T05:07:01.750448+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:51:31.873476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.775792+0000","last_change":"2023-10-06T05:07:01.766672+0000","last_active":"2023-10-06T05:07:01.775792+0000","last_peered":"2023-10-06T05:07:01.775792+0000","last_clean":"2023-10-06T05:07:01.775792+0000","last_became_active":"2023-10-06T05:07:01.766118+0000","last_became_peered":"2023-10-06T05:07:01.766118+0000","last_unstale":"2023-10-06T05:07:01.775792+0000","last_undegraded":"2023-10-06T05:07:01.775792+0000","last_fullsized":"2023-10-06T05:07:01.775792+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:55:02.566598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5,3],"acting":[7,2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.745050+0000","last_change":"2023-10-06T05:06:52.587778+0000","last_active":"2023-10-06T05:07:01.745050+0000","last_peered":"2023-10-06T05:07:01.745050+0000","last_clean":"2023-10-06T05:07:01.745050+0000","last_became_active":"2023-10-06T05:06:52.587388+0000","last_became_peered":"2023-10-06T05:06:52.587388+0000","last_unstale":"2023-10-06T05:07:01.745050+0000","last_undegraded":"2023-10-06T05:07:01.745050+0000","last_fullsized":"2023-10-06T05:07:01.745050+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:45:52.502298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":169,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.748339+0000","last_change":"2023-10-06T05:06:46.307169+0000","last_active":"2023-10-06T05:07:01.748339+0000","last_peered":"2023-10-06T05:07:01.748339+0000","last_clean":"2023-10-06T05:07:01.748339+0000","last_became_active":"2023-10-06T05:06:46.306107+0000","last_became_peered":"2023-10-06T05:06:46.306107+0000","last_unstale":"2023-10-06T05:07:01.748339+0000","last_undegraded":"2023-10-06T05:07:01.748339+0000","last_fullsized":"2023-10-06T05:07:01.748339+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:45.018006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:45.018006+0000","last_clean_scrub_stamp":"2023-10-06T05:06:45.018006+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:38:17.064131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750303+0000","last_change":"2023-10-06T05:06:56.706870+0000","last_active":"2023-10-06T05:07:01.750303+0000","last_peered":"2023-10-06T05:07:01.750303+0000","last_clean":"2023-10-06T05:07:01.750303+0000","last_became_active":"2023-10-06T05:06:56.706636+0000","last_became_peered":"2023-10-06T05:06:56.706636+0000","last_unstale":"2023-10-06T05:07:01.750303+0000","last_undegraded":"2023-10-06T05:07:01.750303+0000","last_fullsized":"2023-10-06T05:07:01.750303+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:00:29.033929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.753036+0000","last_change":"2023-10-06T05:06:57.604630+0000","last_active":"2023-10-06T05:07:01.753036+0000","last_peered":"2023-10-06T05:07:01.753036+0000","last_clean":"2023-10-06T05:07:01.753036+0000","last_became_active":"2023-10-06T05:06:57.604341+0000","last_became_peered":"2023-10-06T05:06:57.604341+0000","last_unstale":"2023-10-06T05:07:01.753036+0000","last_undegraded":"2023-10-06T05:07:01.753036+0000","last_fullsized":"2023-10-06T05:07:01.753036+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:57:12.615864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.767715+0000","last_change":"2023-10-06T05:07:01.763448+0000","last_active":"2023-10-06T05:07:01.767715+0000","last_peered":"2023-10-06T05:07:01.767715+0000","last_clean":"2023-10-06T05:07:01.767715+0000","last_became_active":"2023-10-06T05:07:01.762832+0000","last_became_peered":"2023-10-06T05:07:01.762832+0000","last_unstale":"2023-10-06T05:07:01.767715+0000","last_undegraded":"2023-10-06T05:07:01.767715+0000","last_fullsized":"2023-10-06T05:07:01.767715+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:49:53.682865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0,4],"acting":[5,1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.748590+0000","last_change":"2023-10-06T05:06:52.588475+0000","last_active":"2023-10-06T05:07:01.748590+0000","last_peered":"2023-10-06T05:07:01.748590+0000","last_clean":"2023-10-06T05:07:01.748590+0000","last_became_active":"2023-10-06T05:06:52.588134+0000","last_became_peered":"2023-10-06T05:06:52.588134+0000","last_unstale":"2023-10-06T05:07:01.748590+0000","last_undegraded":"2023-10-06T05:07:01.748590+0000","last_fullsized":"2023-10-06T05:07:01.748590+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:35:05.269778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.749699+0000","last_change":"2023-10-06T05:06:56.723240+0000","last_active":"2023-10-06T05:07:01.749699+0000","last_peered":"2023-10-06T05:07:01.749699+0000","last_clean":"2023-10-06T05:07:01.749699+0000","last_became_active":"2023-10-06T05:06:56.722965+0000","last_became_peered":"2023-10-06T05:06:56.722965+0000","last_unstale":"2023-10-06T05:07:01.749699+0000","last_undegraded":"2023-10-06T05:07:01.749699+0000","last_fullsized":"2023-10-06T05:07:01.749699+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:33:23.298771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.749910+0000","last_change":"2023-10-06T05:06:57.616494+0000","last_active":"2023-10-06T05:07:01.749910+0000","last_peered":"2023-10-06T05:07:01.749910+0000","last_clean":"2023-10-06T05:07:01.749910+0000","last_became_active":"2023-10-06T05:06:57.616234+0000","last_became_peered":"2023-10-06T05:06:57.616234+0000","last_unstale":"2023-10-06T05:07:01.749910+0000","last_undegraded":"2023-10-06T05:07:01.749910+0000","last_fullsized":"2023-10-06T05:07:01.749910+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:34:55.987076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.773039+0000","last_change":"2023-10-06T05:07:01.768822+0000","last_active":"2023-10-06T05:07:01.773039+0000","last_peered":"2023-10-06T05:07:01.773039+0000","last_clean":"2023-10-06T05:07:01.773039+0000","last_became_active":"2023-10-06T05:07:01.768425+0000","last_became_peered":"2023-10-06T05:07:01.768425+0000","last_unstale":"2023-10-06T05:07:01.773039+0000","last_undegraded":"2023-10-06T05:07:01.773039+0000","last_fullsized":"2023-10-06T05:07:01.773039+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:52:18.818435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5,2],"acting":[6,0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.754250+0000","last_change":"2023-10-06T05:06:52.627289+0000","last_active":"2023-10-06T05:07:01.754250+0000","last_peered":"2023-10-06T05:07:01.754250+0000","last_clean":"2023-10-06T05:07:01.754250+0000","last_became_active":"2023-10-06T05:06:52.626937+0000","last_became_peered":"2023-10-06T05:06:52.626937+0000","last_unstale":"2023-10-06T05:07:01.754250+0000","last_undegraded":"2023-10-06T05:07:01.754250+0000","last_fullsized":"2023-10-06T05:07:01.754250+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:46:59.070674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.748611+0000","last_change":"2023-10-06T05:06:56.720424+0000","last_active":"2023-10-06T05:07:01.748611+0000","last_peered":"2023-10-06T05:07:01.748611+0000","last_clean":"2023-10-06T05:07:01.748611+0000","last_became_active":"2023-10-06T05:06:56.720133+0000","last_became_peered":"2023-10-06T05:06:56.720133+0000","last_unstale":"2023-10-06T05:07:01.748611+0000","last_undegraded":"2023-10-06T05:07:01.748611+0000","last_fullsized":"2023-10-06T05:07:01.748611+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T16:11:31.404755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750404+0000","last_change":"2023-10-06T05:06:57.619048+0000","last_active":"2023-10-06T05:07:01.750404+0000","last_peered":"2023-10-06T05:07:01.750404+0000","last_clean":"2023-10-06T05:07:01.750404+0000","last_became_active":"2023-10-06T05:06:57.618823+0000","last_became_peered":"2023-10-06T05:06:57.618823+0000","last_unstale":"2023-10-06T05:07:01.750404+0000","last_undegraded":"2023-10-06T05:07:01.750404+0000","last_fullsized":"2023-10-06T05:07:01.750404+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:25:56.905027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.768167+0000","last_change":"2023-10-06T05:07:01.765125+0000","last_active":"2023-10-06T05:07:01.768167+0000","last_peered":"2023-10-06T05:07:01.768167+0000","last_clean":"2023-10-06T05:07:01.768167+0000","last_became_active":"2023-10-06T05:07:01.764733+0000","last_became_peered":"2023-10-06T05:07:01.764733+0000","last_unstale":"2023-10-06T05:07:01.768167+0000","last_undegraded":"2023-10-06T05:07:01.768167+0000","last_fullsized":"2023-10-06T05:07:01.768167+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T16:29:47.226299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7,0],"acting":[4,3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"24'2","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750393+0000","last_change":"2023-10-06T05:06:56.710924+0000","last_active":"2023-10-06T05:07:01.750393+0000","last_peered":"2023-10-06T05:07:01.750393+0000","last_clean":"2023-10-06T05:07:01.750393+0000","last_became_active":"2023-10-06T05:06:56.710518+0000","last_became_peered":"2023-10-06T05:06:56.710518+0000","last_unstale":"2023-10-06T05:07:01.750393+0000","last_undegraded":"2023-10-06T05:07:01.750393+0000","last_fullsized":"2023-10-06T05:07:01.750393+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:49:27.022333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.748508+0000","last_change":"2023-10-06T05:06:52.588853+0000","last_active":"2023-10-06T05:07:01.748508+0000","last_peered":"2023-10-06T05:07:01.748508+0000","last_clean":"2023-10-06T05:07:01.748508+0000","last_became_active":"2023-10-06T05:06:52.588561+0000","last_became_peered":"2023-10-06T05:06:52.588561+0000","last_unstale":"2023-10-06T05:07:01.748508+0000","last_undegraded":"2023-10-06T05:07:01.748508+0000","last_fullsized":"2023-10-06T05:07:01.748508+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:06:28.729493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.749357+0000","last_change":"2023-10-06T05:06:57.619145+0000","last_active":"2023-10-06T05:07:01.749357+0000","last_peered":"2023-10-06T05:07:01.749357+0000","last_clean":"2023-10-06T05:07:01.749357+0000","last_became_active":"2023-10-06T05:06:57.618666+0000","last_became_peered":"2023-10-06T05:06:57.618666+0000","last_unstale":"2023-10-06T05:07:01.749357+0000","last_undegraded":"2023-10-06T05:07:01.749357+0000","last_fullsized":"2023-10-06T05:07:01.749357+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:29:19.951056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.765168+0000","last_change":"2023-10-06T05:07:01.762410+0000","last_active":"2023-10-06T05:07:01.765168+0000","last_peered":"2023-10-06T05:07:01.765168+0000","last_clean":"2023-10-06T05:07:01.765168+0000","last_became_active":"2023-10-06T05:07:01.762060+0000","last_became_peered":"2023-10-06T05:07:01.762060+0000","last_unstale":"2023-10-06T05:07:01.765168+0000","last_undegraded":"2023-10-06T05:07:01.765168+0000","last_fullsized":"2023-10-06T05:07:01.765168+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:49:53.682865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,0],"acting":[3,1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.751181+0000","last_change":"2023-10-06T05:06:52.589519+0000","last_active":"2023-10-06T05:07:01.751181+0000","last_peered":"2023-10-06T05:07:01.751181+0000","last_clean":"2023-10-06T05:07:01.751181+0000","last_became_active":"2023-10-06T05:06:52.589246+0000","last_became_peered":"2023-10-06T05:06:52.589246+0000","last_unstale":"2023-10-06T05:07:01.751181+0000","last_undegraded":"2023-10-06T05:07:01.751181+0000","last_fullsized":"2023-10-06T05:07:01.751181+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:54:23.352764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"28'1","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:02.685849+0000","last_change":"2023-10-06T05:06:56.714058+0000","last_active":"2023-10-06T05:07:02.685849+0000","last_peered":"2023-10-06T05:07:02.685849+0000","last_clean":"2023-10-06T05:07:02.685849+0000","last_became_active":"2023-10-06T05:06:56.713772+0000","last_became_peered":"2023-10-06T05:06:56.713772+0000","last_unstale":"2023-10-06T05:07:02.685849+0000","last_undegraded":"2023-10-06T05:07:02.685849+0000","last_fullsized":"2023-10-06T05:07:02.685849+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:53:47.748350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.761483+0000","last_change":"2023-10-06T05:06:57.608832+0000","last_active":"2023-10-06T05:07:01.761483+0000","last_peered":"2023-10-06T05:07:01.761483+0000","last_clean":"2023-10-06T05:07:01.761483+0000","last_became_active":"2023-10-06T05:06:57.608499+0000","last_became_peered":"2023-10-06T05:06:57.608499+0000","last_unstale":"2023-10-06T05:07:01.761483+0000","last_undegraded":"2023-10-06T05:07:01.761483+0000","last_fullsized":"2023-10-06T05:07:01.761483+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:29:19.951056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.772123+0000","last_change":"2023-10-06T05:07:01.765615+0000","last_active":"2023-10-06T05:07:01.772123+0000","last_peered":"2023-10-06T05:07:01.772123+0000","last_clean":"2023-10-06T05:07:01.772123+0000","last_became_active":"2023-10-06T05:07:01.762552+0000","last_became_peered":"2023-10-06T05:07:01.762552+0000","last_unstale":"2023-10-06T05:07:01.772123+0000","last_undegraded":"2023-10-06T05:07:01.772123+0000","last_fullsized":"2023-10-06T05:07:01.772123+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:30:14.995001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,0],"acting":[1,3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"24'1","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750483+0000","last_change":"2023-10-06T05:06:56.723224+0000","last_active":"2023-10-06T05:07:01.750483+0000","last_peered":"2023-10-06T05:07:01.750483+0000","last_clean":"2023-10-06T05:07:01.750483+0000","last_became_active":"2023-10-06T05:06:56.722971+0000","last_became_peered":"2023-10-06T05:06:56.722971+0000","last_unstale":"2023-10-06T05:07:01.750483+0000","last_undegraded":"2023-10-06T05:07:01.750483+0000","last_fullsized":"2023-10-06T05:07:01.750483+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:10:32.670079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.746067+0000","last_change":"2023-10-06T05:06:57.610613+0000","last_active":"2023-10-06T05:07:01.746067+0000","last_peered":"2023-10-06T05:07:01.746067+0000","last_clean":"2023-10-06T05:07:01.746067+0000","last_became_active":"2023-10-06T05:06:57.610363+0000","last_became_peered":"2023-10-06T05:06:57.610363+0000","last_unstale":"2023-10-06T05:07:01.746067+0000","last_undegraded":"2023-10-06T05:07:01.746067+0000","last_fullsized":"2023-10-06T05:07:01.746067+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:55:09.125970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.775891+0000","last_change":"2023-10-06T05:07:01.771319+0000","last_active":"2023-10-06T05:07:01.775891+0000","last_peered":"2023-10-06T05:07:01.775891+0000","last_clean":"2023-10-06T05:07:01.775891+0000","last_became_active":"2023-10-06T05:07:01.770953+0000","last_became_peered":"2023-10-06T05:07:01.770953+0000","last_unstale":"2023-10-06T05:07:01.775891+0000","last_undegraded":"2023-10-06T05:07:01.775891+0000","last_fullsized":"2023-10-06T05:07:01.775891+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:13:19.947679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1,6],"acting":[7,4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"24'2","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.745566+0000","last_change":"2023-10-06T05:06:56.723033+0000","last_active":"2023-10-06T05:07:01.745566+0000","last_peered":"2023-10-06T05:07:01.745566+0000","last_clean":"2023-10-06T05:07:01.745566+0000","last_became_active":"2023-10-06T05:06:56.722747+0000","last_became_peered":"2023-10-06T05:06:56.722747+0000","last_unstale":"2023-10-06T05:07:01.745566+0000","last_undegraded":"2023-10-06T05:07:01.745566+0000","last_fullsized":"2023-10-06T05:07:01.745566+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:17:02.349663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.761541+0000","last_change":"2023-10-06T05:06:57.621769+0000","last_active":"2023-10-06T05:07:01.761541+0000","last_peered":"2023-10-06T05:07:01.761541+0000","last_clean":"2023-10-06T05:07:01.761541+0000","last_became_active":"2023-10-06T05:06:57.621489+0000","last_became_peered":"2023-10-06T05:06:57.621489+0000","last_unstale":"2023-10-06T05:07:01.761541+0000","last_undegraded":"2023-10-06T05:07:01.761541+0000","last_fullsized":"2023-10-06T05:07:01.761541+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:59:39.994965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.773062+0000","last_change":"2023-10-06T05:07:01.769584+0000","last_active":"2023-10-06T05:07:01.773062+0000","last_peered":"2023-10-06T05:07:01.773062+0000","last_clean":"2023-10-06T05:07:01.773062+0000","last_became_active":"2023-10-06T05:07:01.769168+0000","last_became_peered":"2023-10-06T05:07:01.769168+0000","last_unstale":"2023-10-06T05:07:01.773062+0000","last_undegraded":"2023-10-06T05:07:01.773062+0000","last_fullsized":"2023-10-06T05:07:01.773062+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:25:41.281713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7,4],"acting":[5,0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"24'2","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.744323+0000","last_change":"2023-10-06T05:06:56.721940+0000","last_active":"2023-10-06T05:07:01.744323+0000","last_peered":"2023-10-06T05:07:01.744323+0000","last_clean":"2023-10-06T05:07:01.744323+0000","last_became_active":"2023-10-06T05:06:56.721668+0000","last_became_peered":"2023-10-06T05:06:56.721668+0000","last_unstale":"2023-10-06T05:07:01.744323+0000","last_undegraded":"2023-10-06T05:07:01.744323+0000","last_fullsized":"2023-10-06T05:07:01.744323+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:55:08.124129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.760815+0000","last_change":"2023-10-06T05:06:57.614557+0000","last_active":"2023-10-06T05:07:01.760815+0000","last_peered":"2023-10-06T05:07:01.760815+0000","last_clean":"2023-10-06T05:07:01.760815+0000","last_became_active":"2023-10-06T05:06:57.614113+0000","last_became_peered":"2023-10-06T05:06:57.614113+0000","last_unstale":"2023-10-06T05:07:01.760815+0000","last_undegraded":"2023-10-06T05:07:01.760815+0000","last_fullsized":"2023-10-06T05:07:01.760815+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:57:12.615864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.772225+0000","last_change":"2023-10-06T05:07:01.766934+0000","last_active":"2023-10-06T05:07:01.772225+0000","last_peered":"2023-10-06T05:07:01.772225+0000","last_clean":"2023-10-06T05:07:01.772225+0000","last_became_active":"2023-10-06T05:07:01.766522+0000","last_became_peered":"2023-10-06T05:07:01.766522+0000","last_unstale":"2023-10-06T05:07:01.772225+0000","last_undegraded":"2023-10-06T05:07:01.772225+0000","last_fullsized":"2023-10-06T05:07:01.772225+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:15:30.724218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,2],"acting":[4,6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"24'1","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.753608+0000","last_change":"2023-10-06T05:06:56.717832+0000","last_active":"2023-10-06T05:07:01.753608+0000","last_peered":"2023-10-06T05:07:01.753608+0000","last_clean":"2023-10-06T05:07:01.753608+0000","last_became_active":"2023-10-06T05:06:56.717568+0000","last_became_peered":"2023-10-06T05:06:56.717568+0000","last_unstale":"2023-10-06T05:07:01.753608+0000","last_undegraded":"2023-10-06T05:07:01.753608+0000","last_fullsized":"2023-10-06T05:07:01.753608+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:30:18.507322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750496+0000","last_change":"2023-10-06T05:06:57.620391+0000","last_active":"2023-10-06T05:07:01.750496+0000","last_peered":"2023-10-06T05:07:01.750496+0000","last_clean":"2023-10-06T05:07:01.750496+0000","last_became_active":"2023-10-06T05:06:57.619822+0000","last_became_peered":"2023-10-06T05:06:57.619822+0000","last_unstale":"2023-10-06T05:07:01.750496+0000","last_undegraded":"2023-10-06T05:07:01.750496+0000","last_fullsized":"2023-10-06T05:07:01.750496+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:59:39.994965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.772089+0000","last_change":"2023-10-06T05:07:01.762139+0000","last_active":"2023-10-06T05:07:01.772089+0000","last_peered":"2023-10-06T05:07:01.772089+0000","last_clean":"2023-10-06T05:07:01.772089+0000","last_became_active":"2023-10-06T05:07:01.761672+0000","last_became_peered":"2023-10-06T05:07:01.761672+0000","last_unstale":"2023-10-06T05:07:01.772089+0000","last_undegraded":"2023-10-06T05:07:01.772089+0000","last_fullsized":"2023-10-06T05:07:01.772089+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:07:51.743165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7,2],"acting":[1,5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"24'4","reported_seq":18,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.751360+0000","last_change":"2023-10-06T05:06:56.714871+0000","last_active":"2023-10-06T05:07:01.751360+0000","last_peered":"2023-10-06T05:07:01.751360+0000","last_clean":"2023-10-06T05:07:01.751360+0000","last_became_active":"2023-10-06T05:06:56.714616+0000","last_became_peered":"2023-10-06T05:06:56.714616+0000","last_unstale":"2023-10-06T05:07:01.751360+0000","last_undegraded":"2023-10-06T05:07:01.751360+0000","last_fullsized":"2023-10-06T05:07:01.751360+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:35:42.402195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.758731+0000","last_change":"2023-10-06T05:06:57.605198+0000","last_active":"2023-10-06T05:07:01.758731+0000","last_peered":"2023-10-06T05:07:01.758731+0000","last_clean":"2023-10-06T05:07:01.758731+0000","last_became_active":"2023-10-06T05:06:57.604924+0000","last_became_peered":"2023-10-06T05:06:57.604924+0000","last_unstale":"2023-10-06T05:07:01.758731+0000","last_undegraded":"2023-10-06T05:07:01.758731+0000","last_fullsized":"2023-10-06T05:07:01.758731+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:01:11.983432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750721+0000","last_change":"2023-10-06T05:06:57.608286+0000","last_active":"2023-10-06T05:07:01.750721+0000","last_peered":"2023-10-06T05:07:01.750721+0000","last_clean":"2023-10-06T05:07:01.750721+0000","last_became_active":"2023-10-06T05:06:57.607986+0000","last_became_peered":"2023-10-06T05:06:57.607986+0000","last_unstale":"2023-10-06T05:07:01.750721+0000","last_undegraded":"2023-10-06T05:07:01.750721+0000","last_fullsized":"2023-10-06T05:07:01.750721+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:51:31.370159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750626+0000","last_change":"2023-10-06T05:06:57.623937+0000","last_active":"2023-10-06T05:07:01.750626+0000","last_peered":"2023-10-06T05:07:01.750626+0000","last_clean":"2023-10-06T05:07:01.750626+0000","last_became_active":"2023-10-06T05:06:57.623705+0000","last_became_peered":"2023-10-06T05:06:57.623705+0000","last_unstale":"2023-10-06T05:07:01.750626+0000","last_undegraded":"2023-10-06T05:07:01.750626+0000","last_fullsized":"2023-10-06T05:07:01.750626+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:20:04.673613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.761577+0000","last_change":"2023-10-06T05:06:57.611066+0000","last_active":"2023-10-06T05:07:01.761577+0000","last_peered":"2023-10-06T05:07:01.761577+0000","last_clean":"2023-10-06T05:07:01.761577+0000","last_became_active":"2023-10-06T05:06:57.610827+0000","last_became_peered":"2023-10-06T05:06:57.610827+0000","last_unstale":"2023-10-06T05:07:01.761577+0000","last_undegraded":"2023-10-06T05:07:01.761577+0000","last_fullsized":"2023-10-06T05:07:01.761577+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:43:18.121629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750840+0000","last_change":"2023-10-06T05:06:57.615191+0000","last_active":"2023-10-06T05:07:01.750840+0000","last_peered":"2023-10-06T05:07:01.750840+0000","last_clean":"2023-10-06T05:07:01.750840+0000","last_became_active":"2023-10-06T05:06:57.614632+0000","last_became_peered":"2023-10-06T05:06:57.614632+0000","last_unstale":"2023-10-06T05:07:01.750840+0000","last_undegraded":"2023-10-06T05:07:01.750840+0000","last_fullsized":"2023-10-06T05:07:01.750840+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:32:33.731449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.748531+0000","last_change":"2023-10-06T05:06:57.616743+0000","last_active":"2023-10-06T05:07:01.748531+0000","last_peered":"2023-10-06T05:07:01.748531+0000","last_clean":"2023-10-06T05:07:01.748531+0000","last_became_active":"2023-10-06T05:06:57.616487+0000","last_became_peered":"2023-10-06T05:06:57.616487+0000","last_unstale":"2023-10-06T05:07:01.748531+0000","last_undegraded":"2023-10-06T05:07:01.748531+0000","last_fullsized":"2023-10-06T05:07:01.748531+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:15:55.618648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":4484,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":8968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509449,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297588,"kb_used_data":300,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074252,"statfs":{"total":96636764160,"available":96332034048,"internally_reserved":0,"allocated":307200,"data_stored":126099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66600000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65600000000000003}]}]},{"osd":5,"up_from":15,"seq":64424509449,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297596,"kb_used_data":308,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074244,"statfs":{"total":96636764160,"available":96332025856,"internally_reserved":0,"allocated":315392,"data_stored":126723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42099999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.372}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39400000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48699999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51400000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61699999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]}]},{"osd":0,"up_from":14,"seq":60129542154,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298172,"kb_used_data":884,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073668,"statfs":{"total":96636764160,"available":96331436032,"internally_reserved":0,"allocated":905216,"data_stored":717021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2809999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.1529999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.327}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0329999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0549999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95899999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.079}]}]},{"osd":7,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298196,"kb_used_data":908,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073644,"statfs":{"total":96636764160,"available":96331411456,"internally_reserved":0,"allocated":929792,"data_stored":720397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33500000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51200000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39900000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41299999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.436}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45000000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42399999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542153,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297588,"kb_used_data":300,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074252,"statfs":{"total":96636764160,"available":96332034048,"internally_reserved":0,"allocated":307200,"data_stored":126099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32700000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36499999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40200000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64900000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60299999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66200000000000003}]}]},{"osd":2,"up_from":15,"seq":64424509448,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297588,"kb_used_data":300,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074252,"statfs":{"total":96636764160,"available":96332034048,"internally_reserved":0,"allocated":307200,"data_stored":126099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26400000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30599999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28000000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39400000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42299999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59799999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.504}]}]},{"osd":3,"up_from":15,"seq":64424509448,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297592,"kb_used_data":304,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074248,"statfs":{"total":96636764160,"available":96332029952,"internally_reserved":0,"allocated":311296,"data_stored":126189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30199999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46400000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]}]},{"osd":4,"up_from":15,"seq":64424509449,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297608,"kb_used_data":320,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074232,"statfs":{"total":96636764160,"available":96332013568,"internally_reserved":0,"allocated":327680,"data_stored":129869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40100000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.379}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":3770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":3930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-06T05:07:16.408 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-06T05:07:16.553 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.551+0000 7fb5b5391700 1 Processor -- start 2023-10-06T05:07:16.554 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.551+0000 7fb5b5391700 1 -- start start 2023-10-06T05:07:16.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.552+0000 7fb5b5391700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb5b0159560 0x7fb5b00aa530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:16.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.552+0000 7fb5b5391700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fb5b00b0760 0x7fb5b00b1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:16.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.552+0000 7fb5b5391700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5b00b0c60 con 0x7fb5b00b49a0 2023-10-06T05:07:16.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.552+0000 7fb5b5391700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5b00b0f40 con 0x7fb5b0159560 2023-10-06T05:07:16.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.552+0000 7fb5b5391700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5b00b1220 con 0x7fb5b00b0760 2023-10-06T05:07:16.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.552+0000 7fb5aeffd700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb5b0159560 0x7fb5b00aa530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:16.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.552+0000 7fb5aeffd700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb5b0159560 0x7fb5b00aa530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46024/0 (socket says 172.21.15.98:46024) 2023-10-06T05:07:16.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.552+0000 7fb5aeffd700 1 -- 172.21.15.98:0/193181324 learned_addr learned my addr 172.21.15.98:0/193181324 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:16.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.552+0000 7fb5af7fe700 1 --2- 172.21.15.98:0/193181324 >> v2:172.21.15.174:3300/0 conn(0x7fb5b00b0760 0x7fb5b00b1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:16.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.552+0000 7fb5aeffd700 1 -- 172.21.15.98:0/193181324 >> v2:172.21.15.174:3300/0 conn(0x7fb5b00b0760 msgr2=0x7fb5b00b1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:16.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.552+0000 7fb5aeffd700 1 --2- 172.21.15.98:0/193181324 >> v2:172.21.15.174:3300/0 conn(0x7fb5b00b0760 0x7fb5b00b1bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:16.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.552+0000 7fb5aeffd700 1 -- 172.21.15.98:0/193181324 >> v1:172.21.15.174:6789/0 conn(0x7fb5b00b49a0 legacy=0x7fb5b00aaa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:16.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.552+0000 7fb5aeffd700 1 -- 172.21.15.98:0/193181324 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5b00b1500 con 0x7fb5b0159560 2023-10-06T05:07:16.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.553+0000 7fb5aeffd700 1 --2- 172.21.15.98:0/193181324 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5b0159560 0x7fb5b00aa530 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb5a0009990 tx=0x7fb5a0009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=438cc8bf116d1019 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:16.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.553+0000 7fb5ae7fc700 1 -- 172.21.15.98:0/193181324 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5a0005c10 con 0x7fb5b0159560 2023-10-06T05:07:16.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.553+0000 7fb5ae7fc700 1 -- 172.21.15.98:0/193181324 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5a0005d70 con 0x7fb5b0159560 2023-10-06T05:07:16.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.553+0000 7fb5ae7fc700 1 -- 172.21.15.98:0/193181324 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5a000b800 con 0x7fb5b0159560 2023-10-06T05:07:16.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.553+0000 7fb5b5391700 1 -- 172.21.15.98:0/193181324 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5b0159560 msgr2=0x7fb5b00aa530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:16.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.553+0000 7fb5b5391700 1 --2- 172.21.15.98:0/193181324 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5b0159560 0x7fb5b00aa530 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb5a0009990 tx=0x7fb5a0009220 comp rx=0 tx=0).stop 2023-10-06T05:07:16.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.553+0000 7fb5b5391700 1 -- 172.21.15.98:0/193181324 shutdown_connections 2023-10-06T05:07:16.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.553+0000 7fb5b5391700 1 --2- 172.21.15.98:0/193181324 >> v2:172.21.15.174:3300/0 conn(0x7fb5b00b0760 0x7fb5b00b1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:16.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.553+0000 7fb5b5391700 1 --2- 172.21.15.98:0/193181324 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5b0159560 0x7fb5b00aa530 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:16.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.553+0000 7fb5b5391700 1 -- 172.21.15.98:0/193181324 >> 172.21.15.98:0/193181324 conn(0x7fb5b00997f0 msgr2=0x7fb5b006f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:16.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.553+0000 7fb5b5391700 1 -- 172.21.15.98:0/193181324 shutdown_connections 2023-10-06T05:07:16.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.553+0000 7fb5b5391700 1 -- 172.21.15.98:0/193181324 wait complete. 2023-10-06T05:07:16.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.555+0000 7fb5b5391700 1 Processor -- start 2023-10-06T05:07:16.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.555+0000 7fb5b5391700 1 -- start start 2023-10-06T05:07:16.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.555+0000 7fb5b5391700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb5b00b0760 0x7fb5b00ab200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:16.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.555+0000 7fb5b5391700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb5b00b49a0 0x7fb5b009b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:16.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.555+0000 7fb5b5391700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5b0159560 0x7fb5b009c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:16.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.555+0000 7fb5b5391700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5b00ab740 con 0x7fb5b0159560 2023-10-06T05:07:16.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.555+0000 7fb5b5391700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5b009a3b0 con 0x7fb5b00b49a0 2023-10-06T05:07:16.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.555+0000 7fb5b5391700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5b009a690 con 0x7fb5b00b0760 2023-10-06T05:07:16.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.556+0000 7fb5af7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5b0159560 0x7fb5b009c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:16.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.556+0000 7fb5af7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5b0159560 0x7fb5b009c130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46030/0 (socket says 172.21.15.98:46030) 2023-10-06T05:07:16.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.556+0000 7fb5af7fe700 1 -- 172.21.15.98:0/1381409654 learned_addr learned my addr 172.21.15.98:0/1381409654 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:16.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.556+0000 7fb5a67fc700 1 --2- 172.21.15.98:0/1381409654 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb5b00b49a0 0x7fb5b009b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:16.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.556+0000 7fb5aeffd700 1 --2- 172.21.15.98:0/1381409654 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb5b00b0760 0x7fb5b00ab200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:16.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.556+0000 7fb5af7fe700 1 -- 172.21.15.98:0/1381409654 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb5b00b0760 msgr2=0x7fb5b00ab200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:16.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.556+0000 7fb5af7fe700 1 --2- 172.21.15.98:0/1381409654 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb5b00b0760 0x7fb5b00ab200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:16.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.556+0000 7fb5af7fe700 1 -- 172.21.15.98:0/1381409654 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb5b00b49a0 msgr2=0x7fb5b009b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:16.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.556+0000 7fb5af7fe700 1 --2- 172.21.15.98:0/1381409654 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb5b00b49a0 0x7fb5b009b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:16.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.556+0000 7fb5af7fe700 1 -- 172.21.15.98:0/1381409654 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5a0009480 con 0x7fb5b0159560 2023-10-06T05:07:16.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.556+0000 7fb5a67fc700 1 --2- 172.21.15.98:0/1381409654 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb5b00b49a0 0x7fb5b009b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:07:16.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.556+0000 7fb5af7fe700 1 --2- 172.21.15.98:0/1381409654 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5b0159560 0x7fb5b009c130 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb5a8009fa0 tx=0x7fb5a8009900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:16.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.556+0000 7fb5a77fe700 1 -- 172.21.15.98:0/1381409654 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5a800ed20 con 0x7fb5b0159560 2023-10-06T05:07:16.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.556+0000 7fb5a77fe700 1 -- 172.21.15.98:0/1381409654 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5a800ee80 con 0x7fb5b0159560 2023-10-06T05:07:16.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.556+0000 7fb5b5391700 1 -- 172.21.15.98:0/1381409654 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5b009a970 con 0x7fb5b0159560 2023-10-06T05:07:16.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.556+0000 7fb5a77fe700 1 -- 172.21.15.98:0/1381409654 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5a8014670 con 0x7fb5b0159560 2023-10-06T05:07:16.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.556+0000 7fb5b5391700 1 -- 172.21.15.98:0/1381409654 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5b009ad00 con 0x7fb5b0159560 2023-10-06T05:07:16.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.557+0000 7fb5b5391700 1 -- 172.21.15.98:0/1381409654 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5b009b2a0 con 0x7fb5b0159560 2023-10-06T05:07:16.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.558+0000 7fb5a77fe700 1 -- 172.21.15.98:0/1381409654 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fb5a80149e0 con 0x7fb5b0159560 2023-10-06T05:07:16.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.558+0000 7fb5a77fe700 1 --2- 172.21.15.98:0/1381409654 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fb590066f30 0x7fb5900693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:16.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.558+0000 7fb5a77fe700 1 -- 172.21.15.98:0/1381409654 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fb5a8085220 con 0x7fb5b0159560 2023-10-06T05:07:16.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.558+0000 7fb5aeffd700 1 --2- 172.21.15.98:0/1381409654 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fb590066f30 0x7fb5900693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:16.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.559+0000 7fb5aeffd700 1 --2- 172.21.15.98:0/1381409654 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fb590066f30 0x7fb5900693e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb5a0007fd0 tx=0x7fb5a0015000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:16.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.563+0000 7fb5a77fe700 1 -- 172.21.15.98:0/1381409654 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb5a8051fa0 con 0x7fb5b0159560 2023-10-06T05:07:16.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.743+0000 7fb5b5391700 1 -- 172.21.15.98:0/1381409654 --> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb5b008aa10 con 0x7fb590066f30 2023-10-06T05:07:16.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.753+0000 7fb5a77fe700 1 -- 172.21.15.98:0/1381409654 <== mgr.4104 v2:172.21.15.98:6832/90397 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+367385 (secure 0 0 0) 0x7fb5b008aa10 con 0x7fb590066f30 2023-10-06T05:07:16.760 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:07:16.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.756+0000 7fb5b5391700 1 -- 172.21.15.98:0/1381409654 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fb590066f30 msgr2=0x7fb5900693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:16.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.756+0000 7fb5b5391700 1 --2- 172.21.15.98:0/1381409654 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fb590066f30 0x7fb5900693e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb5a0007fd0 tx=0x7fb5a0015000 comp rx=0 tx=0).stop 2023-10-06T05:07:16.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.756+0000 7fb5b5391700 1 -- 172.21.15.98:0/1381409654 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5b0159560 msgr2=0x7fb5b009c130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:16.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.756+0000 7fb5b5391700 1 --2- 172.21.15.98:0/1381409654 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5b0159560 0x7fb5b009c130 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb5a8009fa0 tx=0x7fb5a8009900 comp rx=0 tx=0).stop 2023-10-06T05:07:16.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.756+0000 7fb5b5391700 1 -- 172.21.15.98:0/1381409654 shutdown_connections 2023-10-06T05:07:16.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.756+0000 7fb5b5391700 1 --2- 172.21.15.98:0/1381409654 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fb590066f30 0x7fb5900693e0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:16.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.756+0000 7fb5b5391700 1 --2- 172.21.15.98:0/1381409654 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5b0159560 0x7fb5b009c130 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:16.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.756+0000 7fb5b5391700 1 --2- 172.21.15.98:0/1381409654 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb5b00b0760 0x7fb5b00ab200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:16.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.756+0000 7fb5b5391700 1 --2- 172.21.15.98:0/1381409654 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb5b00b49a0 0x7fb5b009b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:16.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.756+0000 7fb5b5391700 1 -- 172.21.15.98:0/1381409654 >> 172.21.15.98:0/1381409654 conn(0x7fb5b00997f0 msgr2=0x7fb5b00b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:16.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.756+0000 7fb5b5391700 1 -- 172.21.15.98:0/1381409654 shutdown_connections 2023-10-06T05:07:16.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.756+0000 7fb5b5391700 1 -- 172.21.15.98:0/1381409654 wait complete. 2023-10-06T05:07:16.766 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2023-10-06T05:07:16.770 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":39,"stamp":"2023-10-06T05:07:16.523698+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594852,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":338,"acting":338,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":338,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2381928,"kb_used_data":3624,"kb_used_omap":0,"kb_used_meta":2378240,"kb_avail":752592792,"statfs":{"total":773094113280,"available":770655019008,"internally_reserved":0,"allocated":3710976,"data_stored":2198496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2435317760},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2178,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002893"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.753513+0000","last_change":"2023-10-06T05:06:57.619460+0000","last_active":"2023-10-06T05:07:01.753513+0000","last_peered":"2023-10-06T05:07:01.753513+0000","last_clean":"2023-10-06T05:07:01.753513+0000","last_became_active":"2023-10-06T05:06:57.619199+0000","last_became_peered":"2023-10-06T05:06:57.619199+0000","last_unstale":"2023-10-06T05:07:01.753513+0000","last_undegraded":"2023-10-06T05:07:01.753513+0000","last_fullsized":"2023-10-06T05:07:01.753513+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:49:28.024174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.747458+0000","last_change":"2023-10-06T05:06:57.610280+0000","last_active":"2023-10-06T05:07:01.747458+0000","last_peered":"2023-10-06T05:07:01.747458+0000","last_clean":"2023-10-06T05:07:01.747458+0000","last_became_active":"2023-10-06T05:06:57.609973+0000","last_became_peered":"2023-10-06T05:06:57.609973+0000","last_unstale":"2023-10-06T05:07:01.747458+0000","last_undegraded":"2023-10-06T05:07:01.747458+0000","last_fullsized":"2023-10-06T05:07:01.747458+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:22:11.051575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.762104+0000","last_change":"2023-10-06T05:06:57.615468+0000","last_active":"2023-10-06T05:07:01.762104+0000","last_peered":"2023-10-06T05:07:01.762104+0000","last_clean":"2023-10-06T05:07:01.762104+0000","last_became_active":"2023-10-06T05:06:57.615221+0000","last_became_peered":"2023-10-06T05:06:57.615221+0000","last_unstale":"2023-10-06T05:07:01.762104+0000","last_undegraded":"2023-10-06T05:07:01.762104+0000","last_fullsized":"2023-10-06T05:07:01.762104+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:49:48.309158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.754474+0000","last_change":"2023-10-06T05:06:57.614470+0000","last_active":"2023-10-06T05:07:01.754474+0000","last_peered":"2023-10-06T05:07:01.754474+0000","last_clean":"2023-10-06T05:07:01.754474+0000","last_became_active":"2023-10-06T05:06:57.614169+0000","last_became_peered":"2023-10-06T05:06:57.614169+0000","last_unstale":"2023-10-06T05:07:01.754474+0000","last_undegraded":"2023-10-06T05:07:01.754474+0000","last_fullsized":"2023-10-06T05:07:01.754474+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:23:03.302621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750277+0000","last_change":"2023-10-06T05:06:57.622050+0000","last_active":"2023-10-06T05:07:01.750277+0000","last_peered":"2023-10-06T05:07:01.750277+0000","last_clean":"2023-10-06T05:07:01.750277+0000","last_became_active":"2023-10-06T05:06:57.621777+0000","last_became_peered":"2023-10-06T05:06:57.621777+0000","last_unstale":"2023-10-06T05:07:01.750277+0000","last_undegraded":"2023-10-06T05:07:01.750277+0000","last_fullsized":"2023-10-06T05:07:01.750277+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:04:40.577463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.761680+0000","last_change":"2023-10-06T05:06:57.611449+0000","last_active":"2023-10-06T05:07:01.761680+0000","last_peered":"2023-10-06T05:07:01.761680+0000","last_clean":"2023-10-06T05:07:01.761680+0000","last_became_active":"2023-10-06T05:06:57.610301+0000","last_became_peered":"2023-10-06T05:06:57.610301+0000","last_unstale":"2023-10-06T05:07:01.761680+0000","last_undegraded":"2023-10-06T05:07:01.761680+0000","last_fullsized":"2023-10-06T05:07:01.761680+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:43:21.654853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.755118+0000","last_change":"2023-10-06T05:06:57.610678+0000","last_active":"2023-10-06T05:07:01.755118+0000","last_peered":"2023-10-06T05:07:01.755118+0000","last_clean":"2023-10-06T05:07:01.755118+0000","last_became_active":"2023-10-06T05:06:57.609746+0000","last_became_peered":"2023-10-06T05:06:57.609746+0000","last_unstale":"2023-10-06T05:07:01.755118+0000","last_undegraded":"2023-10-06T05:07:01.755118+0000","last_fullsized":"2023-10-06T05:07:01.755118+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:11:00.525585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.745457+0000","last_change":"2023-10-06T05:06:57.612559+0000","last_active":"2023-10-06T05:07:01.745457+0000","last_peered":"2023-10-06T05:07:01.745457+0000","last_clean":"2023-10-06T05:07:01.745457+0000","last_became_active":"2023-10-06T05:06:57.612199+0000","last_became_peered":"2023-10-06T05:06:57.612199+0000","last_unstale":"2023-10-06T05:07:01.745457+0000","last_undegraded":"2023-10-06T05:07:01.745457+0000","last_fullsized":"2023-10-06T05:07:01.745457+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:01:11.983432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.751122+0000","last_change":"2023-10-06T05:06:57.618496+0000","last_active":"2023-10-06T05:07:01.751122+0000","last_peered":"2023-10-06T05:07:01.751122+0000","last_clean":"2023-10-06T05:07:01.751122+0000","last_became_active":"2023-10-06T05:06:57.618261+0000","last_became_peered":"2023-10-06T05:06:57.618261+0000","last_unstale":"2023-10-06T05:07:01.751122+0000","last_undegraded":"2023-10-06T05:07:01.751122+0000","last_fullsized":"2023-10-06T05:07:01.751122+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:24:33.077189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.747636+0000","last_change":"2023-10-06T05:06:57.614417+0000","last_active":"2023-10-06T05:07:01.747636+0000","last_peered":"2023-10-06T05:07:01.747636+0000","last_clean":"2023-10-06T05:07:01.747636+0000","last_became_active":"2023-10-06T05:06:57.614144+0000","last_became_peered":"2023-10-06T05:06:57.614144+0000","last_unstale":"2023-10-06T05:07:01.747636+0000","last_undegraded":"2023-10-06T05:07:01.747636+0000","last_fullsized":"2023-10-06T05:07:01.747636+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:32:33.731449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.762013+0000","last_change":"2023-10-06T05:06:57.610272+0000","last_active":"2023-10-06T05:07:01.762013+0000","last_peered":"2023-10-06T05:07:01.762013+0000","last_clean":"2023-10-06T05:07:01.762013+0000","last_became_active":"2023-10-06T05:06:57.609985+0000","last_became_peered":"2023-10-06T05:06:57.609985+0000","last_unstale":"2023-10-06T05:07:01.762013+0000","last_undegraded":"2023-10-06T05:07:01.762013+0000","last_fullsized":"2023-10-06T05:07:01.762013+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:20:04.673613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.745990+0000","last_change":"2023-10-06T05:06:57.606852+0000","last_active":"2023-10-06T05:07:01.745990+0000","last_peered":"2023-10-06T05:07:01.745990+0000","last_clean":"2023-10-06T05:07:01.745990+0000","last_became_active":"2023-10-06T05:06:57.606582+0000","last_became_peered":"2023-10-06T05:06:57.606582+0000","last_unstale":"2023-10-06T05:07:01.745990+0000","last_undegraded":"2023-10-06T05:07:01.745990+0000","last_fullsized":"2023-10-06T05:07:01.745990+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:23:03.302621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.753767+0000","last_change":"2023-10-06T05:06:57.615706+0000","last_active":"2023-10-06T05:07:01.753767+0000","last_peered":"2023-10-06T05:07:01.753767+0000","last_clean":"2023-10-06T05:07:01.753767+0000","last_became_active":"2023-10-06T05:06:57.615408+0000","last_became_peered":"2023-10-06T05:06:57.615408+0000","last_unstale":"2023-10-06T05:07:01.753767+0000","last_undegraded":"2023-10-06T05:07:01.753767+0000","last_fullsized":"2023-10-06T05:07:01.753767+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:38:51.879514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.744412+0000","last_change":"2023-10-06T05:06:57.613222+0000","last_active":"2023-10-06T05:07:01.744412+0000","last_peered":"2023-10-06T05:07:01.744412+0000","last_clean":"2023-10-06T05:07:01.744412+0000","last_became_active":"2023-10-06T05:06:57.612924+0000","last_became_peered":"2023-10-06T05:06:57.612924+0000","last_unstale":"2023-10-06T05:07:01.744412+0000","last_undegraded":"2023-10-06T05:07:01.744412+0000","last_fullsized":"2023-10-06T05:07:01.744412+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:29:19.951056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.754947+0000","last_change":"2023-10-06T05:06:57.610023+0000","last_active":"2023-10-06T05:07:01.754947+0000","last_peered":"2023-10-06T05:07:01.754947+0000","last_clean":"2023-10-06T05:07:01.754947+0000","last_became_active":"2023-10-06T05:06:57.609367+0000","last_became_peered":"2023-10-06T05:06:57.609367+0000","last_unstale":"2023-10-06T05:07:01.754947+0000","last_undegraded":"2023-10-06T05:07:01.754947+0000","last_fullsized":"2023-10-06T05:07:01.754947+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:34:55.987076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.758813+0000","last_change":"2023-10-06T05:06:57.610369+0000","last_active":"2023-10-06T05:07:01.758813+0000","last_peered":"2023-10-06T05:07:01.758813+0000","last_clean":"2023-10-06T05:07:01.758813+0000","last_became_active":"2023-10-06T05:06:57.610044+0000","last_became_peered":"2023-10-06T05:06:57.610044+0000","last_unstale":"2023-10-06T05:07:01.758813+0000","last_undegraded":"2023-10-06T05:07:01.758813+0000","last_fullsized":"2023-10-06T05:07:01.758813+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:10:12.061112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.751198+0000","last_change":"2023-10-06T05:06:57.619477+0000","last_active":"2023-10-06T05:07:01.751198+0000","last_peered":"2023-10-06T05:07:01.751198+0000","last_clean":"2023-10-06T05:07:01.751198+0000","last_became_active":"2023-10-06T05:06:57.618021+0000","last_became_peered":"2023-10-06T05:06:57.618021+0000","last_unstale":"2023-10-06T05:07:01.751198+0000","last_undegraded":"2023-10-06T05:07:01.751198+0000","last_fullsized":"2023-10-06T05:07:01.751198+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:58:02.572423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.761093+0000","last_change":"2023-10-06T05:06:57.602802+0000","last_active":"2023-10-06T05:07:01.761093+0000","last_peered":"2023-10-06T05:07:01.761093+0000","last_clean":"2023-10-06T05:07:01.761093+0000","last_became_active":"2023-10-06T05:06:57.602513+0000","last_became_peered":"2023-10-06T05:06:57.602513+0000","last_unstale":"2023-10-06T05:07:01.761093+0000","last_undegraded":"2023-10-06T05:07:01.761093+0000","last_fullsized":"2023-10-06T05:07:01.761093+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T16:34:15.990651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.747139+0000","last_change":"2023-10-06T05:06:57.608190+0000","last_active":"2023-10-06T05:07:01.747139+0000","last_peered":"2023-10-06T05:07:01.747139+0000","last_clean":"2023-10-06T05:07:01.747139+0000","last_became_active":"2023-10-06T05:06:57.607936+0000","last_became_peered":"2023-10-06T05:06:57.607936+0000","last_unstale":"2023-10-06T05:07:01.747139+0000","last_undegraded":"2023-10-06T05:07:01.747139+0000","last_fullsized":"2023-10-06T05:07:01.747139+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:00:30.035770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.761424+0000","last_change":"2023-10-06T05:06:57.611226+0000","last_active":"2023-10-06T05:07:01.761424+0000","last_peered":"2023-10-06T05:07:01.761424+0000","last_clean":"2023-10-06T05:07:01.761424+0000","last_became_active":"2023-10-06T05:06:57.610931+0000","last_became_peered":"2023-10-06T05:06:57.610931+0000","last_unstale":"2023-10-06T05:07:01.761424+0000","last_undegraded":"2023-10-06T05:07:01.761424+0000","last_fullsized":"2023-10-06T05:07:01.761424+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:52:14.772801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.749471+0000","last_change":"2023-10-06T05:06:57.615682+0000","last_active":"2023-10-06T05:07:01.749471+0000","last_peered":"2023-10-06T05:07:01.749471+0000","last_clean":"2023-10-06T05:07:01.749471+0000","last_became_active":"2023-10-06T05:06:57.615408+0000","last_became_peered":"2023-10-06T05:06:57.615408+0000","last_unstale":"2023-10-06T05:07:01.749471+0000","last_undegraded":"2023-10-06T05:07:01.749471+0000","last_fullsized":"2023-10-06T05:07:01.749471+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:58:02.572423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.749868+0000","last_change":"2023-10-06T05:06:57.619795+0000","last_active":"2023-10-06T05:07:01.749868+0000","last_peered":"2023-10-06T05:07:01.749868+0000","last_clean":"2023-10-06T05:07:01.749868+0000","last_became_active":"2023-10-06T05:06:57.619533+0000","last_became_peered":"2023-10-06T05:06:57.619533+0000","last_unstale":"2023-10-06T05:07:01.749868+0000","last_undegraded":"2023-10-06T05:07:01.749868+0000","last_fullsized":"2023-10-06T05:07:01.749868+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:11:00.525585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750332+0000","last_change":"2023-10-06T05:06:57.617926+0000","last_active":"2023-10-06T05:07:01.750332+0000","last_peered":"2023-10-06T05:07:01.750332+0000","last_clean":"2023-10-06T05:07:01.750332+0000","last_became_active":"2023-10-06T05:06:57.617128+0000","last_became_peered":"2023-10-06T05:06:57.617128+0000","last_unstale":"2023-10-06T05:07:01.750332+0000","last_undegraded":"2023-10-06T05:07:01.750332+0000","last_fullsized":"2023-10-06T05:07:01.750332+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:38:16.871694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.748479+0000","last_change":"2023-10-06T05:06:57.620090+0000","last_active":"2023-10-06T05:07:01.748479+0000","last_peered":"2023-10-06T05:07:01.748479+0000","last_clean":"2023-10-06T05:07:01.748479+0000","last_became_active":"2023-10-06T05:06:57.619892+0000","last_became_peered":"2023-10-06T05:06:57.619892+0000","last_unstale":"2023-10-06T05:07:01.748479+0000","last_undegraded":"2023-10-06T05:07:01.748479+0000","last_fullsized":"2023-10-06T05:07:01.748479+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:45:23.272110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750371+0000","last_change":"2023-10-06T05:06:57.613844+0000","last_active":"2023-10-06T05:07:01.750371+0000","last_peered":"2023-10-06T05:07:01.750371+0000","last_clean":"2023-10-06T05:07:01.750371+0000","last_became_active":"2023-10-06T05:06:57.613513+0000","last_became_peered":"2023-10-06T05:06:57.613513+0000","last_unstale":"2023-10-06T05:07:01.750371+0000","last_undegraded":"2023-10-06T05:07:01.750371+0000","last_fullsized":"2023-10-06T05:07:01.750371+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:53:03.171108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.748759+0000","last_change":"2023-10-06T05:06:57.612464+0000","last_active":"2023-10-06T05:07:01.748759+0000","last_peered":"2023-10-06T05:07:01.748759+0000","last_clean":"2023-10-06T05:07:01.748759+0000","last_became_active":"2023-10-06T05:06:57.612179+0000","last_became_peered":"2023-10-06T05:06:57.612179+0000","last_unstale":"2023-10-06T05:07:01.748759+0000","last_undegraded":"2023-10-06T05:07:01.748759+0000","last_fullsized":"2023-10-06T05:07:01.748759+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:12:34.550298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.744832+0000","last_change":"2023-10-06T05:06:57.612625+0000","last_active":"2023-10-06T05:07:01.744832+0000","last_peered":"2023-10-06T05:07:01.744832+0000","last_clean":"2023-10-06T05:07:01.744832+0000","last_became_active":"2023-10-06T05:06:57.611582+0000","last_became_peered":"2023-10-06T05:06:57.611582+0000","last_unstale":"2023-10-06T05:07:01.744832+0000","last_undegraded":"2023-10-06T05:07:01.744832+0000","last_fullsized":"2023-10-06T05:07:01.744832+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:59:39.994965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.768197+0000","last_change":"2023-10-06T05:07:01.765965+0000","last_active":"2023-10-06T05:07:01.768197+0000","last_peered":"2023-10-06T05:07:01.768197+0000","last_clean":"2023-10-06T05:07:01.768197+0000","last_became_active":"2023-10-06T05:07:01.764635+0000","last_became_peered":"2023-10-06T05:07:01.764635+0000","last_unstale":"2023-10-06T05:07:01.768197+0000","last_undegraded":"2023-10-06T05:07:01.768197+0000","last_fullsized":"2023-10-06T05:07:01.768197+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:18:43.042335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2,0],"acting":[4,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"24'1","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.760983+0000","last_change":"2023-10-06T05:06:56.716498+0000","last_active":"2023-10-06T05:07:01.760983+0000","last_peered":"2023-10-06T05:07:01.760983+0000","last_clean":"2023-10-06T05:07:01.760983+0000","last_became_active":"2023-10-06T05:06:56.714771+0000","last_became_peered":"2023-10-06T05:06:56.714771+0000","last_unstale":"2023-10-06T05:07:01.760983+0000","last_undegraded":"2023-10-06T05:07:01.760983+0000","last_fullsized":"2023-10-06T05:07:01.760983+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:53:56.989908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.752793+0000","last_change":"2023-10-06T05:06:57.617712+0000","last_active":"2023-10-06T05:07:01.752793+0000","last_peered":"2023-10-06T05:07:01.752793+0000","last_clean":"2023-10-06T05:07:01.752793+0000","last_became_active":"2023-10-06T05:06:57.616946+0000","last_became_peered":"2023-10-06T05:06:57.616946+0000","last_unstale":"2023-10-06T05:07:01.752793+0000","last_undegraded":"2023-10-06T05:07:01.752793+0000","last_fullsized":"2023-10-06T05:07:01.752793+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:21:13.192156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.767576+0000","last_change":"2023-10-06T05:07:01.763425+0000","last_active":"2023-10-06T05:07:01.767576+0000","last_peered":"2023-10-06T05:07:01.767576+0000","last_clean":"2023-10-06T05:07:01.767576+0000","last_became_active":"2023-10-06T05:07:01.763035+0000","last_became_peered":"2023-10-06T05:07:01.763035+0000","last_unstale":"2023-10-06T05:07:01.767576+0000","last_undegraded":"2023-10-06T05:07:01.767576+0000","last_fullsized":"2023-10-06T05:07:01.767576+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:51:19.615655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3,7],"acting":[6,4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.745972+0000","last_change":"2023-10-06T05:06:56.714205+0000","last_active":"2023-10-06T05:07:01.745972+0000","last_peered":"2023-10-06T05:07:01.745972+0000","last_clean":"2023-10-06T05:07:01.745972+0000","last_became_active":"2023-10-06T05:06:56.713947+0000","last_became_peered":"2023-10-06T05:06:56.713947+0000","last_unstale":"2023-10-06T05:07:01.745972+0000","last_undegraded":"2023-10-06T05:07:01.745972+0000","last_fullsized":"2023-10-06T05:07:01.745972+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:54:07.584005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750046+0000","last_change":"2023-10-06T05:06:57.616019+0000","last_active":"2023-10-06T05:07:01.750046+0000","last_peered":"2023-10-06T05:07:01.750046+0000","last_clean":"2023-10-06T05:07:01.750046+0000","last_became_active":"2023-10-06T05:06:57.615747+0000","last_became_peered":"2023-10-06T05:06:57.615747+0000","last_unstale":"2023-10-06T05:07:01.750046+0000","last_undegraded":"2023-10-06T05:07:01.750046+0000","last_fullsized":"2023-10-06T05:07:01.750046+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:07:52.020683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.770991+0000","last_change":"2023-10-06T05:07:01.767383+0000","last_active":"2023-10-06T05:07:01.770991+0000","last_peered":"2023-10-06T05:07:01.770991+0000","last_clean":"2023-10-06T05:07:01.770991+0000","last_became_active":"2023-10-06T05:07:01.766997+0000","last_became_peered":"2023-10-06T05:07:01.766997+0000","last_unstale":"2023-10-06T05:07:01.770991+0000","last_undegraded":"2023-10-06T05:07:01.770991+0000","last_fullsized":"2023-10-06T05:07:01.770991+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:45:03.209108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1,2],"acting":[0,7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.749601+0000","last_change":"2023-10-06T05:06:56.706133+0000","last_active":"2023-10-06T05:07:01.749601+0000","last_peered":"2023-10-06T05:07:01.749601+0000","last_clean":"2023-10-06T05:07:01.749601+0000","last_became_active":"2023-10-06T05:06:56.705867+0000","last_became_peered":"2023-10-06T05:06:56.705867+0000","last_unstale":"2023-10-06T05:07:01.749601+0000","last_undegraded":"2023-10-06T05:07:01.749601+0000","last_fullsized":"2023-10-06T05:07:01.749601+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:55:08.124129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.751243+0000","last_change":"2023-10-06T05:06:57.608813+0000","last_active":"2023-10-06T05:07:01.751243+0000","last_peered":"2023-10-06T05:07:01.751243+0000","last_clean":"2023-10-06T05:07:01.751243+0000","last_became_active":"2023-10-06T05:06:57.608497+0000","last_became_peered":"2023-10-06T05:06:57.608497+0000","last_unstale":"2023-10-06T05:07:01.751243+0000","last_undegraded":"2023-10-06T05:07:01.751243+0000","last_fullsized":"2023-10-06T05:07:01.751243+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:30:19.509164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.770179+0000","last_change":"2023-10-06T05:07:01.765801+0000","last_active":"2023-10-06T05:07:01.770179+0000","last_peered":"2023-10-06T05:07:01.770179+0000","last_clean":"2023-10-06T05:07:01.770179+0000","last_became_active":"2023-10-06T05:07:01.765422+0000","last_became_peered":"2023-10-06T05:07:01.765422+0000","last_unstale":"2023-10-06T05:07:01.770179+0000","last_undegraded":"2023-10-06T05:07:01.770179+0000","last_fullsized":"2023-10-06T05:07:01.770179+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:20:08.719247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7,1],"acting":[6,4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.751276+0000","last_change":"2023-10-06T05:06:56.714826+0000","last_active":"2023-10-06T05:07:01.751276+0000","last_peered":"2023-10-06T05:07:01.751276+0000","last_clean":"2023-10-06T05:07:01.751276+0000","last_became_active":"2023-10-06T05:06:56.714564+0000","last_became_peered":"2023-10-06T05:06:56.714564+0000","last_unstale":"2023-10-06T05:07:01.751276+0000","last_undegraded":"2023-10-06T05:07:01.751276+0000","last_fullsized":"2023-10-06T05:07:01.751276+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:19:13.930707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.749520+0000","last_change":"2023-10-06T05:06:57.618934+0000","last_active":"2023-10-06T05:07:01.749520+0000","last_peered":"2023-10-06T05:07:01.749520+0000","last_clean":"2023-10-06T05:07:01.749520+0000","last_became_active":"2023-10-06T05:06:57.618410+0000","last_became_peered":"2023-10-06T05:06:57.618410+0000","last_unstale":"2023-10-06T05:07:01.749520+0000","last_undegraded":"2023-10-06T05:07:01.749520+0000","last_fullsized":"2023-10-06T05:07:01.749520+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:51:15.570020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.775608+0000","last_change":"2023-10-06T05:07:01.766370+0000","last_active":"2023-10-06T05:07:01.775608+0000","last_peered":"2023-10-06T05:07:01.775608+0000","last_clean":"2023-10-06T05:07:01.775608+0000","last_became_active":"2023-10-06T05:07:01.765994+0000","last_became_peered":"2023-10-06T05:07:01.765994+0000","last_unstale":"2023-10-06T05:07:01.775608+0000","last_undegraded":"2023-10-06T05:07:01.775608+0000","last_fullsized":"2023-10-06T05:07:01.775608+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:20:46.564009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6,0],"acting":[7,4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.747549+0000","last_change":"2023-10-06T05:06:56.708479+0000","last_active":"2023-10-06T05:07:01.747549+0000","last_peered":"2023-10-06T05:07:01.747549+0000","last_clean":"2023-10-06T05:07:01.747549+0000","last_became_active":"2023-10-06T05:06:56.708155+0000","last_became_peered":"2023-10-06T05:06:56.708155+0000","last_unstale":"2023-10-06T05:07:01.747549+0000","last_undegraded":"2023-10-06T05:07:01.747549+0000","last_fullsized":"2023-10-06T05:07:01.747549+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:30:18.507322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750417+0000","last_change":"2023-10-06T05:06:57.617995+0000","last_active":"2023-10-06T05:07:01.750417+0000","last_peered":"2023-10-06T05:07:01.750417+0000","last_clean":"2023-10-06T05:07:01.750417+0000","last_became_active":"2023-10-06T05:06:57.617577+0000","last_became_peered":"2023-10-06T05:06:57.617577+0000","last_unstale":"2023-10-06T05:07:01.750417+0000","last_undegraded":"2023-10-06T05:07:01.750417+0000","last_fullsized":"2023-10-06T05:07:01.750417+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:01:11.983432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.763161+0000","last_change":"2023-10-06T05:07:01.761436+0000","last_active":"2023-10-06T05:07:01.763161+0000","last_peered":"2023-10-06T05:07:01.763161+0000","last_clean":"2023-10-06T05:07:01.763161+0000","last_became_active":"2023-10-06T05:07:01.761049+0000","last_became_peered":"2023-10-06T05:07:01.761049+0000","last_unstale":"2023-10-06T05:07:01.763161+0000","last_undegraded":"2023-10-06T05:07:01.763161+0000","last_fullsized":"2023-10-06T05:07:01.763161+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:09:13.387478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,5],"acting":[3,0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"24'3","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.753446+0000","last_change":"2023-10-06T05:06:56.717320+0000","last_active":"2023-10-06T05:07:01.753446+0000","last_peered":"2023-10-06T05:07:01.753446+0000","last_clean":"2023-10-06T05:07:01.753446+0000","last_became_active":"2023-10-06T05:06:56.716985+0000","last_became_peered":"2023-10-06T05:06:56.716985+0000","last_unstale":"2023-10-06T05:07:01.753446+0000","last_undegraded":"2023-10-06T05:07:01.753446+0000","last_fullsized":"2023-10-06T05:07:01.753446+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:53:02.169267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.747499+0000","last_change":"2023-10-06T05:06:57.613655+0000","last_active":"2023-10-06T05:07:01.747499+0000","last_peered":"2023-10-06T05:07:01.747499+0000","last_clean":"2023-10-06T05:07:01.747499+0000","last_became_active":"2023-10-06T05:06:57.613378+0000","last_became_peered":"2023-10-06T05:06:57.613378+0000","last_unstale":"2023-10-06T05:07:01.747499+0000","last_undegraded":"2023-10-06T05:07:01.747499+0000","last_fullsized":"2023-10-06T05:07:01.747499+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:23:03.919621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.772389+0000","last_change":"2023-10-06T05:07:01.766400+0000","last_active":"2023-10-06T05:07:01.772389+0000","last_peered":"2023-10-06T05:07:01.772389+0000","last_clean":"2023-10-06T05:07:01.772389+0000","last_became_active":"2023-10-06T05:07:01.766034+0000","last_became_peered":"2023-10-06T05:07:01.766034+0000","last_unstale":"2023-10-06T05:07:01.772389+0000","last_undegraded":"2023-10-06T05:07:01.772389+0000","last_fullsized":"2023-10-06T05:07:01.772389+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:04:14.761320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3,7],"acting":[1,5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"29'9","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-06T05:07:07.500759+0000","last_change":"2023-10-06T05:06:56.716406+0000","last_active":"2023-10-06T05:07:07.500759+0000","last_peered":"2023-10-06T05:07:07.500759+0000","last_clean":"2023-10-06T05:07:07.500759+0000","last_became_active":"2023-10-06T05:06:56.716075+0000","last_became_peered":"2023-10-06T05:06:56.716075+0000","last_unstale":"2023-10-06T05:07:07.500759+0000","last_undegraded":"2023-10-06T05:07:07.500759+0000","last_fullsized":"2023-10-06T05:07:07.500759+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:36:39.323759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3714,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.761355+0000","last_change":"2023-10-06T05:06:57.605798+0000","last_active":"2023-10-06T05:07:01.761355+0000","last_peered":"2023-10-06T05:07:01.761355+0000","last_clean":"2023-10-06T05:07:01.761355+0000","last_became_active":"2023-10-06T05:06:57.605523+0000","last_became_peered":"2023-10-06T05:06:57.605523+0000","last_unstale":"2023-10-06T05:07:01.761355+0000","last_undegraded":"2023-10-06T05:07:01.761355+0000","last_fullsized":"2023-10-06T05:07:01.761355+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:32:33.731449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.765132+0000","last_change":"2023-10-06T05:07:01.760576+0000","last_active":"2023-10-06T05:07:01.765132+0000","last_peered":"2023-10-06T05:07:01.765132+0000","last_clean":"2023-10-06T05:07:01.765132+0000","last_became_active":"2023-10-06T05:07:01.760146+0000","last_became_peered":"2023-10-06T05:07:01.760146+0000","last_unstale":"2023-10-06T05:07:01.765132+0000","last_undegraded":"2023-10-06T05:07:01.765132+0000","last_fullsized":"2023-10-06T05:07:01.765132+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:30:21.024442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3,0],"acting":[5,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"24'2","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.748025+0000","last_change":"2023-10-06T05:06:56.722076+0000","last_active":"2023-10-06T05:07:01.748025+0000","last_peered":"2023-10-06T05:07:01.748025+0000","last_clean":"2023-10-06T05:07:01.748025+0000","last_became_active":"2023-10-06T05:06:56.721786+0000","last_became_peered":"2023-10-06T05:06:56.721786+0000","last_unstale":"2023-10-06T05:07:01.748025+0000","last_undegraded":"2023-10-06T05:07:01.748025+0000","last_fullsized":"2023-10-06T05:07:01.748025+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:15:54.616807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.749669+0000","last_change":"2023-10-06T05:06:57.620208+0000","last_active":"2023-10-06T05:07:01.749669+0000","last_peered":"2023-10-06T05:07:01.749669+0000","last_clean":"2023-10-06T05:07:01.749669+0000","last_became_active":"2023-10-06T05:06:57.619747+0000","last_became_peered":"2023-10-06T05:06:57.619747+0000","last_unstale":"2023-10-06T05:07:01.749669+0000","last_undegraded":"2023-10-06T05:07:01.749669+0000","last_fullsized":"2023-10-06T05:07:01.749669+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:52:14.772801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.764815+0000","last_change":"2023-10-06T05:07:01.762230+0000","last_active":"2023-10-06T05:07:01.764815+0000","last_peered":"2023-10-06T05:07:01.764815+0000","last_clean":"2023-10-06T05:07:01.764815+0000","last_became_active":"2023-10-06T05:07:01.761878+0000","last_became_peered":"2023-10-06T05:07:01.761878+0000","last_unstale":"2023-10-06T05:07:01.764815+0000","last_undegraded":"2023-10-06T05:07:01.764815+0000","last_fullsized":"2023-10-06T05:07:01.764815+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:18:24.586582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2,1],"acting":[3,7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.749791+0000","last_change":"2023-10-06T05:06:56.715016+0000","last_active":"2023-10-06T05:07:01.749791+0000","last_peered":"2023-10-06T05:07:01.749791+0000","last_clean":"2023-10-06T05:07:01.749791+0000","last_became_active":"2023-10-06T05:06:56.714513+0000","last_became_peered":"2023-10-06T05:06:56.714513+0000","last_unstale":"2023-10-06T05:07:01.749791+0000","last_undegraded":"2023-10-06T05:07:01.749791+0000","last_fullsized":"2023-10-06T05:07:01.749791+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T16:11:31.404755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.744725+0000","last_change":"2023-10-06T05:06:57.613062+0000","last_active":"2023-10-06T05:07:01.744725+0000","last_peered":"2023-10-06T05:07:01.744725+0000","last_clean":"2023-10-06T05:07:01.744725+0000","last_became_active":"2023-10-06T05:06:57.611524+0000","last_became_peered":"2023-10-06T05:06:57.611524+0000","last_unstale":"2023-10-06T05:07:01.744725+0000","last_undegraded":"2023-10-06T05:07:01.744725+0000","last_fullsized":"2023-10-06T05:07:01.744725+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:01:11.480811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.770644+0000","last_change":"2023-10-06T05:07:01.764987+0000","last_active":"2023-10-06T05:07:01.770644+0000","last_peered":"2023-10-06T05:07:01.770644+0000","last_clean":"2023-10-06T05:07:01.770644+0000","last_became_active":"2023-10-06T05:07:01.764475+0000","last_became_peered":"2023-10-06T05:07:01.764475+0000","last_unstale":"2023-10-06T05:07:01.770644+0000","last_undegraded":"2023-10-06T05:07:01.770644+0000","last_fullsized":"2023-10-06T05:07:01.770644+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:25:41.281713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,0],"acting":[3,1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.755037+0000","last_change":"2023-10-06T05:06:56.719174+0000","last_active":"2023-10-06T05:07:01.755037+0000","last_peered":"2023-10-06T05:07:01.755037+0000","last_clean":"2023-10-06T05:07:01.755037+0000","last_became_active":"2023-10-06T05:06:56.718930+0000","last_became_peered":"2023-10-06T05:06:56.718930+0000","last_unstale":"2023-10-06T05:07:01.755037+0000","last_undegraded":"2023-10-06T05:07:01.755037+0000","last_fullsized":"2023-10-06T05:07:01.755037+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:51:30.871635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750114+0000","last_change":"2023-10-06T05:06:57.617389+0000","last_active":"2023-10-06T05:07:01.750114+0000","last_peered":"2023-10-06T05:07:01.750114+0000","last_clean":"2023-10-06T05:07:01.750114+0000","last_became_active":"2023-10-06T05:06:57.617099+0000","last_became_peered":"2023-10-06T05:06:57.617099+0000","last_unstale":"2023-10-06T05:07:01.750114+0000","last_undegraded":"2023-10-06T05:07:01.750114+0000","last_fullsized":"2023-10-06T05:07:01.750114+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:51:15.929322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.759618+0000","last_change":"2023-10-06T05:07:01.756721+0000","last_active":"2023-10-06T05:07:01.759618+0000","last_peered":"2023-10-06T05:07:01.759618+0000","last_clean":"2023-10-06T05:07:01.759618+0000","last_became_active":"2023-10-06T05:07:01.756335+0000","last_became_peered":"2023-10-06T05:07:01.756335+0000","last_unstale":"2023-10-06T05:07:01.759618+0000","last_undegraded":"2023-10-06T05:07:01.759618+0000","last_fullsized":"2023-10-06T05:07:01.759618+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:20:29.759167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1,6],"acting":[5,3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"24'8","reported_seq":22,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750615+0000","last_change":"2023-10-06T05:06:56.713625+0000","last_active":"2023-10-06T05:07:01.750615+0000","last_peered":"2023-10-06T05:07:01.750615+0000","last_clean":"2023-10-06T05:07:01.750615+0000","last_became_active":"2023-10-06T05:06:56.713346+0000","last_became_peered":"2023-10-06T05:06:56.713346+0000","last_unstale":"2023-10-06T05:07:01.750615+0000","last_undegraded":"2023-10-06T05:07:01.750615+0000","last_fullsized":"2023-10-06T05:07:01.750615+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:57:59.037083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750303+0000","last_change":"2023-10-06T05:06:57.617670+0000","last_active":"2023-10-06T05:07:01.750303+0000","last_peered":"2023-10-06T05:07:01.750303+0000","last_clean":"2023-10-06T05:07:01.750303+0000","last_became_active":"2023-10-06T05:06:57.617259+0000","last_became_peered":"2023-10-06T05:06:57.617259+0000","last_unstale":"2023-10-06T05:07:01.750303+0000","last_undegraded":"2023-10-06T05:07:01.750303+0000","last_fullsized":"2023-10-06T05:07:01.750303+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:22:11.051575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.775868+0000","last_change":"2023-10-06T05:07:01.771047+0000","last_active":"2023-10-06T05:07:01.775868+0000","last_peered":"2023-10-06T05:07:01.775868+0000","last_clean":"2023-10-06T05:07:01.775868+0000","last_became_active":"2023-10-06T05:07:01.770692+0000","last_became_peered":"2023-10-06T05:07:01.770692+0000","last_unstale":"2023-10-06T05:07:01.775868+0000","last_undegraded":"2023-10-06T05:07:01.775868+0000","last_fullsized":"2023-10-06T05:07:01.775868+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:36:04.280555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3,4],"acting":[7,6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"24'1","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.752603+0000","last_change":"2023-10-06T05:06:56.722082+0000","last_active":"2023-10-06T05:07:01.752603+0000","last_peered":"2023-10-06T05:07:01.752603+0000","last_clean":"2023-10-06T05:07:01.752603+0000","last_became_active":"2023-10-06T05:06:56.721758+0000","last_became_peered":"2023-10-06T05:06:56.721758+0000","last_unstale":"2023-10-06T05:07:01.752603+0000","last_undegraded":"2023-10-06T05:07:01.752603+0000","last_fullsized":"2023-10-06T05:07:01.752603+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:10:26.413910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.745354+0000","last_change":"2023-10-06T05:06:57.611665+0000","last_active":"2023-10-06T05:07:01.745354+0000","last_peered":"2023-10-06T05:07:01.745354+0000","last_clean":"2023-10-06T05:07:01.745354+0000","last_became_active":"2023-10-06T05:06:57.611226+0000","last_became_peered":"2023-10-06T05:06:57.611226+0000","last_unstale":"2023-10-06T05:07:01.745354+0000","last_undegraded":"2023-10-06T05:07:01.745354+0000","last_fullsized":"2023-10-06T05:07:01.745354+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:26:29.348498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.761343+0000","last_change":"2023-10-06T05:07:01.760161+0000","last_active":"2023-10-06T05:07:01.761343+0000","last_peered":"2023-10-06T05:07:01.761343+0000","last_clean":"2023-10-06T05:07:01.761343+0000","last_became_active":"2023-10-06T05:07:01.759742+0000","last_became_peered":"2023-10-06T05:07:01.759742+0000","last_unstale":"2023-10-06T05:07:01.761343+0000","last_undegraded":"2023-10-06T05:07:01.761343+0000","last_fullsized":"2023-10-06T05:07:01.761343+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:10:16.106747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1,5],"acting":[2,0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"24'2","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.751770+0000","last_change":"2023-10-06T05:06:56.722004+0000","last_active":"2023-10-06T05:07:01.751770+0000","last_peered":"2023-10-06T05:07:01.751770+0000","last_clean":"2023-10-06T05:07:01.751770+0000","last_became_active":"2023-10-06T05:06:56.721717+0000","last_became_peered":"2023-10-06T05:06:56.721717+0000","last_unstale":"2023-10-06T05:07:01.751770+0000","last_undegraded":"2023-10-06T05:07:01.751770+0000","last_fullsized":"2023-10-06T05:07:01.751770+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:50:35.786032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.761552+0000","last_change":"2023-10-06T05:06:57.609279+0000","last_active":"2023-10-06T05:07:01.761552+0000","last_peered":"2023-10-06T05:07:01.761552+0000","last_clean":"2023-10-06T05:07:01.761552+0000","last_became_active":"2023-10-06T05:06:57.608982+0000","last_became_peered":"2023-10-06T05:06:57.608982+0000","last_unstale":"2023-10-06T05:07:01.761552+0000","last_undegraded":"2023-10-06T05:07:01.761552+0000","last_fullsized":"2023-10-06T05:07:01.761552+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:38:16.871694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.765310+0000","last_change":"2023-10-06T05:07:01.762960+0000","last_active":"2023-10-06T05:07:01.765310+0000","last_peered":"2023-10-06T05:07:01.765310+0000","last_clean":"2023-10-06T05:07:01.765310+0000","last_became_active":"2023-10-06T05:07:01.762308+0000","last_became_peered":"2023-10-06T05:07:01.762308+0000","last_unstale":"2023-10-06T05:07:01.765310+0000","last_undegraded":"2023-10-06T05:07:01.765310+0000","last_fullsized":"2023-10-06T05:07:01.765310+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:49:52.354792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5,6],"acting":[2,0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"24'1","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.754352+0000","last_change":"2023-10-06T05:06:56.710907+0000","last_active":"2023-10-06T05:07:01.754352+0000","last_peered":"2023-10-06T05:07:01.754352+0000","last_clean":"2023-10-06T05:07:01.754352+0000","last_became_active":"2023-10-06T05:06:56.710595+0000","last_became_peered":"2023-10-06T05:06:56.710595+0000","last_unstale":"2023-10-06T05:07:01.754352+0000","last_undegraded":"2023-10-06T05:07:01.754352+0000","last_fullsized":"2023-10-06T05:07:01.754352+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:17:31.213115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.751968+0000","last_change":"2023-10-06T05:06:57.611226+0000","last_active":"2023-10-06T05:07:01.751968+0000","last_peered":"2023-10-06T05:07:01.751968+0000","last_clean":"2023-10-06T05:07:01.751968+0000","last_became_active":"2023-10-06T05:06:57.610966+0000","last_became_peered":"2023-10-06T05:06:57.610966+0000","last_unstale":"2023-10-06T05:07:01.751968+0000","last_undegraded":"2023-10-06T05:07:01.751968+0000","last_fullsized":"2023-10-06T05:07:01.751968+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:15:48.746279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.768101+0000","last_change":"2023-10-06T05:07:01.764757+0000","last_active":"2023-10-06T05:07:01.768101+0000","last_peered":"2023-10-06T05:07:01.768101+0000","last_clean":"2023-10-06T05:07:01.768101+0000","last_became_active":"2023-10-06T05:07:01.762819+0000","last_became_peered":"2023-10-06T05:07:01.762819+0000","last_unstale":"2023-10-06T05:07:01.768101+0000","last_undegraded":"2023-10-06T05:07:01.768101+0000","last_fullsized":"2023-10-06T05:07:01.768101+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:21:17.237790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3,6],"acting":[2,4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.745471+0000","last_change":"2023-10-06T05:06:56.704818+0000","last_active":"2023-10-06T05:07:01.745471+0000","last_peered":"2023-10-06T05:07:01.745471+0000","last_clean":"2023-10-06T05:07:01.745471+0000","last_became_active":"2023-10-06T05:06:56.704596+0000","last_became_peered":"2023-10-06T05:06:56.704596+0000","last_unstale":"2023-10-06T05:07:01.745471+0000","last_undegraded":"2023-10-06T05:07:01.745471+0000","last_fullsized":"2023-10-06T05:07:01.745471+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:23:02.300780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750614+0000","last_change":"2023-10-06T05:06:57.615733+0000","last_active":"2023-10-06T05:07:01.750614+0000","last_peered":"2023-10-06T05:07:01.750614+0000","last_clean":"2023-10-06T05:07:01.750614+0000","last_became_active":"2023-10-06T05:06:57.615061+0000","last_became_peered":"2023-10-06T05:06:57.615061+0000","last_unstale":"2023-10-06T05:07:01.750614+0000","last_undegraded":"2023-10-06T05:07:01.750614+0000","last_fullsized":"2023-10-06T05:07:01.750614+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:45:23.272110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.768073+0000","last_change":"2023-10-06T05:07:01.763754+0000","last_active":"2023-10-06T05:07:01.768073+0000","last_peered":"2023-10-06T05:07:01.768073+0000","last_clean":"2023-10-06T05:07:01.768073+0000","last_became_active":"2023-10-06T05:07:01.763392+0000","last_became_peered":"2023-10-06T05:07:01.763392+0000","last_unstale":"2023-10-06T05:07:01.768073+0000","last_undegraded":"2023-10-06T05:07:01.768073+0000","last_fullsized":"2023-10-06T05:07:01.768073+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:38:20.917328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5,6],"acting":[2,7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"24'2","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750935+0000","last_change":"2023-10-06T05:06:56.717936+0000","last_active":"2023-10-06T05:07:01.750935+0000","last_peered":"2023-10-06T05:07:01.750935+0000","last_clean":"2023-10-06T05:07:01.750935+0000","last_became_active":"2023-10-06T05:06:56.717683+0000","last_became_peered":"2023-10-06T05:06:56.717683+0000","last_unstale":"2023-10-06T05:07:01.750935+0000","last_undegraded":"2023-10-06T05:07:01.750935+0000","last_fullsized":"2023-10-06T05:07:01.750935+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:23:02.300780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.760808+0000","last_change":"2023-10-06T05:06:57.617318+0000","last_active":"2023-10-06T05:07:01.760808+0000","last_peered":"2023-10-06T05:07:01.760808+0000","last_clean":"2023-10-06T05:07:01.760808+0000","last_became_active":"2023-10-06T05:06:57.617044+0000","last_became_peered":"2023-10-06T05:06:57.617044+0000","last_unstale":"2023-10-06T05:07:01.760808+0000","last_undegraded":"2023-10-06T05:07:01.760808+0000","last_fullsized":"2023-10-06T05:07:01.760808+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:51:15.570020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.772045+0000","last_change":"2023-10-06T05:07:01.762386+0000","last_active":"2023-10-06T05:07:01.772045+0000","last_peered":"2023-10-06T05:07:01.772045+0000","last_clean":"2023-10-06T05:07:01.772045+0000","last_became_active":"2023-10-06T05:07:01.761040+0000","last_became_peered":"2023-10-06T05:07:01.761040+0000","last_unstale":"2023-10-06T05:07:01.772045+0000","last_undegraded":"2023-10-06T05:07:01.772045+0000","last_fullsized":"2023-10-06T05:07:01.772045+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:11:22.468821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0,6],"acting":[1,4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.752055+0000","last_change":"2023-10-06T05:06:56.723332+0000","last_active":"2023-10-06T05:07:01.752055+0000","last_peered":"2023-10-06T05:07:01.752055+0000","last_clean":"2023-10-06T05:07:01.752055+0000","last_became_active":"2023-10-06T05:06:56.723023+0000","last_became_peered":"2023-10-06T05:06:56.723023+0000","last_unstale":"2023-10-06T05:07:01.752055+0000","last_undegraded":"2023-10-06T05:07:01.752055+0000","last_fullsized":"2023-10-06T05:07:01.752055+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:36:39.323759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.761806+0000","last_change":"2023-10-06T05:06:57.609386+0000","last_active":"2023-10-06T05:07:01.761806+0000","last_peered":"2023-10-06T05:07:01.761806+0000","last_clean":"2023-10-06T05:07:01.761806+0000","last_became_active":"2023-10-06T05:06:57.609019+0000","last_became_peered":"2023-10-06T05:06:57.609019+0000","last_unstale":"2023-10-06T05:07:01.761806+0000","last_undegraded":"2023-10-06T05:07:01.761806+0000","last_fullsized":"2023-10-06T05:07:01.761806+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:20:18.722541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.767691+0000","last_change":"2023-10-06T05:07:01.763186+0000","last_active":"2023-10-06T05:07:01.767691+0000","last_peered":"2023-10-06T05:07:01.767691+0000","last_clean":"2023-10-06T05:07:01.767691+0000","last_became_active":"2023-10-06T05:07:01.762259+0000","last_became_peered":"2023-10-06T05:07:01.762259+0000","last_unstale":"2023-10-06T05:07:01.767691+0000","last_undegraded":"2023-10-06T05:07:01.767691+0000","last_fullsized":"2023-10-06T05:07:01.767691+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:46:13.640987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6,2],"acting":[5,4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.761404+0000","last_change":"2023-10-06T05:06:56.722507+0000","last_active":"2023-10-06T05:07:01.761404+0000","last_peered":"2023-10-06T05:07:01.761404+0000","last_clean":"2023-10-06T05:07:01.761404+0000","last_became_active":"2023-10-06T05:06:56.722243+0000","last_became_peered":"2023-10-06T05:06:56.722243+0000","last_unstale":"2023-10-06T05:07:01.761404+0000","last_undegraded":"2023-10-06T05:07:01.761404+0000","last_fullsized":"2023-10-06T05:07:01.761404+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:57:59.037083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.761170+0000","last_change":"2023-10-06T05:06:57.606979+0000","last_active":"2023-10-06T05:07:01.761170+0000","last_peered":"2023-10-06T05:07:01.761170+0000","last_clean":"2023-10-06T05:07:01.761170+0000","last_became_active":"2023-10-06T05:06:57.606633+0000","last_became_peered":"2023-10-06T05:06:57.606633+0000","last_unstale":"2023-10-06T05:07:01.761170+0000","last_undegraded":"2023-10-06T05:07:01.761170+0000","last_fullsized":"2023-10-06T05:07:01.761170+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:21:13.192156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.768126+0000","last_change":"2023-10-06T05:07:01.765104+0000","last_active":"2023-10-06T05:07:01.768126+0000","last_peered":"2023-10-06T05:07:01.768126+0000","last_clean":"2023-10-06T05:07:01.768126+0000","last_became_active":"2023-10-06T05:07:01.764692+0000","last_became_peered":"2023-10-06T05:07:01.764692+0000","last_unstale":"2023-10-06T05:07:01.768126+0000","last_undegraded":"2023-10-06T05:07:01.768126+0000","last_fullsized":"2023-10-06T05:07:01.768126+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:40:47.847563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0,2],"acting":[4,5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.749439+0000","last_change":"2023-10-06T05:06:56.708403+0000","last_active":"2023-10-06T05:07:01.749439+0000","last_peered":"2023-10-06T05:07:01.749439+0000","last_clean":"2023-10-06T05:07:01.749439+0000","last_became_active":"2023-10-06T05:06:56.708144+0000","last_became_peered":"2023-10-06T05:06:56.708144+0000","last_unstale":"2023-10-06T05:07:01.749439+0000","last_undegraded":"2023-10-06T05:07:01.749439+0000","last_fullsized":"2023-10-06T05:07:01.749439+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:17:02.349663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750522+0000","last_change":"2023-10-06T05:06:57.620557+0000","last_active":"2023-10-06T05:07:01.750522+0000","last_peered":"2023-10-06T05:07:01.750522+0000","last_clean":"2023-10-06T05:07:01.750522+0000","last_became_active":"2023-10-06T05:06:57.620286+0000","last_became_peered":"2023-10-06T05:06:57.620286+0000","last_unstale":"2023-10-06T05:07:01.750522+0000","last_undegraded":"2023-10-06T05:07:01.750522+0000","last_fullsized":"2023-10-06T05:07:01.750522+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T16:14:57.399087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.765266+0000","last_change":"2023-10-06T05:07:01.762346+0000","last_active":"2023-10-06T05:07:01.765266+0000","last_peered":"2023-10-06T05:07:01.765266+0000","last_clean":"2023-10-06T05:07:01.765266+0000","last_became_active":"2023-10-06T05:07:01.761861+0000","last_became_peered":"2023-10-06T05:07:01.761861+0000","last_unstale":"2023-10-06T05:07:01.765266+0000","last_undegraded":"2023-10-06T05:07:01.765266+0000","last_fullsized":"2023-10-06T05:07:01.765266+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:41:09.047631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6,5],"acting":[0,1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.748453+0000","last_change":"2023-10-06T05:06:52.627232+0000","last_active":"2023-10-06T05:07:01.748453+0000","last_peered":"2023-10-06T05:07:01.748453+0000","last_clean":"2023-10-06T05:07:01.748453+0000","last_became_active":"2023-10-06T05:06:52.626936+0000","last_became_peered":"2023-10-06T05:06:52.626936+0000","last_unstale":"2023-10-06T05:07:01.748453+0000","last_undegraded":"2023-10-06T05:07:01.748453+0000","last_fullsized":"2023-10-06T05:07:01.748453+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:20:13.925170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.744621+0000","last_change":"2023-10-06T05:06:56.712027+0000","last_active":"2023-10-06T05:07:01.744621+0000","last_peered":"2023-10-06T05:07:01.744621+0000","last_clean":"2023-10-06T05:07:01.744621+0000","last_became_active":"2023-10-06T05:06:56.711779+0000","last_became_peered":"2023-10-06T05:06:56.711779+0000","last_unstale":"2023-10-06T05:07:01.744621+0000","last_undegraded":"2023-10-06T05:07:01.744621+0000","last_fullsized":"2023-10-06T05:07:01.744621+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:49:27.022333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.747306+0000","last_change":"2023-10-06T05:06:57.613847+0000","last_active":"2023-10-06T05:07:01.747306+0000","last_peered":"2023-10-06T05:07:01.747306+0000","last_clean":"2023-10-06T05:07:01.747306+0000","last_became_active":"2023-10-06T05:06:57.613560+0000","last_became_peered":"2023-10-06T05:06:57.613560+0000","last_unstale":"2023-10-06T05:07:01.747306+0000","last_undegraded":"2023-10-06T05:07:01.747306+0000","last_fullsized":"2023-10-06T05:07:01.747306+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:50:36.787873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.766219+0000","last_change":"2023-10-06T05:07:01.764522+0000","last_active":"2023-10-06T05:07:01.766219+0000","last_peered":"2023-10-06T05:07:01.766219+0000","last_clean":"2023-10-06T05:07:01.766219+0000","last_became_active":"2023-10-06T05:07:01.764199+0000","last_became_peered":"2023-10-06T05:07:01.764199+0000","last_unstale":"2023-10-06T05:07:01.766219+0000","last_undegraded":"2023-10-06T05:07:01.766219+0000","last_fullsized":"2023-10-06T05:07:01.766219+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:13:19.947679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,6],"acting":[0,1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.744573+0000","last_change":"2023-10-06T05:06:52.587795+0000","last_active":"2023-10-06T05:07:01.744573+0000","last_peered":"2023-10-06T05:07:01.744573+0000","last_clean":"2023-10-06T05:07:01.744573+0000","last_became_active":"2023-10-06T05:06:52.587407+0000","last_became_peered":"2023-10-06T05:06:52.587407+0000","last_unstale":"2023-10-06T05:07:01.744573+0000","last_undegraded":"2023-10-06T05:07:01.744573+0000","last_fullsized":"2023-10-06T05:07:01.744573+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:36:31.526532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.761857+0000","last_change":"2023-10-06T05:06:56.721976+0000","last_active":"2023-10-06T05:07:01.761857+0000","last_peered":"2023-10-06T05:07:01.761857+0000","last_clean":"2023-10-06T05:07:01.761857+0000","last_became_active":"2023-10-06T05:06:56.721679+0000","last_became_peered":"2023-10-06T05:06:56.721679+0000","last_unstale":"2023-10-06T05:07:01.761857+0000","last_undegraded":"2023-10-06T05:07:01.761857+0000","last_fullsized":"2023-10-06T05:07:01.761857+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:38:50.877673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.748383+0000","last_change":"2023-10-06T05:06:57.612621+0000","last_active":"2023-10-06T05:07:01.748383+0000","last_peered":"2023-10-06T05:07:01.748383+0000","last_clean":"2023-10-06T05:07:01.748383+0000","last_became_active":"2023-10-06T05:06:57.612305+0000","last_became_peered":"2023-10-06T05:06:57.612305+0000","last_unstale":"2023-10-06T05:07:01.748383+0000","last_undegraded":"2023-10-06T05:07:01.748383+0000","last_fullsized":"2023-10-06T05:07:01.748383+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:35:18.828024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.768737+0000","last_change":"2023-10-06T05:07:01.764514+0000","last_active":"2023-10-06T05:07:01.768737+0000","last_peered":"2023-10-06T05:07:01.768737+0000","last_clean":"2023-10-06T05:07:01.768737+0000","last_became_active":"2023-10-06T05:07:01.764176+0000","last_became_peered":"2023-10-06T05:07:01.764176+0000","last_unstale":"2023-10-06T05:07:01.768737+0000","last_undegraded":"2023-10-06T05:07:01.768737+0000","last_fullsized":"2023-10-06T05:07:01.768737+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:20:08.719247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1,7],"acting":[2,5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.746279+0000","last_change":"2023-10-06T05:06:52.675238+0000","last_active":"2023-10-06T05:07:01.746279+0000","last_peered":"2023-10-06T05:07:01.746279+0000","last_clean":"2023-10-06T05:07:01.746279+0000","last_became_active":"2023-10-06T05:06:52.674952+0000","last_became_peered":"2023-10-06T05:06:52.674952+0000","last_unstale":"2023-10-06T05:07:01.746279+0000","last_undegraded":"2023-10-06T05:07:01.746279+0000","last_fullsized":"2023-10-06T05:07:01.746279+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:27:51.141184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.746657+0000","last_change":"2023-10-06T05:06:56.711280+0000","last_active":"2023-10-06T05:07:01.746657+0000","last_peered":"2023-10-06T05:07:01.746657+0000","last_clean":"2023-10-06T05:07:01.746657+0000","last_became_active":"2023-10-06T05:06:56.711033+0000","last_became_peered":"2023-10-06T05:06:56.711033+0000","last_unstale":"2023-10-06T05:07:01.746657+0000","last_undegraded":"2023-10-06T05:07:01.746657+0000","last_fullsized":"2023-10-06T05:07:01.746657+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:38:12.815936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750448+0000","last_change":"2023-10-06T05:06:57.614460+0000","last_active":"2023-10-06T05:07:01.750448+0000","last_peered":"2023-10-06T05:07:01.750448+0000","last_clean":"2023-10-06T05:07:01.750448+0000","last_became_active":"2023-10-06T05:06:57.614245+0000","last_became_peered":"2023-10-06T05:06:57.614245+0000","last_unstale":"2023-10-06T05:07:01.750448+0000","last_undegraded":"2023-10-06T05:07:01.750448+0000","last_fullsized":"2023-10-06T05:07:01.750448+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:51:31.873476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.775792+0000","last_change":"2023-10-06T05:07:01.766672+0000","last_active":"2023-10-06T05:07:01.775792+0000","last_peered":"2023-10-06T05:07:01.775792+0000","last_clean":"2023-10-06T05:07:01.775792+0000","last_became_active":"2023-10-06T05:07:01.766118+0000","last_became_peered":"2023-10-06T05:07:01.766118+0000","last_unstale":"2023-10-06T05:07:01.775792+0000","last_undegraded":"2023-10-06T05:07:01.775792+0000","last_fullsized":"2023-10-06T05:07:01.775792+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:55:02.566598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5,3],"acting":[7,2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.745050+0000","last_change":"2023-10-06T05:06:52.587778+0000","last_active":"2023-10-06T05:07:01.745050+0000","last_peered":"2023-10-06T05:07:01.745050+0000","last_clean":"2023-10-06T05:07:01.745050+0000","last_became_active":"2023-10-06T05:06:52.587388+0000","last_became_peered":"2023-10-06T05:06:52.587388+0000","last_unstale":"2023-10-06T05:07:01.745050+0000","last_undegraded":"2023-10-06T05:07:01.745050+0000","last_fullsized":"2023-10-06T05:07:01.745050+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:45:52.502298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":169,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.748339+0000","last_change":"2023-10-06T05:06:46.307169+0000","last_active":"2023-10-06T05:07:01.748339+0000","last_peered":"2023-10-06T05:07:01.748339+0000","last_clean":"2023-10-06T05:07:01.748339+0000","last_became_active":"2023-10-06T05:06:46.306107+0000","last_became_peered":"2023-10-06T05:06:46.306107+0000","last_unstale":"2023-10-06T05:07:01.748339+0000","last_undegraded":"2023-10-06T05:07:01.748339+0000","last_fullsized":"2023-10-06T05:07:01.748339+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:45.018006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:45.018006+0000","last_clean_scrub_stamp":"2023-10-06T05:06:45.018006+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:38:17.064131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750303+0000","last_change":"2023-10-06T05:06:56.706870+0000","last_active":"2023-10-06T05:07:01.750303+0000","last_peered":"2023-10-06T05:07:01.750303+0000","last_clean":"2023-10-06T05:07:01.750303+0000","last_became_active":"2023-10-06T05:06:56.706636+0000","last_became_peered":"2023-10-06T05:06:56.706636+0000","last_unstale":"2023-10-06T05:07:01.750303+0000","last_undegraded":"2023-10-06T05:07:01.750303+0000","last_fullsized":"2023-10-06T05:07:01.750303+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:00:29.033929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.753036+0000","last_change":"2023-10-06T05:06:57.604630+0000","last_active":"2023-10-06T05:07:01.753036+0000","last_peered":"2023-10-06T05:07:01.753036+0000","last_clean":"2023-10-06T05:07:01.753036+0000","last_became_active":"2023-10-06T05:06:57.604341+0000","last_became_peered":"2023-10-06T05:06:57.604341+0000","last_unstale":"2023-10-06T05:07:01.753036+0000","last_undegraded":"2023-10-06T05:07:01.753036+0000","last_fullsized":"2023-10-06T05:07:01.753036+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:57:12.615864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.767715+0000","last_change":"2023-10-06T05:07:01.763448+0000","last_active":"2023-10-06T05:07:01.767715+0000","last_peered":"2023-10-06T05:07:01.767715+0000","last_clean":"2023-10-06T05:07:01.767715+0000","last_became_active":"2023-10-06T05:07:01.762832+0000","last_became_peered":"2023-10-06T05:07:01.762832+0000","last_unstale":"2023-10-06T05:07:01.767715+0000","last_undegraded":"2023-10-06T05:07:01.767715+0000","last_fullsized":"2023-10-06T05:07:01.767715+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:49:53.682865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0,4],"acting":[5,1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.748590+0000","last_change":"2023-10-06T05:06:52.588475+0000","last_active":"2023-10-06T05:07:01.748590+0000","last_peered":"2023-10-06T05:07:01.748590+0000","last_clean":"2023-10-06T05:07:01.748590+0000","last_became_active":"2023-10-06T05:06:52.588134+0000","last_became_peered":"2023-10-06T05:06:52.588134+0000","last_unstale":"2023-10-06T05:07:01.748590+0000","last_undegraded":"2023-10-06T05:07:01.748590+0000","last_fullsized":"2023-10-06T05:07:01.748590+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:35:05.269778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.749699+0000","last_change":"2023-10-06T05:06:56.723240+0000","last_active":"2023-10-06T05:07:01.749699+0000","last_peered":"2023-10-06T05:07:01.749699+0000","last_clean":"2023-10-06T05:07:01.749699+0000","last_became_active":"2023-10-06T05:06:56.722965+0000","last_became_peered":"2023-10-06T05:06:56.722965+0000","last_unstale":"2023-10-06T05:07:01.749699+0000","last_undegraded":"2023-10-06T05:07:01.749699+0000","last_fullsized":"2023-10-06T05:07:01.749699+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:33:23.298771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.749910+0000","last_change":"2023-10-06T05:06:57.616494+0000","last_active":"2023-10-06T05:07:01.749910+0000","last_peered":"2023-10-06T05:07:01.749910+0000","last_clean":"2023-10-06T05:07:01.749910+0000","last_became_active":"2023-10-06T05:06:57.616234+0000","last_became_peered":"2023-10-06T05:06:57.616234+0000","last_unstale":"2023-10-06T05:07:01.749910+0000","last_undegraded":"2023-10-06T05:07:01.749910+0000","last_fullsized":"2023-10-06T05:07:01.749910+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:34:55.987076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.773039+0000","last_change":"2023-10-06T05:07:01.768822+0000","last_active":"2023-10-06T05:07:01.773039+0000","last_peered":"2023-10-06T05:07:01.773039+0000","last_clean":"2023-10-06T05:07:01.773039+0000","last_became_active":"2023-10-06T05:07:01.768425+0000","last_became_peered":"2023-10-06T05:07:01.768425+0000","last_unstale":"2023-10-06T05:07:01.773039+0000","last_undegraded":"2023-10-06T05:07:01.773039+0000","last_fullsized":"2023-10-06T05:07:01.773039+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:52:18.818435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5,2],"acting":[6,0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.754250+0000","last_change":"2023-10-06T05:06:52.627289+0000","last_active":"2023-10-06T05:07:01.754250+0000","last_peered":"2023-10-06T05:07:01.754250+0000","last_clean":"2023-10-06T05:07:01.754250+0000","last_became_active":"2023-10-06T05:06:52.626937+0000","last_became_peered":"2023-10-06T05:06:52.626937+0000","last_unstale":"2023-10-06T05:07:01.754250+0000","last_undegraded":"2023-10-06T05:07:01.754250+0000","last_fullsized":"2023-10-06T05:07:01.754250+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:46:59.070674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.748611+0000","last_change":"2023-10-06T05:06:56.720424+0000","last_active":"2023-10-06T05:07:01.748611+0000","last_peered":"2023-10-06T05:07:01.748611+0000","last_clean":"2023-10-06T05:07:01.748611+0000","last_became_active":"2023-10-06T05:06:56.720133+0000","last_became_peered":"2023-10-06T05:06:56.720133+0000","last_unstale":"2023-10-06T05:07:01.748611+0000","last_undegraded":"2023-10-06T05:07:01.748611+0000","last_fullsized":"2023-10-06T05:07:01.748611+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T16:11:31.404755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750404+0000","last_change":"2023-10-06T05:06:57.619048+0000","last_active":"2023-10-06T05:07:01.750404+0000","last_peered":"2023-10-06T05:07:01.750404+0000","last_clean":"2023-10-06T05:07:01.750404+0000","last_became_active":"2023-10-06T05:06:57.618823+0000","last_became_peered":"2023-10-06T05:06:57.618823+0000","last_unstale":"2023-10-06T05:07:01.750404+0000","last_undegraded":"2023-10-06T05:07:01.750404+0000","last_fullsized":"2023-10-06T05:07:01.750404+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:25:56.905027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.768167+0000","last_change":"2023-10-06T05:07:01.765125+0000","last_active":"2023-10-06T05:07:01.768167+0000","last_peered":"2023-10-06T05:07:01.768167+0000","last_clean":"2023-10-06T05:07:01.768167+0000","last_became_active":"2023-10-06T05:07:01.764733+0000","last_became_peered":"2023-10-06T05:07:01.764733+0000","last_unstale":"2023-10-06T05:07:01.768167+0000","last_undegraded":"2023-10-06T05:07:01.768167+0000","last_fullsized":"2023-10-06T05:07:01.768167+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T16:29:47.226299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7,0],"acting":[4,3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"24'2","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750393+0000","last_change":"2023-10-06T05:06:56.710924+0000","last_active":"2023-10-06T05:07:01.750393+0000","last_peered":"2023-10-06T05:07:01.750393+0000","last_clean":"2023-10-06T05:07:01.750393+0000","last_became_active":"2023-10-06T05:06:56.710518+0000","last_became_peered":"2023-10-06T05:06:56.710518+0000","last_unstale":"2023-10-06T05:07:01.750393+0000","last_undegraded":"2023-10-06T05:07:01.750393+0000","last_fullsized":"2023-10-06T05:07:01.750393+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:49:27.022333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.748508+0000","last_change":"2023-10-06T05:06:52.588853+0000","last_active":"2023-10-06T05:07:01.748508+0000","last_peered":"2023-10-06T05:07:01.748508+0000","last_clean":"2023-10-06T05:07:01.748508+0000","last_became_active":"2023-10-06T05:06:52.588561+0000","last_became_peered":"2023-10-06T05:06:52.588561+0000","last_unstale":"2023-10-06T05:07:01.748508+0000","last_undegraded":"2023-10-06T05:07:01.748508+0000","last_fullsized":"2023-10-06T05:07:01.748508+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:06:28.729493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.749357+0000","last_change":"2023-10-06T05:06:57.619145+0000","last_active":"2023-10-06T05:07:01.749357+0000","last_peered":"2023-10-06T05:07:01.749357+0000","last_clean":"2023-10-06T05:07:01.749357+0000","last_became_active":"2023-10-06T05:06:57.618666+0000","last_became_peered":"2023-10-06T05:06:57.618666+0000","last_unstale":"2023-10-06T05:07:01.749357+0000","last_undegraded":"2023-10-06T05:07:01.749357+0000","last_fullsized":"2023-10-06T05:07:01.749357+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:29:19.951056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.765168+0000","last_change":"2023-10-06T05:07:01.762410+0000","last_active":"2023-10-06T05:07:01.765168+0000","last_peered":"2023-10-06T05:07:01.765168+0000","last_clean":"2023-10-06T05:07:01.765168+0000","last_became_active":"2023-10-06T05:07:01.762060+0000","last_became_peered":"2023-10-06T05:07:01.762060+0000","last_unstale":"2023-10-06T05:07:01.765168+0000","last_undegraded":"2023-10-06T05:07:01.765168+0000","last_fullsized":"2023-10-06T05:07:01.765168+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:49:53.682865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,0],"acting":[3,1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.751181+0000","last_change":"2023-10-06T05:06:52.589519+0000","last_active":"2023-10-06T05:07:01.751181+0000","last_peered":"2023-10-06T05:07:01.751181+0000","last_clean":"2023-10-06T05:07:01.751181+0000","last_became_active":"2023-10-06T05:06:52.589246+0000","last_became_peered":"2023-10-06T05:06:52.589246+0000","last_unstale":"2023-10-06T05:07:01.751181+0000","last_undegraded":"2023-10-06T05:07:01.751181+0000","last_fullsized":"2023-10-06T05:07:01.751181+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:54:23.352764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"28'1","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:02.685849+0000","last_change":"2023-10-06T05:06:56.714058+0000","last_active":"2023-10-06T05:07:02.685849+0000","last_peered":"2023-10-06T05:07:02.685849+0000","last_clean":"2023-10-06T05:07:02.685849+0000","last_became_active":"2023-10-06T05:06:56.713772+0000","last_became_peered":"2023-10-06T05:06:56.713772+0000","last_unstale":"2023-10-06T05:07:02.685849+0000","last_undegraded":"2023-10-06T05:07:02.685849+0000","last_fullsized":"2023-10-06T05:07:02.685849+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:53:47.748350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.761483+0000","last_change":"2023-10-06T05:06:57.608832+0000","last_active":"2023-10-06T05:07:01.761483+0000","last_peered":"2023-10-06T05:07:01.761483+0000","last_clean":"2023-10-06T05:07:01.761483+0000","last_became_active":"2023-10-06T05:06:57.608499+0000","last_became_peered":"2023-10-06T05:06:57.608499+0000","last_unstale":"2023-10-06T05:07:01.761483+0000","last_undegraded":"2023-10-06T05:07:01.761483+0000","last_fullsized":"2023-10-06T05:07:01.761483+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:29:19.951056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.772123+0000","last_change":"2023-10-06T05:07:01.765615+0000","last_active":"2023-10-06T05:07:01.772123+0000","last_peered":"2023-10-06T05:07:01.772123+0000","last_clean":"2023-10-06T05:07:01.772123+0000","last_became_active":"2023-10-06T05:07:01.762552+0000","last_became_peered":"2023-10-06T05:07:01.762552+0000","last_unstale":"2023-10-06T05:07:01.772123+0000","last_undegraded":"2023-10-06T05:07:01.772123+0000","last_fullsized":"2023-10-06T05:07:01.772123+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:30:14.995001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,0],"acting":[1,3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"24'1","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750483+0000","last_change":"2023-10-06T05:06:56.723224+0000","last_active":"2023-10-06T05:07:01.750483+0000","last_peered":"2023-10-06T05:07:01.750483+0000","last_clean":"2023-10-06T05:07:01.750483+0000","last_became_active":"2023-10-06T05:06:56.722971+0000","last_became_peered":"2023-10-06T05:06:56.722971+0000","last_unstale":"2023-10-06T05:07:01.750483+0000","last_undegraded":"2023-10-06T05:07:01.750483+0000","last_fullsized":"2023-10-06T05:07:01.750483+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:10:32.670079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.746067+0000","last_change":"2023-10-06T05:06:57.610613+0000","last_active":"2023-10-06T05:07:01.746067+0000","last_peered":"2023-10-06T05:07:01.746067+0000","last_clean":"2023-10-06T05:07:01.746067+0000","last_became_active":"2023-10-06T05:06:57.610363+0000","last_became_peered":"2023-10-06T05:06:57.610363+0000","last_unstale":"2023-10-06T05:07:01.746067+0000","last_undegraded":"2023-10-06T05:07:01.746067+0000","last_fullsized":"2023-10-06T05:07:01.746067+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:55:09.125970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.775891+0000","last_change":"2023-10-06T05:07:01.771319+0000","last_active":"2023-10-06T05:07:01.775891+0000","last_peered":"2023-10-06T05:07:01.775891+0000","last_clean":"2023-10-06T05:07:01.775891+0000","last_became_active":"2023-10-06T05:07:01.770953+0000","last_became_peered":"2023-10-06T05:07:01.770953+0000","last_unstale":"2023-10-06T05:07:01.775891+0000","last_undegraded":"2023-10-06T05:07:01.775891+0000","last_fullsized":"2023-10-06T05:07:01.775891+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:13:19.947679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1,6],"acting":[7,4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"24'2","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.745566+0000","last_change":"2023-10-06T05:06:56.723033+0000","last_active":"2023-10-06T05:07:01.745566+0000","last_peered":"2023-10-06T05:07:01.745566+0000","last_clean":"2023-10-06T05:07:01.745566+0000","last_became_active":"2023-10-06T05:06:56.722747+0000","last_became_peered":"2023-10-06T05:06:56.722747+0000","last_unstale":"2023-10-06T05:07:01.745566+0000","last_undegraded":"2023-10-06T05:07:01.745566+0000","last_fullsized":"2023-10-06T05:07:01.745566+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:17:02.349663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.761541+0000","last_change":"2023-10-06T05:06:57.621769+0000","last_active":"2023-10-06T05:07:01.761541+0000","last_peered":"2023-10-06T05:07:01.761541+0000","last_clean":"2023-10-06T05:07:01.761541+0000","last_became_active":"2023-10-06T05:06:57.621489+0000","last_became_peered":"2023-10-06T05:06:57.621489+0000","last_unstale":"2023-10-06T05:07:01.761541+0000","last_undegraded":"2023-10-06T05:07:01.761541+0000","last_fullsized":"2023-10-06T05:07:01.761541+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:59:39.994965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.773062+0000","last_change":"2023-10-06T05:07:01.769584+0000","last_active":"2023-10-06T05:07:01.773062+0000","last_peered":"2023-10-06T05:07:01.773062+0000","last_clean":"2023-10-06T05:07:01.773062+0000","last_became_active":"2023-10-06T05:07:01.769168+0000","last_became_peered":"2023-10-06T05:07:01.769168+0000","last_unstale":"2023-10-06T05:07:01.773062+0000","last_undegraded":"2023-10-06T05:07:01.773062+0000","last_fullsized":"2023-10-06T05:07:01.773062+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:25:41.281713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7,4],"acting":[5,0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"24'2","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.744323+0000","last_change":"2023-10-06T05:06:56.721940+0000","last_active":"2023-10-06T05:07:01.744323+0000","last_peered":"2023-10-06T05:07:01.744323+0000","last_clean":"2023-10-06T05:07:01.744323+0000","last_became_active":"2023-10-06T05:06:56.721668+0000","last_became_peered":"2023-10-06T05:06:56.721668+0000","last_unstale":"2023-10-06T05:07:01.744323+0000","last_undegraded":"2023-10-06T05:07:01.744323+0000","last_fullsized":"2023-10-06T05:07:01.744323+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:55:08.124129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.760815+0000","last_change":"2023-10-06T05:06:57.614557+0000","last_active":"2023-10-06T05:07:01.760815+0000","last_peered":"2023-10-06T05:07:01.760815+0000","last_clean":"2023-10-06T05:07:01.760815+0000","last_became_active":"2023-10-06T05:06:57.614113+0000","last_became_peered":"2023-10-06T05:06:57.614113+0000","last_unstale":"2023-10-06T05:07:01.760815+0000","last_undegraded":"2023-10-06T05:07:01.760815+0000","last_fullsized":"2023-10-06T05:07:01.760815+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:57:12.615864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.772225+0000","last_change":"2023-10-06T05:07:01.766934+0000","last_active":"2023-10-06T05:07:01.772225+0000","last_peered":"2023-10-06T05:07:01.772225+0000","last_clean":"2023-10-06T05:07:01.772225+0000","last_became_active":"2023-10-06T05:07:01.766522+0000","last_became_peered":"2023-10-06T05:07:01.766522+0000","last_unstale":"2023-10-06T05:07:01.772225+0000","last_undegraded":"2023-10-06T05:07:01.772225+0000","last_fullsized":"2023-10-06T05:07:01.772225+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:15:30.724218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,2],"acting":[4,6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"24'1","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.753608+0000","last_change":"2023-10-06T05:06:56.717832+0000","last_active":"2023-10-06T05:07:01.753608+0000","last_peered":"2023-10-06T05:07:01.753608+0000","last_clean":"2023-10-06T05:07:01.753608+0000","last_became_active":"2023-10-06T05:06:56.717568+0000","last_became_peered":"2023-10-06T05:06:56.717568+0000","last_unstale":"2023-10-06T05:07:01.753608+0000","last_undegraded":"2023-10-06T05:07:01.753608+0000","last_fullsized":"2023-10-06T05:07:01.753608+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:30:18.507322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750496+0000","last_change":"2023-10-06T05:06:57.620391+0000","last_active":"2023-10-06T05:07:01.750496+0000","last_peered":"2023-10-06T05:07:01.750496+0000","last_clean":"2023-10-06T05:07:01.750496+0000","last_became_active":"2023-10-06T05:06:57.619822+0000","last_became_peered":"2023-10-06T05:06:57.619822+0000","last_unstale":"2023-10-06T05:07:01.750496+0000","last_undegraded":"2023-10-06T05:07:01.750496+0000","last_fullsized":"2023-10-06T05:07:01.750496+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:59:39.994965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":11,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.772089+0000","last_change":"2023-10-06T05:07:01.762139+0000","last_active":"2023-10-06T05:07:01.772089+0000","last_peered":"2023-10-06T05:07:01.772089+0000","last_clean":"2023-10-06T05:07:01.772089+0000","last_became_active":"2023-10-06T05:07:01.761672+0000","last_became_peered":"2023-10-06T05:07:01.761672+0000","last_unstale":"2023-10-06T05:07:01.772089+0000","last_undegraded":"2023-10-06T05:07:01.772089+0000","last_fullsized":"2023-10-06T05:07:01.772089+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:00.735710+0000","last_clean_scrub_stamp":"2023-10-06T05:07:00.735710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:07:51.743165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7,2],"acting":[1,5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"24'4","reported_seq":18,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.751360+0000","last_change":"2023-10-06T05:06:56.714871+0000","last_active":"2023-10-06T05:07:01.751360+0000","last_peered":"2023-10-06T05:07:01.751360+0000","last_clean":"2023-10-06T05:07:01.751360+0000","last_became_active":"2023-10-06T05:06:56.714616+0000","last_became_peered":"2023-10-06T05:06:56.714616+0000","last_unstale":"2023-10-06T05:07:01.751360+0000","last_undegraded":"2023-10-06T05:07:01.751360+0000","last_fullsized":"2023-10-06T05:07:01.751360+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:55.688234+0000","last_clean_scrub_stamp":"2023-10-06T05:06:55.688234+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:35:42.402195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.758731+0000","last_change":"2023-10-06T05:06:57.605198+0000","last_active":"2023-10-06T05:07:01.758731+0000","last_peered":"2023-10-06T05:07:01.758731+0000","last_clean":"2023-10-06T05:07:01.758731+0000","last_became_active":"2023-10-06T05:06:57.604924+0000","last_became_peered":"2023-10-06T05:06:57.604924+0000","last_unstale":"2023-10-06T05:07:01.758731+0000","last_undegraded":"2023-10-06T05:07:01.758731+0000","last_fullsized":"2023-10-06T05:07:01.758731+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:01:11.983432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750721+0000","last_change":"2023-10-06T05:06:57.608286+0000","last_active":"2023-10-06T05:07:01.750721+0000","last_peered":"2023-10-06T05:07:01.750721+0000","last_clean":"2023-10-06T05:07:01.750721+0000","last_became_active":"2023-10-06T05:06:57.607986+0000","last_became_peered":"2023-10-06T05:06:57.607986+0000","last_unstale":"2023-10-06T05:07:01.750721+0000","last_undegraded":"2023-10-06T05:07:01.750721+0000","last_fullsized":"2023-10-06T05:07:01.750721+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:51:31.370159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750626+0000","last_change":"2023-10-06T05:06:57.623937+0000","last_active":"2023-10-06T05:07:01.750626+0000","last_peered":"2023-10-06T05:07:01.750626+0000","last_clean":"2023-10-06T05:07:01.750626+0000","last_became_active":"2023-10-06T05:06:57.623705+0000","last_became_peered":"2023-10-06T05:06:57.623705+0000","last_unstale":"2023-10-06T05:07:01.750626+0000","last_undegraded":"2023-10-06T05:07:01.750626+0000","last_fullsized":"2023-10-06T05:07:01.750626+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:20:04.673613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.761577+0000","last_change":"2023-10-06T05:06:57.611066+0000","last_active":"2023-10-06T05:07:01.761577+0000","last_peered":"2023-10-06T05:07:01.761577+0000","last_clean":"2023-10-06T05:07:01.761577+0000","last_became_active":"2023-10-06T05:06:57.610827+0000","last_became_peered":"2023-10-06T05:06:57.610827+0000","last_unstale":"2023-10-06T05:07:01.761577+0000","last_undegraded":"2023-10-06T05:07:01.761577+0000","last_fullsized":"2023-10-06T05:07:01.761577+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:43:18.121629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.750840+0000","last_change":"2023-10-06T05:06:57.615191+0000","last_active":"2023-10-06T05:07:01.750840+0000","last_peered":"2023-10-06T05:07:01.750840+0000","last_clean":"2023-10-06T05:07:01.750840+0000","last_became_active":"2023-10-06T05:06:57.614632+0000","last_became_peered":"2023-10-06T05:06:57.614632+0000","last_unstale":"2023-10-06T05:07:01.750840+0000","last_undegraded":"2023-10-06T05:07:01.750840+0000","last_fullsized":"2023-10-06T05:07:01.750840+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:32:33.731449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-06T05:07:01.748531+0000","last_change":"2023-10-06T05:06:57.616743+0000","last_active":"2023-10-06T05:07:01.748531+0000","last_peered":"2023-10-06T05:07:01.748531+0000","last_clean":"2023-10-06T05:07:01.748531+0000","last_became_active":"2023-10-06T05:06:57.616487+0000","last_became_peered":"2023-10-06T05:06:57.616487+0000","last_unstale":"2023-10-06T05:07:01.748531+0000","last_undegraded":"2023-10-06T05:07:01.748531+0000","last_fullsized":"2023-10-06T05:07:01.748531+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:56.690075+0000","last_clean_scrub_stamp":"2023-10-06T05:06:56.690075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:15:55.618648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":4484,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":8968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509450,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297588,"kb_used_data":300,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074252,"statfs":{"total":96636764160,"available":96332034048,"internally_reserved":0,"allocated":307200,"data_stored":126099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55700000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66600000000000004}]}]},{"osd":5,"up_from":15,"seq":64424509450,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297596,"kb_used_data":308,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074244,"statfs":{"total":96636764160,"available":96332025856,"internally_reserved":0,"allocated":315392,"data_stored":126723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42099999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.372}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39400000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48699999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51400000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61699999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]}]},{"osd":0,"up_from":14,"seq":60129542154,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298172,"kb_used_data":884,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073668,"statfs":{"total":96636764160,"available":96331436032,"internally_reserved":0,"allocated":905216,"data_stored":717021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2809999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.1529999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.327}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0329999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0549999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95899999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.079}]}]},{"osd":7,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298196,"kb_used_data":908,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073644,"statfs":{"total":96636764160,"available":96331411456,"internally_reserved":0,"allocated":929792,"data_stored":720397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33500000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51200000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39900000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41299999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.436}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45000000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42399999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542153,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297588,"kb_used_data":300,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074252,"statfs":{"total":96636764160,"available":96332034048,"internally_reserved":0,"allocated":307200,"data_stored":126099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32700000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36499999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40200000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64900000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60299999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66200000000000003}]}]},{"osd":2,"up_from":15,"seq":64424509449,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297588,"kb_used_data":300,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074252,"statfs":{"total":96636764160,"available":96332034048,"internally_reserved":0,"allocated":307200,"data_stored":126099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29699999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.374}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34499999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55200000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56000000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]}]},{"osd":3,"up_from":15,"seq":64424509449,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297592,"kb_used_data":304,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074248,"statfs":{"total":96636764160,"available":96332029952,"internally_reserved":0,"allocated":311296,"data_stored":126189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30199999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46400000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]}]},{"osd":4,"up_from":15,"seq":64424509450,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297608,"kb_used_data":320,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074232,"statfs":{"total":96636764160,"available":96332013568,"internally_reserved":0,"allocated":327680,"data_stored":129869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57499999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40899999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35299999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48099999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":3770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":3930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-06T05:07:16.779 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-06T05:07:16.779 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-10-06T05:07:16.779 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-10-06T05:07:16.779 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-06T05:07:16.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.923+0000 7f3978c39700 1 Processor -- start 2023-10-06T05:07:16.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.923+0000 7f3978c39700 1 -- start start 2023-10-06T05:07:16.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.924+0000 7f3978c39700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f3974159560 0x7f39740aa530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:16.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.924+0000 7f3978c39700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39740b0c60 con 0x7f39740b49a0 2023-10-06T05:07:16.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.924+0000 7f3978c39700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39740b0f40 con 0x7f39740b0760 2023-10-06T05:07:16.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.924+0000 7f3978c39700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39740b1220 con 0x7f3974159560 2023-10-06T05:07:16.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.924+0000 7f3969d9b700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f39740b49a0 0x7f39740aaa70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:41258/0 (socket says 172.21.15.98:41258) 2023-10-06T05:07:16.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.924+0000 7f3969d9b700 1 -- 172.21.15.98:0/1942513377 learned_addr learned my addr 172.21.15.98:0/1942513377 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:07:16.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.924+0000 7f397259c700 1 --2- 172.21.15.98:0/1942513377 >> v2:172.21.15.174:3300/0 conn(0x7f3974159560 0x7f39740aa530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:16.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.924+0000 7f3971d9b700 1 -- 172.21.15.98:0/1942513377 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 848811893 0 0) 0x7f39740b0c60 con 0x7f39740b49a0 2023-10-06T05:07:16.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.924+0000 7f3971d9b700 1 -- 172.21.15.98:0/1942513377 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3960003570 con 0x7f39740b49a0 2023-10-06T05:07:16.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.925+0000 7f3971d9b700 1 -- 172.21.15.98:0/1942513377 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3153525006 0 0) 0x7f39740b0f40 con 0x7f39740b0760 2023-10-06T05:07:16.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.925+0000 7f3971d9b700 1 -- 172.21.15.98:0/1942513377 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39740b0c60 con 0x7f39740b0760 2023-10-06T05:07:16.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.925+0000 7f3971d9b700 1 -- 172.21.15.98:0/1942513377 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 516713587 0 0) 0x7f3960003570 con 0x7f39740b49a0 2023-10-06T05:07:16.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.925+0000 7f3971d9b700 1 -- 172.21.15.98:0/1942513377 >> v2:172.21.15.174:3300/0 conn(0x7f3974159560 msgr2=0x7f39740aa530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:16.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.925+0000 7f3971d9b700 1 --2- 172.21.15.98:0/1942513377 >> v2:172.21.15.174:3300/0 conn(0x7f3974159560 0x7f39740aa530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:16.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.925+0000 7f3971d9b700 1 -- 172.21.15.98:0/1942513377 >> v1:172.21.15.174:6789/0 conn(0x7f39740b0760 legacy=0x7f39740b1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:16.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.925+0000 7f3971d9b700 1 -- 172.21.15.98:0/1942513377 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39740b1500 con 0x7f39740b49a0 2023-10-06T05:07:16.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.925+0000 7f3971d9b700 1 -- 172.21.15.98:0/1942513377 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1543991876 0 0) 0x7f395c001470 con 0x7f39740b49a0 2023-10-06T05:07:16.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.925+0000 7f397259c700 1 --2- 172.21.15.98:0/1942513377 >> v2:172.21.15.174:3300/0 conn(0x7f3974159560 0x7f39740aa530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T05:07:16.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.925+0000 7f3971d9b700 1 -- 172.21.15.98:0/1942513377 >> v1:172.21.15.98:6789/0 conn(0x7f39740b49a0 legacy=0x7f39740aaa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:16.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.925+0000 7f3971d9b700 1 --2- 172.21.15.98:0/1942513377 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3960003c40 0x7f3960004030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:16.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.925+0000 7f3971d9b700 1 --2- 172.21.15.98:0/1942513377 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3960004810 0x7f3960004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:16.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.925+0000 7f3971d9b700 1 --2- 172.21.15.98:0/1942513377 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3960005210 0x7f396000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:16.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.925+0000 7f3971d9b700 1 -- 172.21.15.98:0/1942513377 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39740b1220 con 0x7f3960004810 2023-10-06T05:07:16.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.925+0000 7f3971d9b700 1 -- 172.21.15.98:0/1942513377 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f396000c130 con 0x7f3960005210 2023-10-06T05:07:16.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.925+0000 7f3971d9b700 1 -- 172.21.15.98:0/1942513377 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f396000c2c0 con 0x7f3960003c40 2023-10-06T05:07:16.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.925+0000 7f3969d9b700 1 --2- 172.21.15.98:0/1942513377 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3960004810 0x7f3960004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:16.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.925+0000 7f397259c700 1 --2- 172.21.15.98:0/1942513377 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3960003c40 0x7f3960004030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:16.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.925+0000 7f3972d9d700 1 --2- 172.21.15.98:0/1942513377 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3960005210 0x7f396000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:16.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.925+0000 7f3969d9b700 1 -- 172.21.15.98:0/1942513377 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3960003c40 msgr2=0x7f3960004030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:16.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.925+0000 7f3969d9b700 1 --2- 172.21.15.98:0/1942513377 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3960003c40 0x7f3960004030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:16.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.925+0000 7f3969d9b700 1 -- 172.21.15.98:0/1942513377 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3960005210 msgr2=0x7f396000ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:16.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.925+0000 7f3969d9b700 1 --2- 172.21.15.98:0/1942513377 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3960005210 0x7f396000ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:16.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.925+0000 7f3972d9d700 1 -- 172.21.15.98:0/1942513377 reap_dead start 2023-10-06T05:07:16.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.925+0000 7f3969d9b700 1 -- 172.21.15.98:0/1942513377 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3960003570 con 0x7f3960004810 2023-10-06T05:07:16.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.925+0000 7f3969d9b700 1 --2- 172.21.15.98:0/1942513377 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3960004810 0x7f3960004c80 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f395c00c1e0 tx=0x7f395c001d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=557cfb36aaaf2130 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:16.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.925+0000 7f3971d9b700 1 -- 172.21.15.98:0/1942513377 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f395c00cba0 con 0x7f3960004810 2023-10-06T05:07:16.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.926+0000 7f3971d9b700 1 -- 172.21.15.98:0/1942513377 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f395c00cd00 con 0x7f3960004810 2023-10-06T05:07:16.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.926+0000 7f3971d9b700 1 -- 172.21.15.98:0/1942513377 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f395c005b70 con 0x7f3960004810 2023-10-06T05:07:16.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.926+0000 7f3978c39700 1 -- 172.21.15.98:0/1942513377 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3960004810 msgr2=0x7f3960004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:16.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.926+0000 7f3978c39700 1 --2- 172.21.15.98:0/1942513377 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3960004810 0x7f3960004c80 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f395c00c1e0 tx=0x7f395c001d40 comp rx=0 tx=0).stop 2023-10-06T05:07:16.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.926+0000 7f3978c39700 1 -- 172.21.15.98:0/1942513377 shutdown_connections 2023-10-06T05:07:16.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.926+0000 7f3978c39700 1 --2- 172.21.15.98:0/1942513377 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3960004810 0x7f3960004c80 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:16.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.926+0000 7f3978c39700 1 -- 172.21.15.98:0/1942513377 >> 172.21.15.98:0/1942513377 conn(0x7f39740997f0 msgr2=0x7f397406f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:16.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.926+0000 7f3978c39700 1 -- 172.21.15.98:0/1942513377 shutdown_connections 2023-10-06T05:07:16.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.926+0000 7f3978c39700 1 -- 172.21.15.98:0/1942513377 wait complete. 2023-10-06T05:07:16.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.927+0000 7f3978c39700 1 Processor -- start 2023-10-06T05:07:16.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.927+0000 7f3978c39700 1 -- start start 2023-10-06T05:07:16.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.928+0000 7f3978c39700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3960004810 0x7f397415cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:16.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.928+0000 7f3978c39700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f397409d9f0 0x7f39740aeec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:16.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.928+0000 7f3978c39700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f39741480d0 0x7f397409b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:16.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.928+0000 7f3978c39700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f397415c290 con 0x7f3960004810 2023-10-06T05:07:16.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.928+0000 7f3978c39700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f397409deb0 con 0x7f397409d9f0 2023-10-06T05:07:16.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.928+0000 7f3978c39700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f397415c550 con 0x7f39741480d0 2023-10-06T05:07:16.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.928+0000 7f397259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3960004810 0x7f397415cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:16.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.928+0000 7f397259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3960004810 0x7f397415cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46052/0 (socket says 172.21.15.98:46052) 2023-10-06T05:07:16.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.928+0000 7f397259c700 1 -- 172.21.15.98:0/1733040345 learned_addr learned my addr 172.21.15.98:0/1733040345 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:16.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.928+0000 7f397259c700 1 -- 172.21.15.98:0/1733040345 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f39741480d0 msgr2=0x7f397409b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:16.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.928+0000 7f397259c700 1 --2- 172.21.15.98:0/1733040345 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f39741480d0 0x7f397409b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:16.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.928+0000 7f397259c700 1 -- 172.21.15.98:0/1733040345 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f397409d9f0 msgr2=0x7f39740aeec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:16.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.928+0000 7f397259c700 1 --2- 172.21.15.98:0/1733040345 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f397409d9f0 0x7f39740aeec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:16.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.928+0000 7f397259c700 1 -- 172.21.15.98:0/1733040345 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f397415c830 con 0x7f3960004810 2023-10-06T05:07:16.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.929+0000 7f397259c700 1 --2- 172.21.15.98:0/1733040345 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3960004810 0x7f397415cc20 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f3964011d20 tx=0x7f3964009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:16.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.929+0000 7f396affd700 1 -- 172.21.15.98:0/1733040345 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3964011ed0 con 0x7f3960004810 2023-10-06T05:07:16.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.929+0000 7f396affd700 1 -- 172.21.15.98:0/1733040345 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f396400fcd0 con 0x7f3960004810 2023-10-06T05:07:16.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.929+0000 7f3978c39700 1 -- 172.21.15.98:0/1733040345 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f397415a370 con 0x7f3960004810 2023-10-06T05:07:16.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.929+0000 7f396affd700 1 -- 172.21.15.98:0/1733040345 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3964014690 con 0x7f3960004810 2023-10-06T05:07:16.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.929+0000 7f3978c39700 1 -- 172.21.15.98:0/1733040345 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3974157ed0 con 0x7f3960004810 2023-10-06T05:07:16.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.930+0000 7f3978c39700 1 -- 172.21.15.98:0/1733040345 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f397415ca20 con 0x7f3960004810 2023-10-06T05:07:16.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.930+0000 7f396affd700 1 -- 172.21.15.98:0/1733040345 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f3964014ac0 con 0x7f3960004810 2023-10-06T05:07:16.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.931+0000 7f396affd700 1 --2- 172.21.15.98:0/1733040345 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f3954066f30 0x7f39540693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:16.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.931+0000 7f3969d9b700 1 --2- 172.21.15.98:0/1733040345 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f3954066f30 0x7f39540693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:16.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.931+0000 7f396affd700 1 -- 172.21.15.98:0/1733040345 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f3964018030 con 0x7f3960004810 2023-10-06T05:07:16.941 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.931+0000 7f3969d9b700 1 --2- 172.21.15.98:0/1733040345 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f3954066f30 0x7f39540693e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f39740ce490 tx=0x7f395c015000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:16.941 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:16.935+0000 7f396affd700 1 -- 172.21.15.98:0/1733040345 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3964052240 con 0x7f3960004810 2023-10-06T05:07:17.163 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.161+0000 7f3978c39700 1 -- 172.21.15.98:0/1733040345 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f3974064a90 con 0x7f3960004810 2023-10-06T05:07:17.163 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.161+0000 7f396affd700 1 -- 172.21.15.98:0/1733040345 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f3964055860 con 0x7f3960004810 2023-10-06T05:07:17.164 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:07:17.164 INFO:teuthology.orchestra.run.smithi098.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-10-06T05:07:17.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.163+0000 7f3978c39700 1 -- 172.21.15.98:0/1733040345 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f3954066f30 msgr2=0x7f39540693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:17.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.163+0000 7f3978c39700 1 --2- 172.21.15.98:0/1733040345 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f3954066f30 0x7f39540693e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f39740ce490 tx=0x7f395c015000 comp rx=0 tx=0).stop 2023-10-06T05:07:17.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.163+0000 7f3978c39700 1 -- 172.21.15.98:0/1733040345 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3960004810 msgr2=0x7f397415cc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:17.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.163+0000 7f3978c39700 1 --2- 172.21.15.98:0/1733040345 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3960004810 0x7f397415cc20 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f3964011d20 tx=0x7f3964009120 comp rx=0 tx=0).stop 2023-10-06T05:07:17.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.163+0000 7f3978c39700 1 -- 172.21.15.98:0/1733040345 shutdown_connections 2023-10-06T05:07:17.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.163+0000 7f3978c39700 1 --2- 172.21.15.98:0/1733040345 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f39741480d0 0x7f397409b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:17.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.163+0000 7f3978c39700 1 --2- 172.21.15.98:0/1733040345 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f3954066f30 0x7f39540693e0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:17.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.163+0000 7f3978c39700 1 --2- 172.21.15.98:0/1733040345 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3960004810 0x7f397415cc20 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:17.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.163+0000 7f3978c39700 1 --2- 172.21.15.98:0/1733040345 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f397409d9f0 0x7f39740aeec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:17.168 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.163+0000 7f3978c39700 1 -- 172.21.15.98:0/1733040345 >> 172.21.15.98:0/1733040345 conn(0x7f39740997f0 msgr2=0x7f39740b1a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:17.168 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.163+0000 7f3978c39700 1 -- 172.21.15.98:0/1733040345 shutdown_connections 2023-10-06T05:07:17.168 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.163+0000 7f3978c39700 1 -- 172.21.15.98:0/1733040345 wait complete. 2023-10-06T05:07:17.178 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-10-06T05:07:17.178 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T05:07:17.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.289+0000 7ff97a474700 1 Processor -- start 2023-10-06T05:07:17.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.289+0000 7ff97a474700 1 -- start start 2023-10-06T05:07:17.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.290+0000 7ff97a474700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff974159560 0x7ff9740aa530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:17.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.290+0000 7ff97a474700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9740b0c60 con 0x7ff9740b49a0 2023-10-06T05:07:17.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.290+0000 7ff97a474700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9740b0f40 con 0x7ff9740b0760 2023-10-06T05:07:17.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.290+0000 7ff97a474700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9740b1220 con 0x7ff974159560 2023-10-06T05:07:17.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.290+0000 7ff9737fe700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7ff9740b49a0 0x7ff9740aaa70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:41264/0 (socket says 172.21.15.98:41264) 2023-10-06T05:07:17.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.290+0000 7ff9737fe700 1 -- 172.21.15.98:0/2872051531 learned_addr learned my addr 172.21.15.98:0/2872051531 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:07:17.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.290+0000 7ff973fff700 1 --2- 172.21.15.98:0/2872051531 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff974159560 0x7ff9740aa530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:17.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.290+0000 7ff972ffd700 1 -- 172.21.15.98:0/2872051531 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4116436744 0 0) 0x7ff9740b0c60 con 0x7ff9740b49a0 2023-10-06T05:07:17.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.290+0000 7ff972ffd700 1 -- 172.21.15.98:0/2872051531 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff95c003570 con 0x7ff9740b49a0 2023-10-06T05:07:17.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.291+0000 7ff972ffd700 1 -- 172.21.15.98:0/2872051531 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3966567725 0 0) 0x7ff95c003570 con 0x7ff9740b49a0 2023-10-06T05:07:17.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.291+0000 7ff972ffd700 1 -- 172.21.15.98:0/2872051531 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff974159560 msgr2=0x7ff9740aa530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:17.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.291+0000 7ff972ffd700 1 --2- 172.21.15.98:0/2872051531 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff974159560 0x7ff9740aa530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:17.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.291+0000 7ff972ffd700 1 -- 172.21.15.98:0/2872051531 >> v1:172.21.15.174:6789/0 conn(0x7ff9740b0760 legacy=0x7ff9740b1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:17.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.291+0000 7ff972ffd700 1 -- 172.21.15.98:0/2872051531 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9740b1500 con 0x7ff9740b49a0 2023-10-06T05:07:17.296 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.291+0000 7ff972ffd700 1 -- 172.21.15.98:0/2872051531 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1543991876 0 0) 0x7ff968001470 con 0x7ff9740b49a0 2023-10-06T05:07:17.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.291+0000 7ff972ffd700 1 -- 172.21.15.98:0/2872051531 >> v1:172.21.15.98:6789/0 conn(0x7ff9740b49a0 legacy=0x7ff9740aaa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:17.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.291+0000 7ff972ffd700 1 --2- 172.21.15.98:0/2872051531 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff95c003a90 0x7ff95c003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:17.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.291+0000 7ff972ffd700 1 --2- 172.21.15.98:0/2872051531 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff974159560 0x7ff95c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:17.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.291+0000 7ff972ffd700 1 --2- 172.21.15.98:0/2872051531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff95c004d50 0x7ff95c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:17.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.291+0000 7ff972ffd700 1 -- 172.21.15.98:0/2872051531 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9740b1220 con 0x7ff95c004d50 2023-10-06T05:07:17.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.291+0000 7ff972ffd700 1 -- 172.21.15.98:0/2872051531 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff95c00bb70 con 0x7ff95c003a90 2023-10-06T05:07:17.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.291+0000 7ff972ffd700 1 -- 172.21.15.98:0/2872051531 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff95c00bd30 con 0x7ff974159560 2023-10-06T05:07:17.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.291+0000 7ff978a11700 1 --2- 172.21.15.98:0/2872051531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff95c004d50 0x7ff95c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:17.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.291+0000 7ff973fff700 1 --2- 172.21.15.98:0/2872051531 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff95c003a90 0x7ff95c003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:17.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.291+0000 7ff9737fe700 1 --2- 172.21.15.98:0/2872051531 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff974159560 0x7ff95c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:17.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.291+0000 7ff978a11700 1 -- 172.21.15.98:0/2872051531 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff974159560 msgr2=0x7ff95c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:17.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.291+0000 7ff978a11700 1 --2- 172.21.15.98:0/2872051531 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff974159560 0x7ff95c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:17.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.291+0000 7ff978a11700 1 -- 172.21.15.98:0/2872051531 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff95c003a90 msgr2=0x7ff95c003ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:17.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.291+0000 7ff978a11700 1 --2- 172.21.15.98:0/2872051531 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff95c003a90 0x7ff95c003ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:17.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.291+0000 7ff978a11700 1 -- 172.21.15.98:0/2872051531 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff95c003570 con 0x7ff95c004d50 2023-10-06T05:07:17.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.291+0000 7ff978a11700 1 --2- 172.21.15.98:0/2872051531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff95c004d50 0x7ff95c0051c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff96c001fc0 tx=0x7ff96c001640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39a009bd0d3f460a server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:17.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.292+0000 7ff972ffd700 1 -- 172.21.15.98:0/2872051531 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff96c001950 con 0x7ff95c004d50 2023-10-06T05:07:17.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.292+0000 7ff972ffd700 1 -- 172.21.15.98:0/2872051531 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff96c00c8b0 con 0x7ff95c004d50 2023-10-06T05:07:17.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.292+0000 7ff972ffd700 1 -- 172.21.15.98:0/2872051531 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff96c00cc50 con 0x7ff95c004d50 2023-10-06T05:07:17.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.292+0000 7ff97a474700 1 -- 172.21.15.98:0/2872051531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff95c004d50 msgr2=0x7ff95c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:17.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.292+0000 7ff97a474700 1 --2- 172.21.15.98:0/2872051531 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff95c004d50 0x7ff95c0051c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff96c001fc0 tx=0x7ff96c001640 comp rx=0 tx=0).stop 2023-10-06T05:07:17.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.292+0000 7ff978a11700 1 -- 172.21.15.98:0/2872051531 reap_dead start 2023-10-06T05:07:17.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.292+0000 7ff97a474700 1 -- 172.21.15.98:0/2872051531 shutdown_connections 2023-10-06T05:07:17.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.292+0000 7ff97a474700 1 -- 172.21.15.98:0/2872051531 >> 172.21.15.98:0/2872051531 conn(0x7ff9740997f0 msgr2=0x7ff97406f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:17.300 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.292+0000 7ff97a474700 1 -- 172.21.15.98:0/2872051531 shutdown_connections 2023-10-06T05:07:17.300 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.292+0000 7ff97a474700 1 -- 172.21.15.98:0/2872051531 wait complete. 2023-10-06T05:07:17.300 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.293+0000 7ff97a474700 1 Processor -- start 2023-10-06T05:07:17.300 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.293+0000 7ff97a474700 1 -- start start 2023-10-06T05:07:17.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.294+0000 7ff97a474700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff97416d160 0x7ff9740ac7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:17.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.294+0000 7ff97a474700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff9740ab990 0x7ff9741698b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:17.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.294+0000 7ff97a474700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff974169df0 0x7ff97409cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:17.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.294+0000 7ff97a474700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9740ad140 con 0x7ff97416d160 2023-10-06T05:07:17.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.294+0000 7ff97a474700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff97409cb30 con 0x7ff9740ab990 2023-10-06T05:07:17.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.294+0000 7ff97a474700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff97409c200 con 0x7ff974169df0 2023-10-06T05:07:17.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.294+0000 7ff973fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff97416d160 0x7ff9740ac7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:17.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.294+0000 7ff973fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff97416d160 0x7ff9740ac7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46074/0 (socket says 172.21.15.98:46074) 2023-10-06T05:07:17.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.294+0000 7ff973fff700 1 -- 172.21.15.98:0/3184834741 learned_addr learned my addr 172.21.15.98:0/3184834741 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:17.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.294+0000 7ff973fff700 1 -- 172.21.15.98:0/3184834741 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff974169df0 msgr2=0x7ff97409cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:17.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.294+0000 7ff973fff700 1 --2- 172.21.15.98:0/3184834741 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff974169df0 0x7ff97409cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:17.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.294+0000 7ff973fff700 1 -- 172.21.15.98:0/3184834741 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff9740ab990 msgr2=0x7ff9741698b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:17.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.294+0000 7ff973fff700 1 --2- 172.21.15.98:0/3184834741 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff9740ab990 0x7ff9741698b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:17.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.294+0000 7ff973fff700 1 -- 172.21.15.98:0/3184834741 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff97409c4e0 con 0x7ff97416d160 2023-10-06T05:07:17.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.295+0000 7ff973fff700 1 --2- 172.21.15.98:0/3184834741 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff97416d160 0x7ff9740ac7f0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff964011b10 tx=0x7ff964009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:17.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.295+0000 7ff95bfff700 1 -- 172.21.15.98:0/3184834741 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff96400f9f0 con 0x7ff97416d160 2023-10-06T05:07:17.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.295+0000 7ff95bfff700 1 -- 172.21.15.98:0/3184834741 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff96400fb50 con 0x7ff97416d160 2023-10-06T05:07:17.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.295+0000 7ff97a474700 1 -- 172.21.15.98:0/3184834741 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff97409c7c0 con 0x7ff97416d160 2023-10-06T05:07:17.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.295+0000 7ff95bfff700 1 -- 172.21.15.98:0/3184834741 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff96401d480 con 0x7ff97416d160 2023-10-06T05:07:17.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.295+0000 7ff97a474700 1 -- 172.21.15.98:0/3184834741 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9740acef0 con 0x7ff97416d160 2023-10-06T05:07:17.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.296+0000 7ff97a474700 1 -- 172.21.15.98:0/3184834741 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff974064a90 con 0x7ff97416d160 2023-10-06T05:07:17.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.296+0000 7ff95bfff700 1 -- 172.21.15.98:0/3184834741 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7ff96401d870 con 0x7ff97416d160 2023-10-06T05:07:17.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.296+0000 7ff95bfff700 1 --2- 172.21.15.98:0/3184834741 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7ff960066f30 0x7ff9600693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:17.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.296+0000 7ff95bfff700 1 -- 172.21.15.98:0/3184834741 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7ff964018070 con 0x7ff97416d160 2023-10-06T05:07:17.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.296+0000 7ff9737fe700 1 --2- 172.21.15.98:0/3184834741 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7ff960066f30 0x7ff9600693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:17.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.297+0000 7ff9737fe700 1 --2- 172.21.15.98:0/3184834741 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7ff960066f30 0x7ff9600693e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ff968000c00 tx=0x7ff968001e50 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:17.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.301+0000 7ff95bfff700 1 -- 172.21.15.98:0/3184834741 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff9640525a0 con 0x7ff97416d160 2023-10-06T05:07:17.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.486+0000 7ff97a474700 1 -- 172.21.15.98:0/3184834741 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9740b17e0 con 0x7ff97416d160 2023-10-06T05:07:17.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.487+0000 7ff95bfff700 1 -- 172.21.15.98:0/3184834741 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+18707 (secure 0 0 0) 0x7ff964055bc0 con 0x7ff97416d160 2023-10-06T05:07:17.489 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:07:17.490 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":29,"fsid":"f50c5df5-cad5-47a4-9bb1-8c791d4311ec","created":"2023-10-06T05:06:32.971175+0000","modified":"2023-10-06T05:07:02.131672+0000","last_up_change":"2023-10-06T05:06:44.016489+0000","last_in_change":"2023-10-06T05:06:35.544511+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-06T05:06:44.559671+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-06T05:06:51.432161+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-10-06T05:06:55.589057+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-10-06T05:06:56.041687+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-10-06T05:06:59.754099+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d9495d95-da75-4c3b-8927-e9bae6a8b093","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":90627},{"type":"v1","addr":"172.21.15.98:6801","nonce":90627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":90627},{"type":"v1","addr":"172.21.15.98:6803","nonce":90627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":90627},{"type":"v1","addr":"172.21.15.98:6807","nonce":90627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":90627},{"type":"v1","addr":"172.21.15.98:6805","nonce":90627}]},"public_addr":"172.21.15.98:6801/90627","cluster_addr":"172.21.15.98:6803/90627","heartbeat_back_addr":"172.21.15.98:6807/90627","heartbeat_front_addr":"172.21.15.98:6805/90627","state":["exists","up"]},{"osd":1,"uuid":"f25ba5be-d65e-4707-a134-154092deb754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":90628},{"type":"v1","addr":"172.21.15.98:6809","nonce":90628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":90628},{"type":"v1","addr":"172.21.15.98:6811","nonce":90628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":90628},{"type":"v1","addr":"172.21.15.98:6815","nonce":90628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":90628},{"type":"v1","addr":"172.21.15.98:6813","nonce":90628}]},"public_addr":"172.21.15.98:6809/90628","cluster_addr":"172.21.15.98:6811/90628","heartbeat_back_addr":"172.21.15.98:6815/90628","heartbeat_front_addr":"172.21.15.98:6813/90628","state":["exists","up"]},{"osd":2,"uuid":"5ab9e028-945c-43e2-8c95-3b0b4cf38e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":90662},{"type":"v1","addr":"172.21.15.98:6818","nonce":90662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":90662},{"type":"v1","addr":"172.21.15.98:6822","nonce":90662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":90662},{"type":"v1","addr":"172.21.15.98:6830","nonce":90662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":90662},{"type":"v1","addr":"172.21.15.98:6826","nonce":90662}]},"public_addr":"172.21.15.98:6818/90662","cluster_addr":"172.21.15.98:6822/90662","heartbeat_back_addr":"172.21.15.98:6830/90662","heartbeat_front_addr":"172.21.15.98:6826/90662","state":["exists","up"]},{"osd":3,"uuid":"3bb803ff-1738-46bf-be62-2267bf378822","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6817","nonce":90663},{"type":"v1","addr":"172.21.15.98:6819","nonce":90663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6821","nonce":90663},{"type":"v1","addr":"172.21.15.98:6823","nonce":90663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6829","nonce":90663},{"type":"v1","addr":"172.21.15.98:6831","nonce":90663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6825","nonce":90663},{"type":"v1","addr":"172.21.15.98:6827","nonce":90663}]},"public_addr":"172.21.15.98:6819/90663","cluster_addr":"172.21.15.98:6823/90663","heartbeat_back_addr":"172.21.15.98:6831/90663","heartbeat_front_addr":"172.21.15.98:6827/90663","state":["exists","up"]},{"osd":4,"uuid":"7d57a4c9-6462-43f6-b64a-7f9e9b4353d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":90847},{"type":"v1","addr":"172.21.15.174:6801","nonce":90847}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":90847},{"type":"v1","addr":"172.21.15.174:6803","nonce":90847}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":90847},{"type":"v1","addr":"172.21.15.174:6807","nonce":90847}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":90847},{"type":"v1","addr":"172.21.15.174:6805","nonce":90847}]},"public_addr":"172.21.15.174:6801/90847","cluster_addr":"172.21.15.174:6803/90847","heartbeat_back_addr":"172.21.15.174:6807/90847","heartbeat_front_addr":"172.21.15.174:6805/90847","state":["exists","up"]},{"osd":5,"uuid":"83287ea4-e44d-4af9-bc77-36d47c60d4a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":90848},{"type":"v1","addr":"172.21.15.174:6809","nonce":90848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":90848},{"type":"v1","addr":"172.21.15.174:6811","nonce":90848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":90848},{"type":"v1","addr":"172.21.15.174:6815","nonce":90848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":90848},{"type":"v1","addr":"172.21.15.174:6813","nonce":90848}]},"public_addr":"172.21.15.174:6809/90848","cluster_addr":"172.21.15.174:6811/90848","heartbeat_back_addr":"172.21.15.174:6815/90848","heartbeat_front_addr":"172.21.15.174:6813/90848","state":["exists","up"]},{"osd":6,"uuid":"496a6272-892e-43c4-a145-206623c22f99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":90877},{"type":"v1","addr":"172.21.15.174:6817","nonce":90877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":90877},{"type":"v1","addr":"172.21.15.174:6819","nonce":90877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":90877},{"type":"v1","addr":"172.21.15.174:6823","nonce":90877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":90877},{"type":"v1","addr":"172.21.15.174:6821","nonce":90877}]},"public_addr":"172.21.15.174:6817/90877","cluster_addr":"172.21.15.174:6819/90877","heartbeat_back_addr":"172.21.15.174:6823/90877","heartbeat_front_addr":"172.21.15.174:6821/90877","state":["exists","up"]},{"osd":7,"uuid":"49967e34-8b07-4bab-a13e-0c16d6a0f3a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":90878},{"type":"v1","addr":"172.21.15.174:6825","nonce":90878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":90878},{"type":"v1","addr":"172.21.15.174:6827","nonce":90878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":90878},{"type":"v1","addr":"172.21.15.174:6831","nonce":90878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":90878},{"type":"v1","addr":"172.21.15.174:6829","nonce":90878}]},"public_addr":"172.21.15.174:6825/90878","cluster_addr":"172.21.15.174:6827/90878","heartbeat_back_addr":"172.21.15.174:6831/90878","heartbeat_front_addr":"172.21.15.174:6829/90878","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.446479+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.514266+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.794108+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.810894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.731919+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.717756+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.737435+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.508682+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T05:07:17.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.489+0000 7ff97a474700 1 -- 172.21.15.98:0/3184834741 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7ff960066f30 msgr2=0x7ff9600693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:17.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.489+0000 7ff97a474700 1 --2- 172.21.15.98:0/3184834741 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7ff960066f30 0x7ff9600693e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ff968000c00 tx=0x7ff968001e50 comp rx=0 tx=0).stop 2023-10-06T05:07:17.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.489+0000 7ff97a474700 1 -- 172.21.15.98:0/3184834741 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff97416d160 msgr2=0x7ff9740ac7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:17.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.489+0000 7ff97a474700 1 --2- 172.21.15.98:0/3184834741 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff97416d160 0x7ff9740ac7f0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff964011b10 tx=0x7ff964009120 comp rx=0 tx=0).stop 2023-10-06T05:07:17.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.489+0000 7ff97a474700 1 -- 172.21.15.98:0/3184834741 shutdown_connections 2023-10-06T05:07:17.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.489+0000 7ff97a474700 1 --2- 172.21.15.98:0/3184834741 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff974169df0 0x7ff97409cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:17.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.489+0000 7ff97a474700 1 --2- 172.21.15.98:0/3184834741 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7ff960066f30 0x7ff9600693e0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:17.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.489+0000 7ff97a474700 1 --2- 172.21.15.98:0/3184834741 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff97416d160 0x7ff9740ac7f0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:17.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.489+0000 7ff97a474700 1 --2- 172.21.15.98:0/3184834741 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff9740ab990 0x7ff9741698b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:17.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.489+0000 7ff97a474700 1 -- 172.21.15.98:0/3184834741 >> 172.21.15.98:0/3184834741 conn(0x7ff9740997f0 msgr2=0x7ff974069b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:17.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.489+0000 7ff97a474700 1 -- 172.21.15.98:0/3184834741 shutdown_connections 2023-10-06T05:07:17.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.489+0000 7ff97a474700 1 -- 172.21.15.98:0/3184834741 wait complete. 2023-10-06T05:07:17.503 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-06T05:06:44.559671+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-10-06T05:06:51.432161+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-06T05:06:55.589057+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-10-06T05:06:56.041687+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-06T05:06:59.754099+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '29', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-06T05:07:17.504 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-06T05:07:17.620 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.618+0000 7f0dbf38a700 1 Processor -- start 2023-10-06T05:07:17.620 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.618+0000 7f0dbf38a700 1 -- start start 2023-10-06T05:07:17.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.619+0000 7f0dbf38a700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f0db8159470 0x7f0db80ce980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:17.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.619+0000 7f0dbf38a700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0db80cd020 0x7f0db80c6880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:17.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.619+0000 7f0dbf38a700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0db80cbb80 con 0x7f0db8159840 2023-10-06T05:07:17.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.619+0000 7f0dbf38a700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0db80c6f00 con 0x7f0db8159470 2023-10-06T05:07:17.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.619+0000 7f0dbf38a700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0db80cb530 con 0x7f0db80cd020 2023-10-06T05:07:17.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.619+0000 7f0dbd126700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f0db8159470 0x7f0db80ce980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:17.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.619+0000 7f0dbd927700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0db80cd020 0x7f0db80c6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:17.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.619+0000 7f0dbc925700 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f0db8159840 0x7f0db80c6170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.98:55644/0 (socket says 172.21.15.98:55644) 2023-10-06T05:07:17.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.620+0000 7f0dbc925700 1 -- 172.21.15.98:0/323693029 learned_addr learned my addr 172.21.15.98:0/323693029 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:07:17.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.620+0000 7f0dbd126700 1 -- 172.21.15.98:0/323693029 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0db80cd020 msgr2=0x7f0db80c6880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:17.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.620+0000 7f0dbd126700 1 --2- 172.21.15.98:0/323693029 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0db80cd020 0x7f0db80c6880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:17.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.620+0000 7f0dbd126700 1 -- 172.21.15.98:0/323693029 >> v1:172.21.15.174:6789/0 conn(0x7f0db8159840 legacy=0x7f0db80c6170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:17.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.620+0000 7f0dbd126700 1 -- 172.21.15.98:0/323693029 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0db80cb810 con 0x7f0db8159470 2023-10-06T05:07:17.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.620+0000 7f0dbd927700 1 --2- 172.21.15.98:0/323693029 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0db80cd020 0x7f0db80c6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T05:07:17.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.620+0000 7f0dbd126700 1 --2- 172.21.15.98:0/323693029 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0db8159470 0x7f0db80ce980 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0db40098d0 tx=0x7f0db4009120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=38df8bd4d52d2835 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:17.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.620+0000 7f0daffff700 1 -- 172.21.15.98:0/323693029 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db4005c10 con 0x7f0db8159470 2023-10-06T05:07:17.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.620+0000 7f0daffff700 1 -- 172.21.15.98:0/323693029 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0db4005d70 con 0x7f0db8159470 2023-10-06T05:07:17.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.620+0000 7f0daffff700 1 -- 172.21.15.98:0/323693029 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db4006280 con 0x7f0db8159470 2023-10-06T05:07:17.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.621+0000 7f0dbf38a700 1 -- 172.21.15.98:0/323693029 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0db8159470 msgr2=0x7f0db80ce980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:17.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.621+0000 7f0dbf38a700 1 --2- 172.21.15.98:0/323693029 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0db8159470 0x7f0db80ce980 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0db40098d0 tx=0x7f0db4009120 comp rx=0 tx=0).stop 2023-10-06T05:07:17.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.621+0000 7f0dbf38a700 1 -- 172.21.15.98:0/323693029 shutdown_connections 2023-10-06T05:07:17.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.621+0000 7f0dbf38a700 1 --2- 172.21.15.98:0/323693029 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0db80cd020 0x7f0db80c6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:17.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.621+0000 7f0dbf38a700 1 --2- 172.21.15.98:0/323693029 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0db8159470 0x7f0db80ce980 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:17.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.621+0000 7f0dbf38a700 1 -- 172.21.15.98:0/323693029 >> 172.21.15.98:0/323693029 conn(0x7f0db80996e0 msgr2=0x7f0db806f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:17.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.621+0000 7f0dbf38a700 1 -- 172.21.15.98:0/323693029 shutdown_connections 2023-10-06T05:07:17.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.621+0000 7f0dbf38a700 1 -- 172.21.15.98:0/323693029 wait complete. 2023-10-06T05:07:17.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.622+0000 7f0dbf38a700 1 Processor -- start 2023-10-06T05:07:17.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.622+0000 7f0dbf38a700 1 -- start start 2023-10-06T05:07:17.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.623+0000 7f0dbf38a700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0db80cd020 0x7f0db80c9db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:17.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.623+0000 7f0dbf38a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0db8159470 0x7f0db815b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:17.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.623+0000 7f0dbf38a700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0db8159840 0x7f0db815e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:17.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.623+0000 7f0dbf38a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0db8169f90 con 0x7f0db8159470 2023-10-06T05:07:17.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.623+0000 7f0dbf38a700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0db80c8f80 con 0x7f0db8159840 2023-10-06T05:07:17.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.623+0000 7f0dbf38a700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0db8159f90 con 0x7f0db80cd020 2023-10-06T05:07:17.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.623+0000 7f0dbc925700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0db8159470 0x7f0db815b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:17.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.623+0000 7f0dbc925700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0db8159470 0x7f0db815b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46082/0 (socket says 172.21.15.98:46082) 2023-10-06T05:07:17.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.623+0000 7f0dbc925700 1 -- 172.21.15.98:0/1641477880 learned_addr learned my addr 172.21.15.98:0/1641477880 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:17.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.623+0000 7f0dbd927700 1 --2- 172.21.15.98:0/1641477880 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0db8159840 0x7f0db815e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:17.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.623+0000 7f0dbd126700 1 --2- 172.21.15.98:0/1641477880 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0db80cd020 0x7f0db80c9db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:17.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.623+0000 7f0dbc925700 1 -- 172.21.15.98:0/1641477880 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0db80cd020 msgr2=0x7f0db80c9db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:17.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.623+0000 7f0dbc925700 1 --2- 172.21.15.98:0/1641477880 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0db80cd020 0x7f0db80c9db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:17.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.623+0000 7f0dbc925700 1 -- 172.21.15.98:0/1641477880 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0db8159840 msgr2=0x7f0db815e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:17.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.623+0000 7f0dbc925700 1 --2- 172.21.15.98:0/1641477880 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0db8159840 0x7f0db815e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:17.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.623+0000 7f0dbc925700 1 -- 172.21.15.98:0/1641477880 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0db0009a20 con 0x7f0db8159470 2023-10-06T05:07:17.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.623+0000 7f0dbc925700 1 --2- 172.21.15.98:0/1641477880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0db8159470 0x7f0db815b290 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f0da800c560 tx=0x7f0da8001020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:17.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.624+0000 7f0dad7fa700 1 -- 172.21.15.98:0/1641477880 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0da8005b30 con 0x7f0db8159470 2023-10-06T05:07:17.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.624+0000 7f0dad7fa700 1 -- 172.21.15.98:0/1641477880 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0da8005cb0 con 0x7f0db8159470 2023-10-06T05:07:17.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.624+0000 7f0dad7fa700 1 -- 172.21.15.98:0/1641477880 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0da8007320 con 0x7f0db8159470 2023-10-06T05:07:17.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.624+0000 7f0dbf38a700 1 -- 172.21.15.98:0/1641477880 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0db40093c0 con 0x7f0db8159470 2023-10-06T05:07:17.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.624+0000 7f0dbf38a700 1 -- 172.21.15.98:0/1641477880 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0db815a550 con 0x7f0db8159470 2023-10-06T05:07:17.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.625+0000 7f0dbf38a700 1 -- 172.21.15.98:0/1641477880 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0db8064a90 con 0x7f0db8159470 2023-10-06T05:07:17.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.625+0000 7f0dad7fa700 1 -- 172.21.15.98:0/1641477880 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f0da8007690 con 0x7f0db8159470 2023-10-06T05:07:17.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.625+0000 7f0dad7fa700 1 --2- 172.21.15.98:0/1641477880 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f0d9c066f30 0x7f0d9c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:17.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.625+0000 7f0dad7fa700 1 -- 172.21.15.98:0/1641477880 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f0da8083cf0 con 0x7f0db8159470 2023-10-06T05:07:17.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.625+0000 7f0dbd126700 1 --2- 172.21.15.98:0/1641477880 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f0d9c066f30 0x7f0d9c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:17.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.626+0000 7f0dbd126700 1 --2- 172.21.15.98:0/1641477880 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f0d9c066f30 0x7f0d9c0693e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f0db4009160 tx=0x7f0db4016000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:17.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.630+0000 7f0dad7fa700 1 -- 172.21.15.98:0/1641477880 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0da8088070 con 0x7f0db8159470 2023-10-06T05:07:17.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.821+0000 7f0dbf38a700 1 -- 172.21.15.98:0/1641477880 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0db80cb1e0 con 0x7f0db8159470 2023-10-06T05:07:17.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.821+0000 7f0dad7fa700 1 -- 172.21.15.98:0/1641477880 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v29) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0da80170a0 con 0x7f0db8159470 2023-10-06T05:07:17.824 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2023-10-06T05:07:17.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.823+0000 7f0dbf38a700 1 -- 172.21.15.98:0/1641477880 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f0d9c066f30 msgr2=0x7f0d9c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:17.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.823+0000 7f0dbf38a700 1 --2- 172.21.15.98:0/1641477880 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f0d9c066f30 0x7f0d9c0693e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f0db4009160 tx=0x7f0db4016000 comp rx=0 tx=0).stop 2023-10-06T05:07:17.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.823+0000 7f0dbf38a700 1 -- 172.21.15.98:0/1641477880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0db8159470 msgr2=0x7f0db815b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:17.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.823+0000 7f0dbf38a700 1 --2- 172.21.15.98:0/1641477880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0db8159470 0x7f0db815b290 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f0da800c560 tx=0x7f0da8001020 comp rx=0 tx=0).stop 2023-10-06T05:07:17.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.824+0000 7f0dbf38a700 1 -- 172.21.15.98:0/1641477880 shutdown_connections 2023-10-06T05:07:17.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.824+0000 7f0dbf38a700 1 --2- 172.21.15.98:0/1641477880 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0db8159840 0x7f0db815e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:17.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.824+0000 7f0dbf38a700 1 --2- 172.21.15.98:0/1641477880 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0db80cd020 0x7f0db80c9db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:17.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.824+0000 7f0dbf38a700 1 --2- 172.21.15.98:0/1641477880 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f0d9c066f30 0x7f0d9c0693e0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:17.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.824+0000 7f0dbf38a700 1 --2- 172.21.15.98:0/1641477880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0db8159470 0x7f0db815b290 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:17.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.824+0000 7f0dbf38a700 1 -- 172.21.15.98:0/1641477880 >> 172.21.15.98:0/1641477880 conn(0x7f0db80996e0 msgr2=0x7f0db8153be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:17.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.824+0000 7f0dbf38a700 1 -- 172.21.15.98:0/1641477880 shutdown_connections 2023-10-06T05:07:17.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.824+0000 7f0dbf38a700 1 -- 172.21.15.98:0/1641477880 wait complete. 2023-10-06T05:07:17.838 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-10-06T05:07:17.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.954+0000 7f0cb2252700 1 Processor -- start 2023-10-06T05:07:17.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.954+0000 7f0cb2252700 1 -- start start 2023-10-06T05:07:17.957 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.955+0000 7f0cb2252700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f0cac1598a0 0x7f0cac0ce7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:17.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.955+0000 7f0cb2252700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0cac0cbc70 0x7f0cac0ced20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:17.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.955+0000 7f0cb2252700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cac0cf5d0 con 0x7f0cac1594d0 2023-10-06T05:07:17.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.955+0000 7f0cb2252700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cac0cf8b0 con 0x7f0cac1598a0 2023-10-06T05:07:17.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.955+0000 7f0cb2252700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cac0cfb90 con 0x7f0cac0cbc70 2023-10-06T05:07:17.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.956+0000 7f0ca2ffd700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f0cac1598a0 0x7f0cac0ce7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:17.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.956+0000 7f0cabfff700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0cac0cbc70 0x7f0cac0ced20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:17.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.956+0000 7f0cab7fe700 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f0cac1594d0 0x7f0cac0cb560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.98:55646/0 (socket says 172.21.15.98:55646) 2023-10-06T05:07:17.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.956+0000 7f0cab7fe700 1 -- 172.21.15.98:0/1154527173 learned_addr learned my addr 172.21.15.98:0/1154527173 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:07:17.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.956+0000 7f0cabfff700 1 -- 172.21.15.98:0/1154527173 >> v2:172.21.15.174:3300/0 conn(0x7f0cac1598a0 msgr2=0x7f0cac0ce7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:17.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.956+0000 7f0cabfff700 1 --2- 172.21.15.98:0/1154527173 >> v2:172.21.15.174:3300/0 conn(0x7f0cac1598a0 0x7f0cac0ce7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:17.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.956+0000 7f0cabfff700 1 -- 172.21.15.98:0/1154527173 >> v1:172.21.15.174:6789/0 conn(0x7f0cac1594d0 legacy=0x7f0cac0cb560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:17.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.956+0000 7f0caaffd700 1 -- 172.21.15.98:0/1154527173 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2743698646 0 0) 0x7f0cac0cf5d0 con 0x7f0cac1594d0 2023-10-06T05:07:17.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.956+0000 7f0cabfff700 1 -- 172.21.15.98:0/1154527173 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cac0cfe70 con 0x7f0cac0cbc70 2023-10-06T05:07:17.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.956+0000 7f0ca2ffd700 1 --2- 172.21.15.98:0/1154527173 >> v2:172.21.15.174:3300/0 conn(0x7f0cac1598a0 0x7f0cac0ce7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T05:07:17.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.956+0000 7f0cabfff700 1 --2- 172.21.15.98:0/1154527173 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0cac0cbc70 0x7f0cac0ced20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0ca4009e60 tx=0x7f0ca4009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d9583141f4b67282 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:17.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.956+0000 7f0caaffd700 1 -- 172.21.15.98:0/1154527173 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca400e070 con 0x7f0cac0cbc70 2023-10-06T05:07:17.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.956+0000 7f0caaffd700 1 -- 172.21.15.98:0/1154527173 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ca400bb60 con 0x7f0cac0cbc70 2023-10-06T05:07:17.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.956+0000 7f0caaffd700 1 -- 172.21.15.98:0/1154527173 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca4012430 con 0x7f0cac0cbc70 2023-10-06T05:07:17.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.957+0000 7f0cb2252700 1 -- 172.21.15.98:0/1154527173 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0cac0cbc70 msgr2=0x7f0cac0ced20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:17.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.957+0000 7f0cb2252700 1 --2- 172.21.15.98:0/1154527173 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0cac0cbc70 0x7f0cac0ced20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0ca4009e60 tx=0x7f0ca4009780 comp rx=0 tx=0).stop 2023-10-06T05:07:17.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.957+0000 7f0cb2252700 1 -- 172.21.15.98:0/1154527173 shutdown_connections 2023-10-06T05:07:17.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.957+0000 7f0cb2252700 1 --2- 172.21.15.98:0/1154527173 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0cac0cbc70 0x7f0cac0ced20 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:17.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.957+0000 7f0cb2252700 1 --2- 172.21.15.98:0/1154527173 >> v2:172.21.15.174:3300/0 conn(0x7f0cac1598a0 0x7f0cac0ce7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:17.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.957+0000 7f0cb2252700 1 -- 172.21.15.98:0/1154527173 >> 172.21.15.98:0/1154527173 conn(0x7f0cac089e50 msgr2=0x7f0cac099770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:17.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.957+0000 7f0cb2252700 1 -- 172.21.15.98:0/1154527173 shutdown_connections 2023-10-06T05:07:17.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.957+0000 7f0cb2252700 1 -- 172.21.15.98:0/1154527173 wait complete. 2023-10-06T05:07:17.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.958+0000 7f0cb2252700 1 Processor -- start 2023-10-06T05:07:17.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.958+0000 7f0cb2252700 1 -- start start 2023-10-06T05:07:17.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.959+0000 7f0cb2252700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0cac0cbc70 0x7f0cac148b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:17.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.959+0000 7f0cb2252700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0cac1594d0 0x7f0cac149070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:17.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.959+0000 7f0cb2252700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0cac1598a0 0x7f0cac149930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:17.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.959+0000 7f0cb2252700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cac15b630 con 0x7f0cac1594d0 2023-10-06T05:07:17.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.959+0000 7f0cb2252700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cac15b820 con 0x7f0cac0cbc70 2023-10-06T05:07:17.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.959+0000 7f0cb2252700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cac15bb00 con 0x7f0cac1598a0 2023-10-06T05:07:17.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.959+0000 7f0ca2ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0cac1594d0 0x7f0cac149070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:17.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.959+0000 7f0ca2ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0cac1594d0 0x7f0cac149070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46098/0 (socket says 172.21.15.98:46098) 2023-10-06T05:07:17.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.959+0000 7f0ca2ffd700 1 -- 172.21.15.98:0/1604429935 learned_addr learned my addr 172.21.15.98:0/1604429935 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:17.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.959+0000 7f0cabfff700 1 --2- 172.21.15.98:0/1604429935 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0cac1598a0 0x7f0cac149930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:17.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.959+0000 7f0cab7fe700 1 --2- 172.21.15.98:0/1604429935 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0cac0cbc70 0x7f0cac148b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:17.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.959+0000 7f0ca2ffd700 1 -- 172.21.15.98:0/1604429935 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0cac1598a0 msgr2=0x7f0cac149930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:17.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.959+0000 7f0ca2ffd700 1 --2- 172.21.15.98:0/1604429935 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0cac1598a0 0x7f0cac149930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:17.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.959+0000 7f0ca2ffd700 1 -- 172.21.15.98:0/1604429935 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0cac0cbc70 msgr2=0x7f0cac148b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:17.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.959+0000 7f0ca2ffd700 1 --2- 172.21.15.98:0/1604429935 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0cac0cbc70 0x7f0cac148b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:17.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.959+0000 7f0ca2ffd700 1 -- 172.21.15.98:0/1604429935 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c94009420 con 0x7f0cac1594d0 2023-10-06T05:07:17.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.960+0000 7f0ca2ffd700 1 --2- 172.21.15.98:0/1604429935 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0cac1594d0 0x7f0cac149070 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f0c9400ea50 tx=0x7f0c940091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:17.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.960+0000 7f0ca3fff700 1 -- 172.21.15.98:0/1604429935 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c9400c920 con 0x7f0cac1594d0 2023-10-06T05:07:17.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.960+0000 7f0ca3fff700 1 -- 172.21.15.98:0/1604429935 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c9400ca80 con 0x7f0cac1594d0 2023-10-06T05:07:17.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.960+0000 7f0cb2252700 1 -- 172.21.15.98:0/1604429935 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ca4009a20 con 0x7f0cac1594d0 2023-10-06T05:07:17.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.960+0000 7f0ca3fff700 1 -- 172.21.15.98:0/1604429935 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c9400ce40 con 0x7f0cac1594d0 2023-10-06T05:07:17.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.960+0000 7f0cb2252700 1 -- 172.21.15.98:0/1604429935 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cac15c140 con 0x7f0cac1594d0 2023-10-06T05:07:17.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.961+0000 7f0cb2252700 1 -- 172.21.15.98:0/1604429935 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cac064a90 con 0x7f0cac1594d0 2023-10-06T05:07:17.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.961+0000 7f0ca3fff700 1 -- 172.21.15.98:0/1604429935 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f0c94013680 con 0x7f0cac1594d0 2023-10-06T05:07:17.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.961+0000 7f0ca3fff700 1 --2- 172.21.15.98:0/1604429935 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f0c8c066f30 0x7f0c8c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:17.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.961+0000 7f0ca3fff700 1 -- 172.21.15.98:0/1604429935 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f0c94084e90 con 0x7f0cac1594d0 2023-10-06T05:07:17.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.961+0000 7f0cab7fe700 1 --2- 172.21.15.98:0/1604429935 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f0c8c066f30 0x7f0c8c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:17.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.962+0000 7f0cab7fe700 1 --2- 172.21.15.98:0/1604429935 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f0c8c066f30 0x7f0c8c0693e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f0c9c0011e0 tx=0x7f0c9c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:17.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:17.966+0000 7f0ca3fff700 1 -- 172.21.15.98:0/1604429935 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0c94055740 con 0x7f0cac1594d0 2023-10-06T05:07:18.152 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.150+0000 7f0cb2252700 1 -- 172.21.15.98:0/1604429935 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f0cac0cf260 con 0x7f0cac1594d0 2023-10-06T05:07:18.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.150+0000 7f0ca3fff700 1 -- 172.21.15.98:0/1604429935 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v29) v1 ==== 92+0+10 (secure 0 0 0) 0x7f0c940552d0 con 0x7f0cac1594d0 2023-10-06T05:07:18.155 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 8 2023-10-06T05:07:18.157 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.152+0000 7f0cb2252700 1 -- 172.21.15.98:0/1604429935 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f0c8c066f30 msgr2=0x7f0c8c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:18.157 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.152+0000 7f0cb2252700 1 --2- 172.21.15.98:0/1604429935 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f0c8c066f30 0x7f0c8c0693e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f0c9c0011e0 tx=0x7f0c9c009040 comp rx=0 tx=0).stop 2023-10-06T05:07:18.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.152+0000 7f0cb2252700 1 -- 172.21.15.98:0/1604429935 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0cac1594d0 msgr2=0x7f0cac149070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:18.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.152+0000 7f0cb2252700 1 --2- 172.21.15.98:0/1604429935 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0cac1594d0 0x7f0cac149070 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f0c9400ea50 tx=0x7f0c940091a0 comp rx=0 tx=0).stop 2023-10-06T05:07:18.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.152+0000 7f0cb2252700 1 -- 172.21.15.98:0/1604429935 shutdown_connections 2023-10-06T05:07:18.159 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.152+0000 7f0cb2252700 1 --2- 172.21.15.98:0/1604429935 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0cac1598a0 0x7f0cac149930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:18.159 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.152+0000 7f0cb2252700 1 --2- 172.21.15.98:0/1604429935 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0cac0cbc70 0x7f0cac148b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:18.159 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.152+0000 7f0cb2252700 1 --2- 172.21.15.98:0/1604429935 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f0c8c066f30 0x7f0c8c0693e0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:18.159 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.152+0000 7f0cb2252700 1 --2- 172.21.15.98:0/1604429935 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0cac1594d0 0x7f0cac149070 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:18.159 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.152+0000 7f0cb2252700 1 -- 172.21.15.98:0/1604429935 >> 172.21.15.98:0/1604429935 conn(0x7f0cac089e50 msgr2=0x7f0cac14b040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:18.160 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.152+0000 7f0cb2252700 1 -- 172.21.15.98:0/1604429935 shutdown_connections 2023-10-06T05:07:18.160 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.152+0000 7f0cb2252700 1 -- 172.21.15.98:0/1604429935 wait complete. 2023-10-06T05:07:18.166 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-06T05:07:18.301 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.283+0000 7ff360981700 1 Processor -- start 2023-10-06T05:07:18.301 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.283+0000 7ff360981700 1 -- start start 2023-10-06T05:07:18.301 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.284+0000 7ff360981700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ff35c0b0a10 0x7ff35c0cbee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:18.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.284+0000 7ff360981700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff35c0c53c0 con 0x7ff35c0cc420 2023-10-06T05:07:18.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.284+0000 7ff360981700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff35c0ccd00 con 0x7ff35c0b0640 2023-10-06T05:07:18.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.284+0000 7ff360981700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff35c0ca790 con 0x7ff35c0b0a10 2023-10-06T05:07:18.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.284+0000 7ff35ad9d700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7ff35c0cc420 0x7ff35c0c9050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:41268/0 (socket says 172.21.15.98:41268) 2023-10-06T05:07:18.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.284+0000 7ff35ad9d700 1 -- 172.21.15.98:0/3002027251 learned_addr learned my addr 172.21.15.98:0/3002027251 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:07:18.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.284+0000 7ff351d9b700 1 --2- 172.21.15.98:0/3002027251 >> v2:172.21.15.174:3300/0 conn(0x7ff35c0b0a10 0x7ff35c0cbee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:18.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.285+0000 7ff359d9b700 1 -- 172.21.15.98:0/3002027251 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2612820653 0 0) 0x7ff35c0c53c0 con 0x7ff35c0cc420 2023-10-06T05:07:18.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.285+0000 7ff359d9b700 1 -- 172.21.15.98:0/3002027251 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff348003570 con 0x7ff35c0cc420 2023-10-06T05:07:18.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.285+0000 7ff359d9b700 1 -- 172.21.15.98:0/3002027251 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3645398142 0 0) 0x7ff35c0ccd00 con 0x7ff35c0b0640 2023-10-06T05:07:18.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.285+0000 7ff359d9b700 1 -- 172.21.15.98:0/3002027251 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff35c0c53c0 con 0x7ff35c0b0640 2023-10-06T05:07:18.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.285+0000 7ff359d9b700 1 -- 172.21.15.98:0/3002027251 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2984931053 0 0) 0x7ff348003570 con 0x7ff35c0cc420 2023-10-06T05:07:18.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.285+0000 7ff359d9b700 1 -- 172.21.15.98:0/3002027251 >> v2:172.21.15.174:3300/0 conn(0x7ff35c0b0a10 msgr2=0x7ff35c0cbee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:18.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.285+0000 7ff359d9b700 1 --2- 172.21.15.98:0/3002027251 >> v2:172.21.15.174:3300/0 conn(0x7ff35c0b0a10 0x7ff35c0cbee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:18.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.285+0000 7ff359d9b700 1 -- 172.21.15.98:0/3002027251 >> v1:172.21.15.174:6789/0 conn(0x7ff35c0b0640 legacy=0x7ff35c0cecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:18.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.285+0000 7ff359d9b700 1 -- 172.21.15.98:0/3002027251 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff35c0ccfe0 con 0x7ff35c0cc420 2023-10-06T05:07:18.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.285+0000 7ff359d9b700 1 -- 172.21.15.98:0/3002027251 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1543991876 0 0) 0x7ff3540017f0 con 0x7ff35c0cc420 2023-10-06T05:07:18.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.285+0000 7ff351d9b700 1 --2- 172.21.15.98:0/3002027251 >> v2:172.21.15.174:3300/0 conn(0x7ff35c0b0a10 0x7ff35c0cbee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T05:07:18.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.285+0000 7ff359d9b700 1 -- 172.21.15.98:0/3002027251 >> v1:172.21.15.98:6789/0 conn(0x7ff35c0cc420 legacy=0x7ff35c0c9050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:18.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.285+0000 7ff359d9b700 1 --2- 172.21.15.98:0/3002027251 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff348003bf0 0x7ff348003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:18.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.285+0000 7ff359d9b700 1 --2- 172.21.15.98:0/3002027251 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff348004810 0x7ff348004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:18.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.285+0000 7ff359d9b700 1 --2- 172.21.15.98:0/3002027251 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff3480051c0 0x7ff34800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:18.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.285+0000 7ff359d9b700 1 -- 172.21.15.98:0/3002027251 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff35c0ca790 con 0x7ff348004810 2023-10-06T05:07:18.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.285+0000 7ff359d9b700 1 -- 172.21.15.98:0/3002027251 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff34800c160 con 0x7ff348003bf0 2023-10-06T05:07:18.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.285+0000 7ff359d9b700 1 -- 172.21.15.98:0/3002027251 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff34800c320 con 0x7ff3480051c0 2023-10-06T05:07:18.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.285+0000 7ff351d9b700 1 --2- 172.21.15.98:0/3002027251 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff348004810 0x7ff348004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:18.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.285+0000 7ff35a59c700 1 --2- 172.21.15.98:0/3002027251 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff348003bf0 0x7ff348003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:18.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.285+0000 7ff35ad9d700 1 --2- 172.21.15.98:0/3002027251 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff3480051c0 0x7ff34800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:18.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.285+0000 7ff351d9b700 1 -- 172.21.15.98:0/3002027251 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff3480051c0 msgr2=0x7ff34800ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:18.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.285+0000 7ff351d9b700 1 --2- 172.21.15.98:0/3002027251 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff3480051c0 0x7ff34800ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:18.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.285+0000 7ff351d9b700 1 -- 172.21.15.98:0/3002027251 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff348003bf0 msgr2=0x7ff348003fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:18.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.285+0000 7ff351d9b700 1 --2- 172.21.15.98:0/3002027251 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff348003bf0 0x7ff348003fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:18.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.285+0000 7ff351d9b700 1 -- 172.21.15.98:0/3002027251 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff348003570 con 0x7ff348004810 2023-10-06T05:07:18.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.285+0000 7ff35ad9d700 1 --2- 172.21.15.98:0/3002027251 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff3480051c0 0x7ff34800ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:07:18.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.285+0000 7ff35ad9d700 1 -- 172.21.15.98:0/3002027251 reap_dead start 2023-10-06T05:07:18.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.285+0000 7ff351d9b700 1 --2- 172.21.15.98:0/3002027251 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff348004810 0x7ff348004c80 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff344006f00 tx=0x7ff3440091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab6c198e5e560bb8 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:18.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.286+0000 7ff359d9b700 1 -- 172.21.15.98:0/3002027251 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff344009740 con 0x7ff348004810 2023-10-06T05:07:18.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.286+0000 7ff359d9b700 1 -- 172.21.15.98:0/3002027251 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff34400e920 con 0x7ff348004810 2023-10-06T05:07:18.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.286+0000 7ff359d9b700 1 -- 172.21.15.98:0/3002027251 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff34400ece0 con 0x7ff348004810 2023-10-06T05:07:18.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.286+0000 7ff360981700 1 -- 172.21.15.98:0/3002027251 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff348004810 msgr2=0x7ff348004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:18.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.286+0000 7ff360981700 1 --2- 172.21.15.98:0/3002027251 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff348004810 0x7ff348004c80 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff344006f00 tx=0x7ff3440091a0 comp rx=0 tx=0).stop 2023-10-06T05:07:18.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.286+0000 7ff360981700 1 -- 172.21.15.98:0/3002027251 shutdown_connections 2023-10-06T05:07:18.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.286+0000 7ff360981700 1 --2- 172.21.15.98:0/3002027251 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff348004810 0x7ff348004c80 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:18.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.286+0000 7ff360981700 1 -- 172.21.15.98:0/3002027251 >> 172.21.15.98:0/3002027251 conn(0x7ff35c087fc0 msgr2=0x7ff35c0ce310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:18.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.286+0000 7ff360981700 1 -- 172.21.15.98:0/3002027251 shutdown_connections 2023-10-06T05:07:18.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.286+0000 7ff360981700 1 -- 172.21.15.98:0/3002027251 wait complete. 2023-10-06T05:07:18.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.287+0000 7ff360981700 1 Processor -- start 2023-10-06T05:07:18.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.287+0000 7ff360981700 1 -- start start 2023-10-06T05:07:18.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.288+0000 7ff360981700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff348004810 0x7ff35c0af4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:18.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.288+0000 7ff360981700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff35c159df0 0x7ff35c155db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:18.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.288+0000 7ff360981700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff35c0b2a90 0x7ff35c1562f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:18.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.288+0000 7ff360981700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff35c156830 con 0x7ff348004810 2023-10-06T05:07:18.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.288+0000 7ff360981700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff35c0b2f90 con 0x7ff35c0b2a90 2023-10-06T05:07:18.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.288+0000 7ff360981700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff35c0b3270 con 0x7ff35c159df0 2023-10-06T05:07:18.312 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.288+0000 7ff35a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff348004810 0x7ff35c0af4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:18.312 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.288+0000 7ff35a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff348004810 0x7ff35c0af4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46116/0 (socket says 172.21.15.98:46116) 2023-10-06T05:07:18.312 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.288+0000 7ff35a59c700 1 -- 172.21.15.98:0/1543856582 learned_addr learned my addr 172.21.15.98:0/1543856582 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:18.312 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.288+0000 7ff35ad9d700 1 --2- 172.21.15.98:0/1543856582 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff35c0b2a90 0x7ff35c1562f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:18.312 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.288+0000 7ff351d9b700 1 --2- 172.21.15.98:0/1543856582 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff35c159df0 0x7ff35c155db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:18.313 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.288+0000 7ff35a59c700 1 -- 172.21.15.98:0/1543856582 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff35c159df0 msgr2=0x7ff35c155db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:18.313 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.288+0000 7ff35a59c700 1 --2- 172.21.15.98:0/1543856582 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff35c159df0 0x7ff35c155db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:18.313 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.288+0000 7ff35a59c700 1 -- 172.21.15.98:0/1543856582 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff35c0b2a90 msgr2=0x7ff35c1562f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:18.313 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.288+0000 7ff35a59c700 1 --2- 172.21.15.98:0/1543856582 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff35c0b2a90 0x7ff35c1562f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:18.313 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.288+0000 7ff35a59c700 1 -- 172.21.15.98:0/1543856582 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff35c0b3550 con 0x7ff348004810 2023-10-06T05:07:18.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.288+0000 7ff35ad9d700 1 --2- 172.21.15.98:0/1543856582 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff35c0b2a90 0x7ff35c1562f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:07:18.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.289+0000 7ff35a59c700 1 --2- 172.21.15.98:0/1543856582 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff348004810 0x7ff35c0af4c0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ff34c00f880 tx=0x7ff34c007d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:18.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.289+0000 7ff352ffd700 1 -- 172.21.15.98:0/1543856582 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff34c006a80 con 0x7ff348004810 2023-10-06T05:07:18.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.289+0000 7ff352ffd700 1 -- 172.21.15.98:0/1543856582 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff34c006c00 con 0x7ff348004810 2023-10-06T05:07:18.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.289+0000 7ff360981700 1 -- 172.21.15.98:0/1543856582 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff35c0b3830 con 0x7ff348004810 2023-10-06T05:07:18.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.289+0000 7ff352ffd700 1 -- 172.21.15.98:0/1543856582 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff34c007050 con 0x7ff348004810 2023-10-06T05:07:18.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.289+0000 7ff360981700 1 -- 172.21.15.98:0/1543856582 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff35c0afa00 con 0x7ff348004810 2023-10-06T05:07:18.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.290+0000 7ff360981700 1 -- 172.21.15.98:0/1543856582 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff35c0afbf0 con 0x7ff348004810 2023-10-06T05:07:18.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.290+0000 7ff352ffd700 1 -- 172.21.15.98:0/1543856582 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7ff34c0073c0 con 0x7ff348004810 2023-10-06T05:07:18.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.291+0000 7ff352ffd700 1 --2- 172.21.15.98:0/1543856582 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7ff33c066f30 0x7ff33c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:18.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.291+0000 7ff351d9b700 1 --2- 172.21.15.98:0/1543856582 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7ff33c066f30 0x7ff33c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:18.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.291+0000 7ff352ffd700 1 -- 172.21.15.98:0/1543856582 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7ff34c084aa0 con 0x7ff348004810 2023-10-06T05:07:18.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.291+0000 7ff351d9b700 1 --2- 172.21.15.98:0/1543856582 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7ff33c066f30 0x7ff33c0693e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ff3440091e0 tx=0x7ff34401a000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:18.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.295+0000 7ff352ffd700 1 -- 172.21.15.98:0/1543856582 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff34c0518c0 con 0x7ff348004810 2023-10-06T05:07:18.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.476+0000 7ff360981700 1 -- 172.21.15.98:0/1543856582 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff35c064a90 con 0x7ff348004810 2023-10-06T05:07:18.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.477+0000 7ff352ffd700 1 -- 172.21.15.98:0/1543856582 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v29) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff34c054ee0 con 0x7ff348004810 2023-10-06T05:07:18.483 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2023-10-06T05:07:18.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.479+0000 7ff360981700 1 -- 172.21.15.98:0/1543856582 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7ff33c066f30 msgr2=0x7ff33c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:18.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.479+0000 7ff360981700 1 --2- 172.21.15.98:0/1543856582 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7ff33c066f30 0x7ff33c0693e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ff3440091e0 tx=0x7ff34401a000 comp rx=0 tx=0).stop 2023-10-06T05:07:18.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.479+0000 7ff360981700 1 -- 172.21.15.98:0/1543856582 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff348004810 msgr2=0x7ff35c0af4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:18.485 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.479+0000 7ff360981700 1 --2- 172.21.15.98:0/1543856582 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff348004810 0x7ff35c0af4c0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ff34c00f880 tx=0x7ff34c007d80 comp rx=0 tx=0).stop 2023-10-06T05:07:18.485 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.479+0000 7ff360981700 1 -- 172.21.15.98:0/1543856582 shutdown_connections 2023-10-06T05:07:18.485 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.479+0000 7ff360981700 1 --2- 172.21.15.98:0/1543856582 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff35c0b2a90 0x7ff35c1562f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:18.485 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.479+0000 7ff360981700 1 --2- 172.21.15.98:0/1543856582 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7ff33c066f30 0x7ff33c0693e0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:18.485 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.479+0000 7ff360981700 1 --2- 172.21.15.98:0/1543856582 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff348004810 0x7ff35c0af4c0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:18.485 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.479+0000 7ff360981700 1 --2- 172.21.15.98:0/1543856582 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff35c159df0 0x7ff35c155db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:18.485 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.479+0000 7ff360981700 1 -- 172.21.15.98:0/1543856582 >> 172.21.15.98:0/1543856582 conn(0x7ff35c087fc0 msgr2=0x7ff35c156ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:18.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.479+0000 7ff360981700 1 -- 172.21.15.98:0/1543856582 shutdown_connections 2023-10-06T05:07:18.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.479+0000 7ff360981700 1 -- 172.21.15.98:0/1543856582 wait complete. 2023-10-06T05:07:18.498 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-06T05:07:18.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.621+0000 7fb34199f700 1 Processor -- start 2023-10-06T05:07:18.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.621+0000 7fb34199f700 1 -- start start 2023-10-06T05:07:18.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.622+0000 7fb34199f700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb33c0b2620 0x7fb33c0cccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:18.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.622+0000 7fb34199f700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb33c0cf1b0 0x7fb33c0cd1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:18.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.622+0000 7fb34199f700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb33c0cc970 con 0x7fb33c0b29f0 2023-10-06T05:07:18.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.622+0000 7fb34199f700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb33c0cbd60 con 0x7fb33c0cf1b0 2023-10-06T05:07:18.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.622+0000 7fb34199f700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb33c0cc040 con 0x7fb33c0b2620 2023-10-06T05:07:18.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.622+0000 7fb34119e700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb33c0cf1b0 0x7fb33c0cd1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:18.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.622+0000 7fb33bfff700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fb33c0b29f0 0x7fb33c0ceaa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:41280/0 (socket says 172.21.15.98:41280) 2023-10-06T05:07:18.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.622+0000 7fb33bfff700 1 -- 172.21.15.98:0/1924742412 learned_addr learned my addr 172.21.15.98:0/1924742412 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:07:18.633 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.622+0000 7fb34099d700 1 --2- 172.21.15.98:0/1924742412 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb33c0b2620 0x7fb33c0cccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:18.633 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.622+0000 7fb34119e700 1 -- 172.21.15.98:0/1924742412 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb33c0b2620 msgr2=0x7fb33c0cccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:18.633 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.622+0000 7fb34119e700 1 --2- 172.21.15.98:0/1924742412 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb33c0b2620 0x7fb33c0cccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:18.633 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.622+0000 7fb34119e700 1 -- 172.21.15.98:0/1924742412 >> v1:172.21.15.98:6789/0 conn(0x7fb33c0b29f0 legacy=0x7fb33c0ceaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:18.633 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.622+0000 7fb34119e700 1 -- 172.21.15.98:0/1924742412 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb33c0cc320 con 0x7fb33c0cf1b0 2023-10-06T05:07:18.634 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.622+0000 7fb33b7fe700 1 -- 172.21.15.98:0/1924742412 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1139122802 0 0) 0x7fb33c0cc970 con 0x7fb33c0b29f0 2023-10-06T05:07:18.634 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.622+0000 7fb34119e700 1 --2- 172.21.15.98:0/1924742412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb33c0cf1b0 0x7fb33c0cd1f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fb3340098e0 tx=0x7fb3340091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6c9b40f4d77336 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:18.634 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.622+0000 7fb33b7fe700 1 -- 172.21.15.98:0/1924742412 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb33400b640 con 0x7fb33c0cf1b0 2023-10-06T05:07:18.634 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.622+0000 7fb33b7fe700 1 -- 172.21.15.98:0/1924742412 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb33400b7a0 con 0x7fb33c0cf1b0 2023-10-06T05:07:18.634 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.622+0000 7fb33b7fe700 1 -- 172.21.15.98:0/1924742412 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb33400bb40 con 0x7fb33c0cf1b0 2023-10-06T05:07:18.634 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.623+0000 7fb34199f700 1 -- 172.21.15.98:0/1924742412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb33c0cf1b0 msgr2=0x7fb33c0cd1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:18.634 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.623+0000 7fb34199f700 1 --2- 172.21.15.98:0/1924742412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb33c0cf1b0 0x7fb33c0cd1f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fb3340098e0 tx=0x7fb3340091a0 comp rx=0 tx=0).stop 2023-10-06T05:07:18.634 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.623+0000 7fb34199f700 1 -- 172.21.15.98:0/1924742412 shutdown_connections 2023-10-06T05:07:18.634 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.623+0000 7fb34199f700 1 --2- 172.21.15.98:0/1924742412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb33c0cf1b0 0x7fb33c0cd1f0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:18.634 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.623+0000 7fb34199f700 1 --2- 172.21.15.98:0/1924742412 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb33c0b2620 0x7fb33c0cccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:18.634 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.623+0000 7fb34199f700 1 -- 172.21.15.98:0/1924742412 >> 172.21.15.98:0/1924742412 conn(0x7fb33c099520 msgr2=0x7fb33c06f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:18.635 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.623+0000 7fb34199f700 1 -- 172.21.15.98:0/1924742412 shutdown_connections 2023-10-06T05:07:18.635 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.623+0000 7fb34199f700 1 -- 172.21.15.98:0/1924742412 wait complete. 2023-10-06T05:07:18.635 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.624+0000 7fb34199f700 1 Processor -- start 2023-10-06T05:07:18.635 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.624+0000 7fb34199f700 1 -- start start 2023-10-06T05:07:18.635 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.625+0000 7fb34199f700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb33c0b2620 0x7fb33c0b3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:18.635 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.625+0000 7fb34199f700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb33c0b29f0 0x7fb33c14c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:18.635 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.625+0000 7fb34199f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb33c0cf1b0 0x7fb33c0b4620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:18.635 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.625+0000 7fb34199f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb33c0b3a60 con 0x7fb33c0cf1b0 2023-10-06T05:07:18.635 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.625+0000 7fb34199f700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb33c0b5030 con 0x7fb33c0b2620 2023-10-06T05:07:18.635 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.625+0000 7fb34199f700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb33c0b3130 con 0x7fb33c0b29f0 2023-10-06T05:07:18.635 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.625+0000 7fb34119e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb33c0cf1b0 0x7fb33c0b4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:18.635 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.625+0000 7fb34119e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb33c0cf1b0 0x7fb33c0b4620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46126/0 (socket says 172.21.15.98:46126) 2023-10-06T05:07:18.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.625+0000 7fb34119e700 1 -- 172.21.15.98:0/59325433 learned_addr learned my addr 172.21.15.98:0/59325433 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:18.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.625+0000 7fb34099d700 1 --2- 172.21.15.98:0/59325433 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb33c0b2620 0x7fb33c0b3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:18.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.625+0000 7fb33bfff700 1 --2- 172.21.15.98:0/59325433 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb33c0b29f0 0x7fb33c14c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:18.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.625+0000 7fb34119e700 1 -- 172.21.15.98:0/59325433 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb33c0b29f0 msgr2=0x7fb33c14c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:18.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.625+0000 7fb34119e700 1 --2- 172.21.15.98:0/59325433 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb33c0b29f0 0x7fb33c14c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:18.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.625+0000 7fb34119e700 1 -- 172.21.15.98:0/59325433 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb33c0b2620 msgr2=0x7fb33c0b3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:18.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.625+0000 7fb34119e700 1 --2- 172.21.15.98:0/59325433 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb33c0b2620 0x7fb33c0b3ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:18.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.625+0000 7fb34119e700 1 -- 172.21.15.98:0/59325433 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb334009420 con 0x7fb33c0cf1b0 2023-10-06T05:07:18.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.625+0000 7fb34099d700 1 --2- 172.21.15.98:0/59325433 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb33c0b2620 0x7fb33c0b3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:07:18.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.625+0000 7fb34119e700 1 --2- 172.21.15.98:0/59325433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb33c0cf1b0 0x7fb33c0b4620 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fb3340093f0 tx=0x7fb334017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:18.638 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.626+0000 7fb338ff9700 1 -- 172.21.15.98:0/59325433 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb334020070 con 0x7fb33c0cf1b0 2023-10-06T05:07:18.638 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.626+0000 7fb338ff9700 1 -- 172.21.15.98:0/59325433 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb334011420 con 0x7fb33c0cf1b0 2023-10-06T05:07:18.638 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.626+0000 7fb34199f700 1 -- 172.21.15.98:0/59325433 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb33c0b3410 con 0x7fb33c0cf1b0 2023-10-06T05:07:18.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.626+0000 7fb338ff9700 1 -- 172.21.15.98:0/59325433 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3340117c0 con 0x7fb33c0cf1b0 2023-10-06T05:07:18.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.626+0000 7fb34199f700 1 -- 172.21.15.98:0/59325433 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb33c0c9a70 con 0x7fb33c0cf1b0 2023-10-06T05:07:18.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.626+0000 7fb34199f700 1 -- 172.21.15.98:0/59325433 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb33c0b3bd0 con 0x7fb33c0cf1b0 2023-10-06T05:07:18.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.627+0000 7fb338ff9700 1 -- 172.21.15.98:0/59325433 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fb334011bf0 con 0x7fb33c0cf1b0 2023-10-06T05:07:18.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.627+0000 7fb338ff9700 1 --2- 172.21.15.98:0/59325433 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fb328066f30 0x7fb3280693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:18.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.627+0000 7fb338ff9700 1 -- 172.21.15.98:0/59325433 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fb334086050 con 0x7fb33c0cf1b0 2023-10-06T05:07:18.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.627+0000 7fb34099d700 1 --2- 172.21.15.98:0/59325433 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fb328066f30 0x7fb3280693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:18.641 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.628+0000 7fb34099d700 1 --2- 172.21.15.98:0/59325433 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fb328066f30 0x7fb3280693e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fb32c009fd0 tx=0x7fb32c00b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:18.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.632+0000 7fb338ff9700 1 -- 172.21.15.98:0/59325433 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb334052e70 con 0x7fb33c0cf1b0 2023-10-06T05:07:18.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.815+0000 7fb34199f700 1 -- 172.21.15.98:0/59325433 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb33c05f230 con 0x7fb33c0cf1b0 2023-10-06T05:07:18.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.815+0000 7fb338ff9700 1 -- 172.21.15.98:0/59325433 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v29) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb334056490 con 0x7fb33c0cf1b0 2023-10-06T05:07:18.824 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2023-10-06T05:07:18.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.817+0000 7fb34199f700 1 -- 172.21.15.98:0/59325433 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fb328066f30 msgr2=0x7fb3280693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:18.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.817+0000 7fb34199f700 1 --2- 172.21.15.98:0/59325433 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fb328066f30 0x7fb3280693e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fb32c009fd0 tx=0x7fb32c00b040 comp rx=0 tx=0).stop 2023-10-06T05:07:18.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.817+0000 7fb34199f700 1 -- 172.21.15.98:0/59325433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb33c0cf1b0 msgr2=0x7fb33c0b4620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:18.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.817+0000 7fb34199f700 1 --2- 172.21.15.98:0/59325433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb33c0cf1b0 0x7fb33c0b4620 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fb3340093f0 tx=0x7fb334017000 comp rx=0 tx=0).stop 2023-10-06T05:07:18.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.818+0000 7fb34199f700 1 -- 172.21.15.98:0/59325433 shutdown_connections 2023-10-06T05:07:18.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.818+0000 7fb34199f700 1 --2- 172.21.15.98:0/59325433 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fb328066f30 0x7fb3280693e0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:18.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.818+0000 7fb34199f700 1 --2- 172.21.15.98:0/59325433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb33c0cf1b0 0x7fb33c0b4620 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:18.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.818+0000 7fb34199f700 1 --2- 172.21.15.98:0/59325433 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb33c0b2620 0x7fb33c0b3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:18.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.818+0000 7fb34199f700 1 --2- 172.21.15.98:0/59325433 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb33c0b29f0 0x7fb33c14c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:18.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.818+0000 7fb34199f700 1 -- 172.21.15.98:0/59325433 >> 172.21.15.98:0/59325433 conn(0x7fb33c099520 msgr2=0x7fb33c068bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:18.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.818+0000 7fb34199f700 1 -- 172.21.15.98:0/59325433 shutdown_connections 2023-10-06T05:07:18.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.818+0000 7fb34199f700 1 -- 172.21.15.98:0/59325433 wait complete. 2023-10-06T05:07:18.832 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-06T05:07:18.950 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.948+0000 7fa76fed6700 1 Processor -- start 2023-10-06T05:07:18.950 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.948+0000 7fa76fed6700 1 -- start start 2023-10-06T05:07:18.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.949+0000 7fa76fed6700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa7680c4450 0x7fa7680686c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:18.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.949+0000 7fa76fed6700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7680c9670 con 0x7fa7680cc740 2023-10-06T05:07:18.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.949+0000 7fa76fed6700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa768159c00 con 0x7fa7680b0820 2023-10-06T05:07:18.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.949+0000 7fa76fed6700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7680c8a60 con 0x7fa7680c4450 2023-10-06T05:07:18.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.949+0000 7fa76e473700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fa7680cc740 0x7fa7680ccba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:41292/0 (socket says 172.21.15.98:41292) 2023-10-06T05:07:18.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.949+0000 7fa76e473700 1 -- 172.21.15.98:0/45249863 learned_addr learned my addr 172.21.15.98:0/45249863 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:07:18.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.949+0000 7fa76d471700 1 --2- 172.21.15.98:0/45249863 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa7680c4450 0x7fa7680686c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:18.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.949+0000 7fa76cc70700 1 -- 172.21.15.98:0/45249863 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2039720860 0 0) 0x7fa7680c9670 con 0x7fa7680cc740 2023-10-06T05:07:18.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.949+0000 7fa76cc70700 1 -- 172.21.15.98:0/45249863 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa754003570 con 0x7fa7680cc740 2023-10-06T05:07:18.960 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.949+0000 7fa76cc70700 1 -- 172.21.15.98:0/45249863 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2163628534 0 0) 0x7fa754003570 con 0x7fa7680cc740 2023-10-06T05:07:18.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.949+0000 7fa76cc70700 1 -- 172.21.15.98:0/45249863 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa7680c4450 msgr2=0x7fa7680686c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:18.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.949+0000 7fa76cc70700 1 --2- 172.21.15.98:0/45249863 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa7680c4450 0x7fa7680686c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:18.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.949+0000 7fa76cc70700 1 -- 172.21.15.98:0/45249863 >> v1:172.21.15.174:6789/0 conn(0x7fa7680b0820 legacy=0x7fa7680cec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:18.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.949+0000 7fa76cc70700 1 -- 172.21.15.98:0/45249863 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7680c8d40 con 0x7fa7680cc740 2023-10-06T05:07:18.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.949+0000 7fa76cc70700 1 -- 172.21.15.98:0/45249863 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1543991876 0 0) 0x7fa760001470 con 0x7fa7680cc740 2023-10-06T05:07:18.961 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.949+0000 7fa76cc70700 1 -- 172.21.15.98:0/45249863 >> v1:172.21.15.98:6789/0 conn(0x7fa7680cc740 legacy=0x7fa7680ccba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:18.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.949+0000 7fa76cc70700 1 --2- 172.21.15.98:0/45249863 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa754003a90 0x7fa754003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:18.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.949+0000 7fa76cc70700 1 --2- 172.21.15.98:0/45249863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa754004810 0x7fa754004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:18.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.949+0000 7fa76cc70700 1 --2- 172.21.15.98:0/45249863 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa7540051c0 0x7fa75400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:18.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.949+0000 7fa76cc70700 1 -- 172.21.15.98:0/45249863 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7680c8a60 con 0x7fa754004810 2023-10-06T05:07:18.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.949+0000 7fa76cc70700 1 -- 172.21.15.98:0/45249863 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa75400c000 con 0x7fa754003a90 2023-10-06T05:07:18.962 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.949+0000 7fa76cc70700 1 -- 172.21.15.98:0/45249863 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa75400c1c0 con 0x7fa7540051c0 2023-10-06T05:07:18.963 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.949+0000 7fa76d471700 1 --2- 172.21.15.98:0/45249863 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa7680c4450 0x7fa7680686c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T05:07:18.963 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.949+0000 7fa76d471700 1 --2- 172.21.15.98:0/45249863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa754004810 0x7fa754004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:18.963 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.950+0000 7fa76dc72700 1 --2- 172.21.15.98:0/45249863 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa754003a90 0x7fa754003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:18.963 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.950+0000 7fa76e473700 1 --2- 172.21.15.98:0/45249863 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa7540051c0 0x7fa75400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:18.963 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.950+0000 7fa76d471700 1 -- 172.21.15.98:0/45249863 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa7540051c0 msgr2=0x7fa75400ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:18.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.950+0000 7fa76d471700 1 --2- 172.21.15.98:0/45249863 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa7540051c0 0x7fa75400ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:18.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.950+0000 7fa76d471700 1 -- 172.21.15.98:0/45249863 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa754003a90 msgr2=0x7fa754003e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:18.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.950+0000 7fa76d471700 1 --2- 172.21.15.98:0/45249863 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa754003a90 0x7fa754003e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:18.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.950+0000 7fa76d471700 1 -- 172.21.15.98:0/45249863 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa754003570 con 0x7fa754004810 2023-10-06T05:07:18.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.950+0000 7fa76d471700 1 --2- 172.21.15.98:0/45249863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa754004810 0x7fa754004c80 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa76806ed70 tx=0x7fa7580091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=675ec9fa62c34430 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:18.964 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.950+0000 7fa76cc70700 1 -- 172.21.15.98:0/45249863 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa75800e610 con 0x7fa754004810 2023-10-06T05:07:18.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.950+0000 7fa76cc70700 1 -- 172.21.15.98:0/45249863 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa75800e770 con 0x7fa754004810 2023-10-06T05:07:18.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.950+0000 7fa76cc70700 1 -- 172.21.15.98:0/45249863 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa75800eb10 con 0x7fa754004810 2023-10-06T05:07:18.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.950+0000 7fa76fed6700 1 -- 172.21.15.98:0/45249863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa754004810 msgr2=0x7fa754004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:18.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.950+0000 7fa76fed6700 1 --2- 172.21.15.98:0/45249863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa754004810 0x7fa754004c80 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa76806ed70 tx=0x7fa7580091a0 comp rx=0 tx=0).stop 2023-10-06T05:07:18.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.950+0000 7fa76e473700 1 -- 172.21.15.98:0/45249863 reap_dead start 2023-10-06T05:07:18.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.951+0000 7fa76fed6700 1 -- 172.21.15.98:0/45249863 shutdown_connections 2023-10-06T05:07:18.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.951+0000 7fa76fed6700 1 -- 172.21.15.98:0/45249863 >> 172.21.15.98:0/45249863 conn(0x7fa768087fc0 msgr2=0x7fa7680ce310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:18.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.951+0000 7fa76fed6700 1 -- 172.21.15.98:0/45249863 shutdown_connections 2023-10-06T05:07:18.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.951+0000 7fa76fed6700 1 -- 172.21.15.98:0/45249863 wait complete. 2023-10-06T05:07:18.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.952+0000 7fa76fed6700 1 Processor -- start 2023-10-06T05:07:18.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.952+0000 7fa76fed6700 1 -- start start 2023-10-06T05:07:18.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.952+0000 7fa76fed6700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa754004810 0x7fa768157750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:18.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.952+0000 7fa76fed6700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa768169390 0x7fa768169800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:18.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.952+0000 7fa76fed6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa768068e40 0x7fa7680692e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:18.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.952+0000 7fa76fed6700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa76816a5c0 con 0x7fa768068e40 2023-10-06T05:07:18.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.952+0000 7fa76fed6700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa768069820 con 0x7fa754004810 2023-10-06T05:07:18.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.952+0000 7fa76fed6700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa76816ab10 con 0x7fa768169390 2023-10-06T05:07:18.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.952+0000 7fa76e473700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa768068e40 0x7fa7680692e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:18.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.953+0000 7fa76e473700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa768068e40 0x7fa7680692e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46136/0 (socket says 172.21.15.98:46136) 2023-10-06T05:07:18.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.953+0000 7fa76e473700 1 -- 172.21.15.98:0/3817719507 learned_addr learned my addr 172.21.15.98:0/3817719507 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:18.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.953+0000 7fa76dc72700 1 --2- 172.21.15.98:0/3817719507 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa754004810 0x7fa768157750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:18.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.953+0000 7fa76d471700 1 --2- 172.21.15.98:0/3817719507 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa768169390 0x7fa768169800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:18.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.953+0000 7fa76e473700 1 -- 172.21.15.98:0/3817719507 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa768169390 msgr2=0x7fa768169800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:18.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.953+0000 7fa76e473700 1 --2- 172.21.15.98:0/3817719507 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa768169390 0x7fa768169800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:18.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.953+0000 7fa76e473700 1 -- 172.21.15.98:0/3817719507 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa754004810 msgr2=0x7fa768157750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:18.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.953+0000 7fa76e473700 1 --2- 172.21.15.98:0/3817719507 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa754004810 0x7fa768157750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:18.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.953+0000 7fa76e473700 1 -- 172.21.15.98:0/3817719507 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa768157c90 con 0x7fa768068e40 2023-10-06T05:07:18.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.953+0000 7fa76d471700 1 --2- 172.21.15.98:0/3817719507 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa768169390 0x7fa768169800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:07:18.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.953+0000 7fa76e473700 1 --2- 172.21.15.98:0/3817719507 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa768068e40 0x7fa7680692e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa76815d240 tx=0x7fa760001da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:18.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.953+0000 7fa75dffb700 1 -- 172.21.15.98:0/3817719507 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa76000f8d0 con 0x7fa768068e40 2023-10-06T05:07:18.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.953+0000 7fa75dffb700 1 -- 172.21.15.98:0/3817719507 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa76000fa30 con 0x7fa768068e40 2023-10-06T05:07:18.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.953+0000 7fa76fed6700 1 -- 172.21.15.98:0/3817719507 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa768169ee0 con 0x7fa768068e40 2023-10-06T05:07:18.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.953+0000 7fa75dffb700 1 -- 172.21.15.98:0/3817719507 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa76000fdf0 con 0x7fa768068e40 2023-10-06T05:07:18.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.953+0000 7fa76fed6700 1 -- 172.21.15.98:0/3817719507 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa76816a7a0 con 0x7fa768068e40 2023-10-06T05:07:18.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.954+0000 7fa76fed6700 1 -- 172.21.15.98:0/3817719507 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa76816a400 con 0x7fa768068e40 2023-10-06T05:07:18.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.954+0000 7fa75dffb700 1 -- 172.21.15.98:0/3817719507 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fa7600135c0 con 0x7fa768068e40 2023-10-06T05:07:18.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.955+0000 7fa75dffb700 1 --2- 172.21.15.98:0/3817719507 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fa74c066f30 0x7fa74c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:18.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.955+0000 7fa75dffb700 1 -- 172.21.15.98:0/3817719507 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fa760084310 con 0x7fa768068e40 2023-10-06T05:07:18.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.955+0000 7fa76dc72700 1 --2- 172.21.15.98:0/3817719507 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fa74c066f30 0x7fa74c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:18.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.955+0000 7fa76dc72700 1 --2- 172.21.15.98:0/3817719507 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fa74c066f30 0x7fa74c0693e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa764001270 tx=0x7fa764001b10 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:18.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:18.959+0000 7fa75dffb700 1 -- 172.21.15.98:0/3817719507 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa76816a400 con 0x7fa768068e40 2023-10-06T05:07:19.147 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.143+0000 7fa76fed6700 1 -- 172.21.15.98:0/3817719507 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fa76816a400 con 0x7fa768068e40 2023-10-06T05:07:19.148 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.143+0000 7fa75dffb700 1 -- 172.21.15.98:0/3817719507 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v29) v1 ==== 103+0+11 (secure 0 0 0) 0x7fa76816a400 con 0x7fa768068e40 2023-10-06T05:07:19.148 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2023-10-06T05:07:19.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.145+0000 7fa76fed6700 1 -- 172.21.15.98:0/3817719507 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fa74c066f30 msgr2=0x7fa74c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:19.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.145+0000 7fa76fed6700 1 --2- 172.21.15.98:0/3817719507 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fa74c066f30 0x7fa74c0693e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa764001270 tx=0x7fa764001b10 comp rx=0 tx=0).stop 2023-10-06T05:07:19.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.145+0000 7fa76fed6700 1 -- 172.21.15.98:0/3817719507 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa768068e40 msgr2=0x7fa7680692e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:19.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.145+0000 7fa76fed6700 1 --2- 172.21.15.98:0/3817719507 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa768068e40 0x7fa7680692e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa76815d240 tx=0x7fa760001da0 comp rx=0 tx=0).stop 2023-10-06T05:07:19.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.145+0000 7fa76fed6700 1 -- 172.21.15.98:0/3817719507 shutdown_connections 2023-10-06T05:07:19.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.145+0000 7fa76fed6700 1 --2- 172.21.15.98:0/3817719507 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fa74c066f30 0x7fa74c0693e0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:19.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.145+0000 7fa76fed6700 1 --2- 172.21.15.98:0/3817719507 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa768068e40 0x7fa7680692e0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:19.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.145+0000 7fa76fed6700 1 --2- 172.21.15.98:0/3817719507 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa754004810 0x7fa768157750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:19.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.145+0000 7fa76fed6700 1 --2- 172.21.15.98:0/3817719507 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa768169390 0x7fa768169800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:19.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.145+0000 7fa76fed6700 1 -- 172.21.15.98:0/3817719507 >> 172.21.15.98:0/3817719507 conn(0x7fa768087fc0 msgr2=0x7fa7680c30b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:19.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.145+0000 7fa76fed6700 1 -- 172.21.15.98:0/3817719507 shutdown_connections 2023-10-06T05:07:19.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.145+0000 7fa76fed6700 1 -- 172.21.15.98:0/3817719507 wait complete. 2023-10-06T05:07:19.162 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-06T05:07:19.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.279+0000 7f1ef3146700 1 Processor -- start 2023-10-06T05:07:19.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.279+0000 7f1ef3146700 1 -- start start 2023-10-06T05:07:19.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.280+0000 7f1ef3146700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1eec159560 0x7f1eec0aa530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:19.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.280+0000 7f1ef3146700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eec0b0c60 con 0x7f1eec0b49a0 2023-10-06T05:07:19.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.280+0000 7f1ef3146700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eec0b0f40 con 0x7f1eec0b0760 2023-10-06T05:07:19.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.280+0000 7f1ef3146700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1eec0b1220 con 0x7f1eec159560 2023-10-06T05:07:19.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.280+0000 7f1ef0ee2700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1eec159560 0x7f1eec0aa530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:19.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.280+0000 7f1eebfff700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f1eec0b49a0 0x7f1eec0aaa70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:60868/0 (socket says 172.21.15.98:60868) 2023-10-06T05:07:19.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.280+0000 7f1eebfff700 1 -- 172.21.15.98:0/2006956974 learned_addr learned my addr 172.21.15.98:0/2006956974 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:07:19.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.281+0000 7f1ef0ee2700 1 -- 172.21.15.98:0/2006956974 >> v1:172.21.15.174:6789/0 conn(0x7f1eec0b0760 legacy=0x7f1eec0b1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:19.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.281+0000 7f1ef0ee2700 1 -- 172.21.15.98:0/2006956974 >> v1:172.21.15.98:6789/0 conn(0x7f1eec0b49a0 legacy=0x7f1eec0aaa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:19.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.281+0000 7f1eeb7fe700 1 -- 172.21.15.98:0/2006956974 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 381914954 0 0) 0x7f1eec0b0c60 con 0x7f1eec0b49a0 2023-10-06T05:07:19.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.281+0000 7f1ef0ee2700 1 -- 172.21.15.98:0/2006956974 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eec0b1500 con 0x7f1eec159560 2023-10-06T05:07:19.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.281+0000 7f1ef0ee2700 1 --2- 172.21.15.98:0/2006956974 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1eec159560 0x7f1eec0aa530 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f1ee0009900 tx=0x7f1ee0009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7315a9c55a24e1f server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:19.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.281+0000 7f1eeb7fe700 1 -- 172.21.15.98:0/2006956974 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ee0009ea0 con 0x7f1eec159560 2023-10-06T05:07:19.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.281+0000 7f1eeb7fe700 1 -- 172.21.15.98:0/2006956974 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ee0005c10 con 0x7f1eec159560 2023-10-06T05:07:19.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.281+0000 7f1eeb7fe700 1 -- 172.21.15.98:0/2006956974 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ee000b5d0 con 0x7f1eec159560 2023-10-06T05:07:19.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.281+0000 7f1ef3146700 1 -- 172.21.15.98:0/2006956974 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1eec159560 msgr2=0x7f1eec0aa530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:19.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.281+0000 7f1ef3146700 1 --2- 172.21.15.98:0/2006956974 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1eec159560 0x7f1eec0aa530 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f1ee0009900 tx=0x7f1ee0009120 comp rx=0 tx=0).stop 2023-10-06T05:07:19.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.281+0000 7f1ef3146700 1 -- 172.21.15.98:0/2006956974 shutdown_connections 2023-10-06T05:07:19.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.281+0000 7f1ef3146700 1 --2- 172.21.15.98:0/2006956974 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1eec159560 0x7f1eec0aa530 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:19.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.281+0000 7f1ef3146700 1 -- 172.21.15.98:0/2006956974 >> 172.21.15.98:0/2006956974 conn(0x7f1eec0997f0 msgr2=0x7f1eec06f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:19.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.281+0000 7f1ef3146700 1 -- 172.21.15.98:0/2006956974 shutdown_connections 2023-10-06T05:07:19.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.282+0000 7f1ef3146700 1 -- 172.21.15.98:0/2006956974 wait complete. 2023-10-06T05:07:19.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.283+0000 7f1ef3146700 1 Processor -- start 2023-10-06T05:07:19.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.283+0000 7f1ef3146700 1 -- start start 2023-10-06T05:07:19.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.283+0000 7f1ef3146700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1eec0b0760 0x7f1eec0ad590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:19.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.283+0000 7f1ef3146700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1eec0b49a0 0x7f1eec157540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:19.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.283+0000 7f1ef3146700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1eec159560 0x7f1eec0af010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:19.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.283+0000 7f1ef3146700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eec0ae9c0 con 0x7f1eec0b49a0 2023-10-06T05:07:19.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.283+0000 7f1ef3146700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eec0adad0 con 0x7f1eec0b0760 2023-10-06T05:07:19.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.284+0000 7f1ef3146700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eec16ac60 con 0x7f1eec159560 2023-10-06T05:07:19.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.284+0000 7f1eebfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1eec0b49a0 0x7f1eec157540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:19.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.284+0000 7f1eebfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1eec0b49a0 0x7f1eec157540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46876/0 (socket says 172.21.15.98:46876) 2023-10-06T05:07:19.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.284+0000 7f1eebfff700 1 -- 172.21.15.98:0/1463728450 learned_addr learned my addr 172.21.15.98:0/1463728450 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:19.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.284+0000 7f1ef0ee2700 1 --2- 172.21.15.98:0/1463728450 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1eec0b0760 0x7f1eec0ad590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:19.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.284+0000 7f1ef16e3700 1 --2- 172.21.15.98:0/1463728450 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1eec159560 0x7f1eec0af010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:19.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.284+0000 7f1eebfff700 1 -- 172.21.15.98:0/1463728450 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1eec159560 msgr2=0x7f1eec0af010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:19.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.284+0000 7f1eebfff700 1 --2- 172.21.15.98:0/1463728450 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1eec159560 0x7f1eec0af010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:19.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.284+0000 7f1eebfff700 1 -- 172.21.15.98:0/1463728450 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1eec0b0760 msgr2=0x7f1eec0ad590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:19.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.284+0000 7f1eebfff700 1 --2- 172.21.15.98:0/1463728450 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1eec0b0760 0x7f1eec0ad590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:19.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.284+0000 7f1eebfff700 1 -- 172.21.15.98:0/1463728450 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ee00093c0 con 0x7f1eec0b49a0 2023-10-06T05:07:19.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.284+0000 7f1ef0ee2700 1 --2- 172.21.15.98:0/1463728450 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1eec0b0760 0x7f1eec0ad590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:07:19.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.284+0000 7f1eebfff700 1 --2- 172.21.15.98:0/1463728450 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1eec0b49a0 0x7f1eec157540 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f1ed800c7e0 tx=0x7f1ed8007d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:19.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.284+0000 7f1ee8ff9700 1 -- 172.21.15.98:0/1463728450 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ed8005c00 con 0x7f1eec0b49a0 2023-10-06T05:07:19.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.284+0000 7f1ee8ff9700 1 -- 172.21.15.98:0/1463728450 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ed8005d60 con 0x7f1eec0b49a0 2023-10-06T05:07:19.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.284+0000 7f1ee8ff9700 1 -- 172.21.15.98:0/1463728450 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ed80072e0 con 0x7f1eec0b49a0 2023-10-06T05:07:19.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.284+0000 7f1ef3146700 1 -- 172.21.15.98:0/1463728450 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1eec16a050 con 0x7f1eec0b49a0 2023-10-06T05:07:19.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.284+0000 7f1ef3146700 1 -- 172.21.15.98:0/1463728450 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1eec16a8f0 con 0x7f1eec0b49a0 2023-10-06T05:07:19.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.285+0000 7f1ef3146700 1 -- 172.21.15.98:0/1463728450 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1eec16a450 con 0x7f1eec0b49a0 2023-10-06T05:07:19.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.286+0000 7f1ee8ff9700 1 -- 172.21.15.98:0/1463728450 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f1ed801b3b0 con 0x7f1eec0b49a0 2023-10-06T05:07:19.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.286+0000 7f1ee8ff9700 1 --2- 172.21.15.98:0/1463728450 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f1ed0066f30 0x7f1ed00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:19.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.286+0000 7f1ef0ee2700 1 --2- 172.21.15.98:0/1463728450 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f1ed0066f30 0x7f1ed00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:19.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.286+0000 7f1ee8ff9700 1 -- 172.21.15.98:0/1463728450 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f1ed8083dd0 con 0x7f1eec0b49a0 2023-10-06T05:07:19.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.286+0000 7f1ef0ee2700 1 --2- 172.21.15.98:0/1463728450 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f1ed0066f30 0x7f1ed00693e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f1ee0003080 tx=0x7f1ee0015000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:19.293 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.290+0000 7f1ee8ff9700 1 -- 172.21.15.98:0/1463728450 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1ed804b5a0 con 0x7f1eec0b49a0 2023-10-06T05:07:19.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.516+0000 7f1ef3146700 1 -- 172.21.15.98:0/1463728450 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1eec064a90 con 0x7f1eec0b49a0 2023-10-06T05:07:19.519 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.516+0000 7f1ee8ff9700 1 -- 172.21.15.98:0/1463728450 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4596 (secure 0 0 0) 0x7f1ed8017090 con 0x7f1eec0b49a0 2023-10-06T05:07:19.519 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:07:19.519 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4285,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6834/925250632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":925250632},{"type":"v1","addr":"172.21.15.174:6834","nonce":925250632}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4288,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/1629449810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6833","nonce":1629449810},{"type":"v1","addr":"172.21.15.174:6835","nonce":1629449810}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4299,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/3961327990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3961327990},{"type":"v1","addr":"172.21.15.98:6836","nonce":3961327990}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-06T05:06:57.086683+0000","modified":"2023-10-06T05:07:09.773654+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.98:6837/2683239094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":2683239094},{"type":"v1","addr":"172.21.15.98:6837","nonce":2683239094}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-06T05:07:19.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.518+0000 7f1ef3146700 1 -- 172.21.15.98:0/1463728450 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f1ed0066f30 msgr2=0x7f1ed00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:19.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.518+0000 7f1ef3146700 1 --2- 172.21.15.98:0/1463728450 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f1ed0066f30 0x7f1ed00693e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f1ee0003080 tx=0x7f1ee0015000 comp rx=0 tx=0).stop 2023-10-06T05:07:19.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.518+0000 7f1ef3146700 1 -- 172.21.15.98:0/1463728450 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1eec0b49a0 msgr2=0x7f1eec157540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:19.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.518+0000 7f1ef3146700 1 --2- 172.21.15.98:0/1463728450 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1eec0b49a0 0x7f1eec157540 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f1ed800c7e0 tx=0x7f1ed8007d90 comp rx=0 tx=0).stop 2023-10-06T05:07:19.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.518+0000 7f1ef3146700 1 -- 172.21.15.98:0/1463728450 shutdown_connections 2023-10-06T05:07:19.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.518+0000 7f1ef3146700 1 --2- 172.21.15.98:0/1463728450 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1eec159560 0x7f1eec0af010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:19.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.518+0000 7f1ef3146700 1 --2- 172.21.15.98:0/1463728450 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1eec0b0760 0x7f1eec0ad590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:19.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.518+0000 7f1ef3146700 1 --2- 172.21.15.98:0/1463728450 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f1ed0066f30 0x7f1ed00693e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:19.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.518+0000 7f1ef3146700 1 --2- 172.21.15.98:0/1463728450 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1eec0b49a0 0x7f1eec157540 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:19.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.518+0000 7f1ef3146700 1 -- 172.21.15.98:0/1463728450 >> 172.21.15.98:0/1463728450 conn(0x7f1eec0997f0 msgr2=0x7f1eec1588d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:19.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.518+0000 7f1ef3146700 1 -- 172.21.15.98:0/1463728450 shutdown_connections 2023-10-06T05:07:19.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.518+0000 7f1ef3146700 1 -- 172.21.15.98:0/1463728450 wait complete. 2023-10-06T05:07:19.522 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 8 2023-10-06T05:07:19.533 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T05:07:19.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.646+0000 7fc91ab99700 1 Processor -- start 2023-10-06T05:07:19.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.646+0000 7fc91ab99700 1 -- start start 2023-10-06T05:07:19.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.647+0000 7fc91ab99700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc914159470 0x7fc9140ccc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:19.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.647+0000 7fc91ab99700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9140cd5d0 con 0x7fc9140cd160 2023-10-06T05:07:19.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.647+0000 7fc91ab99700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9140cc800 con 0x7fc9140c6220 2023-10-06T05:07:19.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.647+0000 7fc91ab99700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9140cbc30 con 0x7fc914159470 2023-10-06T05:07:19.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.647+0000 7fc919b97700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc914159470 0x7fc9140ccc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:19.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.647+0000 7fc919396700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fc9140cd160 0x7fc9140ce980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:60878/0 (socket says 172.21.15.98:60878) 2023-10-06T05:07:19.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.647+0000 7fc919396700 1 -- 172.21.15.98:0/2168480723 learned_addr learned my addr 172.21.15.98:0/2168480723 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:07:19.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.647+0000 7fc919b97700 1 -- 172.21.15.98:0/2168480723 >> v1:172.21.15.174:6789/0 conn(0x7fc9140c6220 legacy=0x7fc914163030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:19.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.647+0000 7fc919b97700 1 -- 172.21.15.98:0/2168480723 >> v1:172.21.15.98:6789/0 conn(0x7fc9140cd160 legacy=0x7fc9140ce980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:19.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.647+0000 7fc919b97700 1 -- 172.21.15.98:0/2168480723 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9140cbed0 con 0x7fc914159470 2023-10-06T05:07:19.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.648+0000 7fc919b97700 1 --2- 172.21.15.98:0/2168480723 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc914159470 0x7fc9140ccc20 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fc9100098d0 tx=0x7fc910009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b356a5832dc0a603 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:19.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.648+0000 7fc918b95700 1 -- 172.21.15.98:0/2168480723 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc910009e70 con 0x7fc914159470 2023-10-06T05:07:19.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.648+0000 7fc918b95700 1 -- 172.21.15.98:0/2168480723 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc910005c10 con 0x7fc914159470 2023-10-06T05:07:19.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.648+0000 7fc918b95700 1 -- 172.21.15.98:0/2168480723 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc91000b610 con 0x7fc914159470 2023-10-06T05:07:19.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.648+0000 7fc91ab99700 1 -- 172.21.15.98:0/2168480723 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc914159470 msgr2=0x7fc9140ccc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:19.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.648+0000 7fc91ab99700 1 --2- 172.21.15.98:0/2168480723 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc914159470 0x7fc9140ccc20 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fc9100098d0 tx=0x7fc910009120 comp rx=0 tx=0).stop 2023-10-06T05:07:19.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.648+0000 7fc91ab99700 1 -- 172.21.15.98:0/2168480723 shutdown_connections 2023-10-06T05:07:19.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.648+0000 7fc91ab99700 1 --2- 172.21.15.98:0/2168480723 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc914159470 0x7fc9140ccc20 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:19.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.648+0000 7fc91ab99700 1 -- 172.21.15.98:0/2168480723 >> 172.21.15.98:0/2168480723 conn(0x7fc9140995c0 msgr2=0x7fc91406f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:19.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.648+0000 7fc91ab99700 1 -- 172.21.15.98:0/2168480723 shutdown_connections 2023-10-06T05:07:19.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.648+0000 7fc91ab99700 1 -- 172.21.15.98:0/2168480723 wait complete. 2023-10-06T05:07:19.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.649+0000 7fc91ab99700 1 Processor -- start 2023-10-06T05:07:19.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.649+0000 7fc91ab99700 1 -- start start 2023-10-06T05:07:19.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.650+0000 7fc91ab99700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9140c6220 0x7fc914153070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:19.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.650+0000 7fc91ab99700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc9140cd160 0x7fc9140c8e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:19.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.650+0000 7fc91ab99700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc914159470 0x7fc9140cdeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:19.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.650+0000 7fc91ab99700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9140c7ba0 con 0x7fc9140c6220 2023-10-06T05:07:19.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.650+0000 7fc91ab99700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc91416a910 con 0x7fc914159470 2023-10-06T05:07:19.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.650+0000 7fc91ab99700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9140c7550 con 0x7fc9140cd160 2023-10-06T05:07:19.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.650+0000 7fc919b97700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9140c6220 0x7fc914153070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:19.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.650+0000 7fc919b97700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9140c6220 0x7fc914153070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46900/0 (socket says 172.21.15.98:46900) 2023-10-06T05:07:19.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.650+0000 7fc919b97700 1 -- 172.21.15.98:0/2334092135 learned_addr learned my addr 172.21.15.98:0/2334092135 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:19.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.650+0000 7fc919396700 1 --2- 172.21.15.98:0/2334092135 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc9140cd160 0x7fc9140c8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:19.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.650+0000 7fc919b97700 1 -- 172.21.15.98:0/2334092135 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc9140cd160 msgr2=0x7fc9140c8e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:19.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.650+0000 7fc91a398700 1 --2- 172.21.15.98:0/2334092135 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc914159470 0x7fc9140cdeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:19.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.650+0000 7fc919b97700 1 --2- 172.21.15.98:0/2334092135 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc9140cd160 0x7fc9140c8e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:19.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.650+0000 7fc919b97700 1 -- 172.21.15.98:0/2334092135 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc914159470 msgr2=0x7fc9140cdeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:19.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.650+0000 7fc919b97700 1 --2- 172.21.15.98:0/2334092135 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc914159470 0x7fc9140cdeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:19.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.650+0000 7fc919b97700 1 -- 172.21.15.98:0/2334092135 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9100093c0 con 0x7fc9140c6220 2023-10-06T05:07:19.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.650+0000 7fc919396700 1 --2- 172.21.15.98:0/2334092135 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc9140cd160 0x7fc9140c8e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:07:19.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.650+0000 7fc91a398700 1 --2- 172.21.15.98:0/2334092135 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc914159470 0x7fc9140cdeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:07:19.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.650+0000 7fc919b97700 1 --2- 172.21.15.98:0/2334092135 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9140c6220 0x7fc914153070 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fc910002000 tx=0x7fc910015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:19.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.651+0000 7fc909ffb700 1 -- 172.21.15.98:0/2334092135 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc910011c40 con 0x7fc9140c6220 2023-10-06T05:07:19.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.651+0000 7fc909ffb700 1 -- 172.21.15.98:0/2334092135 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc910011dc0 con 0x7fc9140c6220 2023-10-06T05:07:19.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.651+0000 7fc91ab99700 1 -- 172.21.15.98:0/2334092135 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9140c7830 con 0x7fc9140c6220 2023-10-06T05:07:19.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.651+0000 7fc909ffb700 1 -- 172.21.15.98:0/2334092135 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc91001d680 con 0x7fc9140c6220 2023-10-06T05:07:19.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.651+0000 7fc91ab99700 1 -- 172.21.15.98:0/2334092135 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9140c93b0 con 0x7fc9140c6220 2023-10-06T05:07:19.659 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.652+0000 7fc91ab99700 1 -- 172.21.15.98:0/2334092135 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9140c7390 con 0x7fc9140c6220 2023-10-06T05:07:19.659 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.652+0000 7fc909ffb700 1 -- 172.21.15.98:0/2334092135 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fc91001da70 con 0x7fc9140c6220 2023-10-06T05:07:19.659 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.653+0000 7fc909ffb700 1 --2- 172.21.15.98:0/2334092135 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fc8f8066f30 0x7fc8f80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:19.659 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.653+0000 7fc919396700 1 --2- 172.21.15.98:0/2334092135 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fc8f8066f30 0x7fc8f80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:19.660 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.653+0000 7fc909ffb700 1 -- 172.21.15.98:0/2334092135 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fc9100867b0 con 0x7fc9140c6220 2023-10-06T05:07:19.660 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.653+0000 7fc919396700 1 --2- 172.21.15.98:0/2334092135 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fc8f8066f30 0x7fc8f80693e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fc91415bf20 tx=0x7fc904009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:19.660 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.657+0000 7fc909ffb700 1 -- 172.21.15.98:0/2334092135 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc9100535d0 con 0x7fc9140c6220 2023-10-06T05:07:19.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.843+0000 7fc91ab99700 1 -- 172.21.15.98:0/2334092135 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc9140c79c0 con 0x7fc9140c6220 2023-10-06T05:07:19.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.844+0000 7fc909ffb700 1 -- 172.21.15.98:0/2334092135 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+18707 (secure 0 0 0) 0x7fc910056bf0 con 0x7fc9140c6220 2023-10-06T05:07:19.849 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:07:19.849 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":29,"fsid":"f50c5df5-cad5-47a4-9bb1-8c791d4311ec","created":"2023-10-06T05:06:32.971175+0000","modified":"2023-10-06T05:07:02.131672+0000","last_up_change":"2023-10-06T05:06:44.016489+0000","last_in_change":"2023-10-06T05:06:35.544511+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-06T05:06:44.559671+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-06T05:06:51.432161+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-10-06T05:06:55.589057+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-10-06T05:06:56.041687+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-10-06T05:06:59.754099+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d9495d95-da75-4c3b-8927-e9bae6a8b093","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":90627},{"type":"v1","addr":"172.21.15.98:6801","nonce":90627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":90627},{"type":"v1","addr":"172.21.15.98:6803","nonce":90627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":90627},{"type":"v1","addr":"172.21.15.98:6807","nonce":90627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":90627},{"type":"v1","addr":"172.21.15.98:6805","nonce":90627}]},"public_addr":"172.21.15.98:6801/90627","cluster_addr":"172.21.15.98:6803/90627","heartbeat_back_addr":"172.21.15.98:6807/90627","heartbeat_front_addr":"172.21.15.98:6805/90627","state":["exists","up"]},{"osd":1,"uuid":"f25ba5be-d65e-4707-a134-154092deb754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":90628},{"type":"v1","addr":"172.21.15.98:6809","nonce":90628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":90628},{"type":"v1","addr":"172.21.15.98:6811","nonce":90628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":90628},{"type":"v1","addr":"172.21.15.98:6815","nonce":90628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":90628},{"type":"v1","addr":"172.21.15.98:6813","nonce":90628}]},"public_addr":"172.21.15.98:6809/90628","cluster_addr":"172.21.15.98:6811/90628","heartbeat_back_addr":"172.21.15.98:6815/90628","heartbeat_front_addr":"172.21.15.98:6813/90628","state":["exists","up"]},{"osd":2,"uuid":"5ab9e028-945c-43e2-8c95-3b0b4cf38e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":90662},{"type":"v1","addr":"172.21.15.98:6818","nonce":90662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":90662},{"type":"v1","addr":"172.21.15.98:6822","nonce":90662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":90662},{"type":"v1","addr":"172.21.15.98:6830","nonce":90662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":90662},{"type":"v1","addr":"172.21.15.98:6826","nonce":90662}]},"public_addr":"172.21.15.98:6818/90662","cluster_addr":"172.21.15.98:6822/90662","heartbeat_back_addr":"172.21.15.98:6830/90662","heartbeat_front_addr":"172.21.15.98:6826/90662","state":["exists","up"]},{"osd":3,"uuid":"3bb803ff-1738-46bf-be62-2267bf378822","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6817","nonce":90663},{"type":"v1","addr":"172.21.15.98:6819","nonce":90663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6821","nonce":90663},{"type":"v1","addr":"172.21.15.98:6823","nonce":90663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6829","nonce":90663},{"type":"v1","addr":"172.21.15.98:6831","nonce":90663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6825","nonce":90663},{"type":"v1","addr":"172.21.15.98:6827","nonce":90663}]},"public_addr":"172.21.15.98:6819/90663","cluster_addr":"172.21.15.98:6823/90663","heartbeat_back_addr":"172.21.15.98:6831/90663","heartbeat_front_addr":"172.21.15.98:6827/90663","state":["exists","up"]},{"osd":4,"uuid":"7d57a4c9-6462-43f6-b64a-7f9e9b4353d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":90847},{"type":"v1","addr":"172.21.15.174:6801","nonce":90847}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":90847},{"type":"v1","addr":"172.21.15.174:6803","nonce":90847}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":90847},{"type":"v1","addr":"172.21.15.174:6807","nonce":90847}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":90847},{"type":"v1","addr":"172.21.15.174:6805","nonce":90847}]},"public_addr":"172.21.15.174:6801/90847","cluster_addr":"172.21.15.174:6803/90847","heartbeat_back_addr":"172.21.15.174:6807/90847","heartbeat_front_addr":"172.21.15.174:6805/90847","state":["exists","up"]},{"osd":5,"uuid":"83287ea4-e44d-4af9-bc77-36d47c60d4a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":90848},{"type":"v1","addr":"172.21.15.174:6809","nonce":90848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":90848},{"type":"v1","addr":"172.21.15.174:6811","nonce":90848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":90848},{"type":"v1","addr":"172.21.15.174:6815","nonce":90848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":90848},{"type":"v1","addr":"172.21.15.174:6813","nonce":90848}]},"public_addr":"172.21.15.174:6809/90848","cluster_addr":"172.21.15.174:6811/90848","heartbeat_back_addr":"172.21.15.174:6815/90848","heartbeat_front_addr":"172.21.15.174:6813/90848","state":["exists","up"]},{"osd":6,"uuid":"496a6272-892e-43c4-a145-206623c22f99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":90877},{"type":"v1","addr":"172.21.15.174:6817","nonce":90877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":90877},{"type":"v1","addr":"172.21.15.174:6819","nonce":90877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":90877},{"type":"v1","addr":"172.21.15.174:6823","nonce":90877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":90877},{"type":"v1","addr":"172.21.15.174:6821","nonce":90877}]},"public_addr":"172.21.15.174:6817/90877","cluster_addr":"172.21.15.174:6819/90877","heartbeat_back_addr":"172.21.15.174:6823/90877","heartbeat_front_addr":"172.21.15.174:6821/90877","state":["exists","up"]},{"osd":7,"uuid":"49967e34-8b07-4bab-a13e-0c16d6a0f3a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":90878},{"type":"v1","addr":"172.21.15.174:6825","nonce":90878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":90878},{"type":"v1","addr":"172.21.15.174:6827","nonce":90878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":90878},{"type":"v1","addr":"172.21.15.174:6831","nonce":90878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":90878},{"type":"v1","addr":"172.21.15.174:6829","nonce":90878}]},"public_addr":"172.21.15.174:6825/90878","cluster_addr":"172.21.15.174:6827/90878","heartbeat_back_addr":"172.21.15.174:6831/90878","heartbeat_front_addr":"172.21.15.174:6829/90878","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.446479+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.514266+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.794108+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.810894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.731919+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.717756+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.737435+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.508682+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T05:07:19.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.846+0000 7fc91ab99700 1 -- 172.21.15.98:0/2334092135 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fc8f8066f30 msgr2=0x7fc8f80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:19.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.846+0000 7fc91ab99700 1 --2- 172.21.15.98:0/2334092135 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fc8f8066f30 0x7fc8f80693e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fc91415bf20 tx=0x7fc904009040 comp rx=0 tx=0).stop 2023-10-06T05:07:19.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.846+0000 7fc91ab99700 1 -- 172.21.15.98:0/2334092135 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9140c6220 msgr2=0x7fc914153070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:19.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.846+0000 7fc91ab99700 1 --2- 172.21.15.98:0/2334092135 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9140c6220 0x7fc914153070 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fc910002000 tx=0x7fc910015000 comp rx=0 tx=0).stop 2023-10-06T05:07:19.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.846+0000 7fc91ab99700 1 -- 172.21.15.98:0/2334092135 shutdown_connections 2023-10-06T05:07:19.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.846+0000 7fc91ab99700 1 --2- 172.21.15.98:0/2334092135 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc914159470 0x7fc9140cdeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:19.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.846+0000 7fc91ab99700 1 --2- 172.21.15.98:0/2334092135 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fc8f8066f30 0x7fc8f80693e0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:19.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.846+0000 7fc91ab99700 1 --2- 172.21.15.98:0/2334092135 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9140c6220 0x7fc914153070 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:19.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.846+0000 7fc91ab99700 1 --2- 172.21.15.98:0/2334092135 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc9140cd160 0x7fc9140c8e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:19.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.846+0000 7fc91ab99700 1 -- 172.21.15.98:0/2334092135 >> 172.21.15.98:0/2334092135 conn(0x7fc9140995c0 msgr2=0x7fc9141539d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:19.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.847+0000 7fc91ab99700 1 -- 172.21.15.98:0/2334092135 shutdown_connections 2023-10-06T05:07:19.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.847+0000 7fc91ab99700 1 -- 172.21.15.98:0/2334092135 wait complete. 2023-10-06T05:07:19.863 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-06T05:06:44.559671+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-10-06T05:06:51.432161+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-06T05:06:55.589057+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-10-06T05:06:56.041687+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-06T05:06:59.754099+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '29', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-06T05:07:19.864 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-06T05:07:19.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.979+0000 7f7f56dad700 1 Processor -- start 2023-10-06T05:07:19.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.979+0000 7f7f56dad700 1 -- start start 2023-10-06T05:07:19.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.980+0000 7f7f56dad700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7f501598a0 0x7f7f500ce7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:19.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.980+0000 7f7f56dad700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7f500cbc70 0x7f7f500ced20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:19.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.980+0000 7f7f56dad700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f500cf5d0 con 0x7f7f501594d0 2023-10-06T05:07:19.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.980+0000 7f7f56dad700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f500cf8b0 con 0x7f7f501598a0 2023-10-06T05:07:19.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.980+0000 7f7f56dad700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f500cfb90 con 0x7f7f500cbc70 2023-10-06T05:07:19.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.980+0000 7f7f47fff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7f501598a0 0x7f7f500ce7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:19.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.980+0000 7f7f54b49700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f7f501594d0 0x7f7f500cb560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:60882/0 (socket says 172.21.15.98:60882) 2023-10-06T05:07:19.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.980+0000 7f7f54b49700 1 -- 172.21.15.98:0/2343599769 learned_addr learned my addr 172.21.15.98:0/2343599769 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:07:19.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.981+0000 7f7f5534a700 1 --2- 172.21.15.98:0/2343599769 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7f500cbc70 0x7f7f500ced20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:19.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.981+0000 7f7f47fff700 1 -- 172.21.15.98:0/2343599769 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7f500cbc70 msgr2=0x7f7f500ced20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:19.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.981+0000 7f7f47fff700 1 --2- 172.21.15.98:0/2343599769 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7f500cbc70 0x7f7f500ced20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:19.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.981+0000 7f7f47fff700 1 -- 172.21.15.98:0/2343599769 >> v1:172.21.15.98:6789/0 conn(0x7f7f501594d0 legacy=0x7f7f500cb560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:19.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.981+0000 7f7f47fff700 1 -- 172.21.15.98:0/2343599769 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f500cfe70 con 0x7f7f501598a0 2023-10-06T05:07:19.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.981+0000 7f7f477fe700 1 -- 172.21.15.98:0/2343599769 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1436105936 0 0) 0x7f7f500cf5d0 con 0x7f7f501594d0 2023-10-06T05:07:19.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.981+0000 7f7f47fff700 1 --2- 172.21.15.98:0/2343599769 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f501598a0 0x7f7f500ce7e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f7f400098e0 tx=0x7f7f400091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac42ebb089f63068 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:19.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.981+0000 7f7f477fe700 1 -- 172.21.15.98:0/2343599769 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f40009e80 con 0x7f7f501598a0 2023-10-06T05:07:19.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.981+0000 7f7f477fe700 1 -- 172.21.15.98:0/2343599769 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f4000b860 con 0x7f7f501598a0 2023-10-06T05:07:19.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.981+0000 7f7f477fe700 1 -- 172.21.15.98:0/2343599769 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f4000bbe0 con 0x7f7f501598a0 2023-10-06T05:07:19.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.981+0000 7f7f56dad700 1 -- 172.21.15.98:0/2343599769 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f501598a0 msgr2=0x7f7f500ce7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:19.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.981+0000 7f7f56dad700 1 --2- 172.21.15.98:0/2343599769 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f501598a0 0x7f7f500ce7e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f7f400098e0 tx=0x7f7f400091a0 comp rx=0 tx=0).stop 2023-10-06T05:07:19.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.981+0000 7f7f56dad700 1 -- 172.21.15.98:0/2343599769 shutdown_connections 2023-10-06T05:07:19.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.981+0000 7f7f56dad700 1 --2- 172.21.15.98:0/2343599769 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7f500cbc70 0x7f7f500ced20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:19.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.981+0000 7f7f56dad700 1 --2- 172.21.15.98:0/2343599769 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f501598a0 0x7f7f500ce7e0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:19.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.981+0000 7f7f56dad700 1 -- 172.21.15.98:0/2343599769 >> 172.21.15.98:0/2343599769 conn(0x7f7f50089e50 msgr2=0x7f7f50099770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:19.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.981+0000 7f7f56dad700 1 -- 172.21.15.98:0/2343599769 shutdown_connections 2023-10-06T05:07:19.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.982+0000 7f7f56dad700 1 -- 172.21.15.98:0/2343599769 wait complete. 2023-10-06T05:07:19.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.983+0000 7f7f56dad700 1 Processor -- start 2023-10-06T05:07:19.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.983+0000 7f7f56dad700 1 -- start start 2023-10-06T05:07:19.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.983+0000 7f7f56dad700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7f500cbc70 0x7f7f5015a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:19.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.983+0000 7f7f56dad700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7f501594d0 0x7f7f5016a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:19.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.983+0000 7f7f56dad700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f501598a0 0x7f7f5016a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:19.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.983+0000 7f7f56dad700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f5014ed10 con 0x7f7f501598a0 2023-10-06T05:07:19.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.983+0000 7f7f56dad700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f5014eff0 con 0x7f7f501594d0 2023-10-06T05:07:19.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.983+0000 7f7f56dad700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f5014f2d0 con 0x7f7f500cbc70 2023-10-06T05:07:19.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.984+0000 7f7f5534a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f501598a0 0x7f7f5016a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:19.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.984+0000 7f7f5534a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f501598a0 0x7f7f5016a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46924/0 (socket says 172.21.15.98:46924) 2023-10-06T05:07:19.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.984+0000 7f7f5534a700 1 -- 172.21.15.98:0/2614283492 learned_addr learned my addr 172.21.15.98:0/2614283492 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:19.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.984+0000 7f7f54b49700 1 --2- 172.21.15.98:0/2614283492 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7f500cbc70 0x7f7f5015a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:19.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.984+0000 7f7f47fff700 1 --2- 172.21.15.98:0/2614283492 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7f501594d0 0x7f7f5016a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:19.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.984+0000 7f7f5534a700 1 -- 172.21.15.98:0/2614283492 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7f500cbc70 msgr2=0x7f7f5015a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:19.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.984+0000 7f7f5534a700 1 --2- 172.21.15.98:0/2614283492 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7f500cbc70 0x7f7f5015a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:19.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.984+0000 7f7f5534a700 1 -- 172.21.15.98:0/2614283492 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7f501594d0 msgr2=0x7f7f5016a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:19.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.984+0000 7f7f5534a700 1 --2- 172.21.15.98:0/2614283492 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7f501594d0 0x7f7f5016a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:19.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.984+0000 7f7f5534a700 1 -- 172.21.15.98:0/2614283492 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f40009420 con 0x7f7f501598a0 2023-10-06T05:07:19.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.984+0000 7f7f5534a700 1 --2- 172.21.15.98:0/2614283492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f501598a0 0x7f7f5016a810 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f7f4800d890 tx=0x7f7f48008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:19.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.984+0000 7f7f44ff9700 1 -- 172.21.15.98:0/2614283492 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f480125f0 con 0x7f7f501598a0 2023-10-06T05:07:19.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.984+0000 7f7f44ff9700 1 -- 172.21.15.98:0/2614283492 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f48012770 con 0x7f7f501598a0 2023-10-06T05:07:19.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.984+0000 7f7f56dad700 1 -- 172.21.15.98:0/2614283492 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f5014f5b0 con 0x7f7f501598a0 2023-10-06T05:07:19.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.984+0000 7f7f44ff9700 1 -- 172.21.15.98:0/2614283492 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f48012b30 con 0x7f7f501598a0 2023-10-06T05:07:19.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.984+0000 7f7f56dad700 1 -- 172.21.15.98:0/2614283492 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f5014e930 con 0x7f7f501598a0 2023-10-06T05:07:19.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.985+0000 7f7f56dad700 1 -- 172.21.15.98:0/2614283492 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f50064a90 con 0x7f7f501598a0 2023-10-06T05:07:19.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.985+0000 7f7f44ff9700 1 -- 172.21.15.98:0/2614283492 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f7f4801e460 con 0x7f7f501598a0 2023-10-06T05:07:19.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.986+0000 7f7f44ff9700 1 --2- 172.21.15.98:0/2614283492 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f7f3c066f30 0x7f7f3c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:19.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.986+0000 7f7f44ff9700 1 -- 172.21.15.98:0/2614283492 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f7f48084ed0 con 0x7f7f501598a0 2023-10-06T05:07:19.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.986+0000 7f7f54b49700 1 --2- 172.21.15.98:0/2614283492 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f7f3c066f30 0x7f7f3c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:19.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.986+0000 7f7f54b49700 1 --2- 172.21.15.98:0/2614283492 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f7f3c066f30 0x7f7f3c0693e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c002250 tx=0x7f7f4c001a20 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:19.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:19.990+0000 7f7f44ff9700 1 -- 172.21.15.98:0/2614283492 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7f48055780 con 0x7f7f501598a0 2023-10-06T05:07:20.175 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.172+0000 7f7f56dad700 1 -- 172.21.15.98:0/2614283492 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7f5004bb40 con 0x7f7f501598a0 2023-10-06T05:07:20.175 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.173+0000 7f7f44ff9700 1 -- 172.21.15.98:0/2614283492 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v29) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7f48055310 con 0x7f7f501598a0 2023-10-06T05:07:20.175 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2023-10-06T05:07:20.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.175+0000 7f7f56dad700 1 -- 172.21.15.98:0/2614283492 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f7f3c066f30 msgr2=0x7f7f3c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:20.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.175+0000 7f7f56dad700 1 --2- 172.21.15.98:0/2614283492 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f7f3c066f30 0x7f7f3c0693e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c002250 tx=0x7f7f4c001a20 comp rx=0 tx=0).stop 2023-10-06T05:07:20.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.175+0000 7f7f56dad700 1 -- 172.21.15.98:0/2614283492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f501598a0 msgr2=0x7f7f5016a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:20.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.175+0000 7f7f56dad700 1 --2- 172.21.15.98:0/2614283492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f501598a0 0x7f7f5016a810 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f7f4800d890 tx=0x7f7f48008040 comp rx=0 tx=0).stop 2023-10-06T05:07:20.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.175+0000 7f7f56dad700 1 -- 172.21.15.98:0/2614283492 shutdown_connections 2023-10-06T05:07:20.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.175+0000 7f7f56dad700 1 --2- 172.21.15.98:0/2614283492 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f7f3c066f30 0x7f7f3c0693e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:20.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.175+0000 7f7f56dad700 1 --2- 172.21.15.98:0/2614283492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f501598a0 0x7f7f5016a810 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:20.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.175+0000 7f7f56dad700 1 --2- 172.21.15.98:0/2614283492 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7f500cbc70 0x7f7f5015a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:20.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.175+0000 7f7f56dad700 1 --2- 172.21.15.98:0/2614283492 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7f501594d0 0x7f7f5016a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:20.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.175+0000 7f7f56dad700 1 -- 172.21.15.98:0/2614283492 >> 172.21.15.98:0/2614283492 conn(0x7f7f50089e50 msgr2=0x7f7f500c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:20.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.175+0000 7f7f56dad700 1 -- 172.21.15.98:0/2614283492 shutdown_connections 2023-10-06T05:07:20.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.175+0000 7f7f56dad700 1 -- 172.21.15.98:0/2614283492 wait complete. 2023-10-06T05:07:20.189 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-10-06T05:07:20.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.312+0000 7ffbc5645700 1 Processor -- start 2023-10-06T05:07:20.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.312+0000 7ffbc5645700 1 -- start start 2023-10-06T05:07:20.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.312+0000 7ffbc5645700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffbc0067bd0 0x7ffbc00cad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:20.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.313+0000 7ffbc5645700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ffbc015dcf0 0x7ffbc00cb2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:20.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.313+0000 7ffbc5645700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbc015d860 con 0x7ffbc0159ae0 2023-10-06T05:07:20.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.313+0000 7ffbc5645700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbc015aad0 con 0x7ffbc015dcf0 2023-10-06T05:07:20.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.313+0000 7ffbc5645700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc015a7f0 con 0x7ffbc0067bd0 2023-10-06T05:07:20.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.313+0000 7ffbbf7fe700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7ffbc0159ae0 0x7ffbc0159f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:60886/0 (socket says 172.21.15.98:60886) 2023-10-06T05:07:20.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.313+0000 7ffbbf7fe700 1 -- 172.21.15.98:0/2054730553 learned_addr learned my addr 172.21.15.98:0/2054730553 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:07:20.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.313+0000 7ffbbe7fc700 1 --2- 172.21.15.98:0/2054730553 >> v2:172.21.15.174:3300/0 conn(0x7ffbc015dcf0 0x7ffbc00cb2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:20.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.313+0000 7ffbbeffd700 1 --2- 172.21.15.98:0/2054730553 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffbc0067bd0 0x7ffbc00cad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:20.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.313+0000 7ffbbdffb700 1 -- 172.21.15.98:0/2054730553 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3534467835 0 0) 0x7ffbc015d860 con 0x7ffbc0159ae0 2023-10-06T05:07:20.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.313+0000 7ffbbdffb700 1 -- 172.21.15.98:0/2054730553 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbac003570 con 0x7ffbc0159ae0 2023-10-06T05:07:20.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.313+0000 7ffbbe7fc700 1 -- 172.21.15.98:0/2054730553 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffbc0067bd0 msgr2=0x7ffbc00cad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:20.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.313+0000 7ffbbe7fc700 1 --2- 172.21.15.98:0/2054730553 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffbc0067bd0 0x7ffbc00cad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:20.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.313+0000 7ffbbe7fc700 1 -- 172.21.15.98:0/2054730553 >> v1:172.21.15.98:6789/0 conn(0x7ffbc0159ae0 legacy=0x7ffbc0159f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:20.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.313+0000 7ffbbe7fc700 1 -- 172.21.15.98:0/2054730553 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbc015d210 con 0x7ffbc015dcf0 2023-10-06T05:07:20.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.313+0000 7ffbbeffd700 1 --2- 172.21.15.98:0/2054730553 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffbc0067bd0 0x7ffbc00cad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T05:07:20.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.313+0000 7ffbbe7fc700 1 --2- 172.21.15.98:0/2054730553 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffbc015dcf0 0x7ffbc00cb2b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ffba80098e0 tx=0x7ffba80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d707c87523d73de0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:20.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.314+0000 7ffbbdffb700 1 -- 172.21.15.98:0/2054730553 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffba8011630 con 0x7ffbc015dcf0 2023-10-06T05:07:20.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.314+0000 7ffbbdffb700 1 -- 172.21.15.98:0/2054730553 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffba8011790 con 0x7ffbc015dcf0 2023-10-06T05:07:20.320 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.314+0000 7ffbbdffb700 1 -- 172.21.15.98:0/2054730553 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffba8011b30 con 0x7ffbc015dcf0 2023-10-06T05:07:20.320 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.314+0000 7ffbc5645700 1 -- 172.21.15.98:0/2054730553 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffbc015dcf0 msgr2=0x7ffbc00cb2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:20.320 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.314+0000 7ffbc5645700 1 --2- 172.21.15.98:0/2054730553 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffbc015dcf0 0x7ffbc00cb2b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ffba80098e0 tx=0x7ffba80091a0 comp rx=0 tx=0).stop 2023-10-06T05:07:20.320 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.314+0000 7ffbc5645700 1 -- 172.21.15.98:0/2054730553 shutdown_connections 2023-10-06T05:07:20.320 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.314+0000 7ffbc5645700 1 --2- 172.21.15.98:0/2054730553 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffbc0067bd0 0x7ffbc00cad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:20.321 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.314+0000 7ffbc5645700 1 --2- 172.21.15.98:0/2054730553 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffbc015dcf0 0x7ffbc00cb2b0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:20.321 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.314+0000 7ffbc5645700 1 -- 172.21.15.98:0/2054730553 >> 172.21.15.98:0/2054730553 conn(0x7ffbc00cd1a0 msgr2=0x7ffbc00cd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:20.321 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.314+0000 7ffbc5645700 1 -- 172.21.15.98:0/2054730553 shutdown_connections 2023-10-06T05:07:20.321 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.314+0000 7ffbc5645700 1 -- 172.21.15.98:0/2054730553 wait complete. 2023-10-06T05:07:20.321 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.315+0000 7ffbc5645700 1 Processor -- start 2023-10-06T05:07:20.321 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.315+0000 7ffbc5645700 1 -- start start 2023-10-06T05:07:20.321 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.316+0000 7ffbc5645700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffbc0067bd0 0x7ffbc014bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:20.321 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.316+0000 7ffbc5645700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffbc0159ae0 0x7ffbc00c2f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:20.321 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.316+0000 7ffbc5645700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffbc015dcf0 0x7ffbc00c3450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:20.321 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.316+0000 7ffbc5645700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc0151390 con 0x7ffbc0067bd0 2023-10-06T05:07:20.321 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.316+0000 7ffbc5645700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc0150780 con 0x7ffbc0159ae0 2023-10-06T05:07:20.321 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.316+0000 7ffbc5645700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc0150a60 con 0x7ffbc015dcf0 2023-10-06T05:07:20.322 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.316+0000 7ffbbeffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffbc0067bd0 0x7ffbc014bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:20.322 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.316+0000 7ffbbeffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffbc0067bd0 0x7ffbc014bdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46932/0 (socket says 172.21.15.98:46932) 2023-10-06T05:07:20.322 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.316+0000 7ffbbeffd700 1 -- 172.21.15.98:0/2698802470 learned_addr learned my addr 172.21.15.98:0/2698802470 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:20.322 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.316+0000 7ffbbf7fe700 1 --2- 172.21.15.98:0/2698802470 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffbc015dcf0 0x7ffbc00c3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:20.322 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.317+0000 7ffbbe7fc700 1 --2- 172.21.15.98:0/2698802470 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffbc0159ae0 0x7ffbc00c2f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:20.322 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.317+0000 7ffbbeffd700 1 -- 172.21.15.98:0/2698802470 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffbc015dcf0 msgr2=0x7ffbc00c3450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:20.322 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.317+0000 7ffbbeffd700 1 --2- 172.21.15.98:0/2698802470 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffbc015dcf0 0x7ffbc00c3450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:20.322 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.317+0000 7ffbbeffd700 1 -- 172.21.15.98:0/2698802470 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffbc0159ae0 msgr2=0x7ffbc00c2f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:20.322 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.317+0000 7ffbbeffd700 1 --2- 172.21.15.98:0/2698802470 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffbc0159ae0 0x7ffbc00c2f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:20.322 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.317+0000 7ffbbeffd700 1 -- 172.21.15.98:0/2698802470 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbb8009a20 con 0x7ffbc0067bd0 2023-10-06T05:07:20.322 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.317+0000 7ffbbe7fc700 1 --2- 172.21.15.98:0/2698802470 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffbc0159ae0 0x7ffbc00c2f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:07:20.322 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.317+0000 7ffbbeffd700 1 --2- 172.21.15.98:0/2698802470 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffbc0067bd0 0x7ffbc014bdd0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ffbb800c170 tx=0x7ffbb800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:20.323 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.317+0000 7ffbb6ffd700 1 -- 172.21.15.98:0/2698802470 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb800ce10 con 0x7ffbc0067bd0 2023-10-06T05:07:20.323 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.317+0000 7ffbb6ffd700 1 -- 172.21.15.98:0/2698802470 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbb800f040 con 0x7ffbc0067bd0 2023-10-06T05:07:20.323 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.317+0000 7ffbc5645700 1 -- 172.21.15.98:0/2698802470 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffba8009420 con 0x7ffbc0067bd0 2023-10-06T05:07:20.323 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.317+0000 7ffbb6ffd700 1 -- 172.21.15.98:0/2698802470 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb8014690 con 0x7ffbc0067bd0 2023-10-06T05:07:20.323 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.317+0000 7ffbc5645700 1 -- 172.21.15.98:0/2698802470 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbc0151020 con 0x7ffbc0067bd0 2023-10-06T05:07:20.323 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.318+0000 7ffbc5645700 1 -- 172.21.15.98:0/2698802470 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbc0064a90 con 0x7ffbc0067bd0 2023-10-06T05:07:20.324 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.318+0000 7ffbb6ffd700 1 -- 172.21.15.98:0/2698802470 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7ffbb8014a00 con 0x7ffbc0067bd0 2023-10-06T05:07:20.324 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.319+0000 7ffbb6ffd700 1 --2- 172.21.15.98:0/2698802470 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7ffba0066f30 0x7ffba00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:20.324 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.319+0000 7ffbb6ffd700 1 -- 172.21.15.98:0/2698802470 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7ffbb8085860 con 0x7ffbc0067bd0 2023-10-06T05:07:20.324 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.319+0000 7ffbbe7fc700 1 --2- 172.21.15.98:0/2698802470 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7ffba0066f30 0x7ffba00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:20.324 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.319+0000 7ffbbe7fc700 1 --2- 172.21.15.98:0/2698802470 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7ffba0066f30 0x7ffba00693e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ffbc00c7380 tx=0x7ffba8017000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:20.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.323+0000 7ffbb6ffd700 1 -- 172.21.15.98:0/2698802470 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ffbb8052680 con 0x7ffbc0067bd0 2023-10-06T05:07:20.509 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.506+0000 7ffbc5645700 1 -- 172.21.15.98:0/2698802470 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7ffbc0159860 con 0x7ffbc0067bd0 2023-10-06T05:07:20.509 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.506+0000 7ffbb6ffd700 1 -- 172.21.15.98:0/2698802470 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v29) v1 ==== 92+0+10 (secure 0 0 0) 0x7ffbb8014cb0 con 0x7ffbc0067bd0 2023-10-06T05:07:20.509 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 8 2023-10-06T05:07:20.511 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.508+0000 7ffbc5645700 1 -- 172.21.15.98:0/2698802470 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7ffba0066f30 msgr2=0x7ffba00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:20.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.508+0000 7ffbc5645700 1 --2- 172.21.15.98:0/2698802470 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7ffba0066f30 0x7ffba00693e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ffbc00c7380 tx=0x7ffba8017000 comp rx=0 tx=0).stop 2023-10-06T05:07:20.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.508+0000 7ffbc5645700 1 -- 172.21.15.98:0/2698802470 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffbc0067bd0 msgr2=0x7ffbc014bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:20.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.508+0000 7ffbc5645700 1 --2- 172.21.15.98:0/2698802470 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffbc0067bd0 0x7ffbc014bdd0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ffbb800c170 tx=0x7ffbb800b040 comp rx=0 tx=0).stop 2023-10-06T05:07:20.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.508+0000 7ffbc5645700 1 -- 172.21.15.98:0/2698802470 shutdown_connections 2023-10-06T05:07:20.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.508+0000 7ffbc5645700 1 --2- 172.21.15.98:0/2698802470 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffbc015dcf0 0x7ffbc00c3450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:20.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.508+0000 7ffbc5645700 1 --2- 172.21.15.98:0/2698802470 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7ffba0066f30 0x7ffba00693e0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:20.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.508+0000 7ffbc5645700 1 --2- 172.21.15.98:0/2698802470 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffbc0067bd0 0x7ffbc014bdd0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:20.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.508+0000 7ffbc5645700 1 --2- 172.21.15.98:0/2698802470 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffbc0159ae0 0x7ffbc00c2f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:20.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.508+0000 7ffbc5645700 1 -- 172.21.15.98:0/2698802470 >> 172.21.15.98:0/2698802470 conn(0x7ffbc00cd1a0 msgr2=0x7ffbc015b6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:20.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.508+0000 7ffbc5645700 1 -- 172.21.15.98:0/2698802470 shutdown_connections 2023-10-06T05:07:20.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.508+0000 7ffbc5645700 1 -- 172.21.15.98:0/2698802470 wait complete. 2023-10-06T05:07:20.523 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-06T05:07:20.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.640+0000 7f2e72778700 1 Processor -- start 2023-10-06T05:07:20.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.640+0000 7f2e72778700 1 -- start start 2023-10-06T05:07:20.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.641+0000 7f2e72778700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f2e6c0b2620 0x7f2e6c0cccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:20.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.641+0000 7f2e72778700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2e6c0cf1b0 0x7f2e6c0cd1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:20.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.641+0000 7f2e72778700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e6c0cc970 con 0x7f2e6c0b29f0 2023-10-06T05:07:20.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.641+0000 7f2e72778700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e6c0cbd60 con 0x7f2e6c0b2620 2023-10-06T05:07:20.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.641+0000 7f2e72778700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e6c0cc040 con 0x7f2e6c0cf1b0 2023-10-06T05:07:20.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.641+0000 7f2e70f75700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f2e6c0b29f0 0x7f2e6c0ceaa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:60892/0 (socket says 172.21.15.98:60892) 2023-10-06T05:07:20.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.641+0000 7f2e70f75700 1 -- 172.21.15.98:0/3578446745 learned_addr learned my addr 172.21.15.98:0/3578446745 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:07:20.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.641+0000 7f2e71f77700 1 --2- 172.21.15.98:0/3578446745 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2e6c0cf1b0 0x7f2e6c0cd1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:20.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.641+0000 7f2e63fff700 1 -- 172.21.15.98:0/3578446745 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3068011569 0 0) 0x7f2e6c0cc970 con 0x7f2e6c0b29f0 2023-10-06T05:07:20.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.641+0000 7f2e71776700 1 --2- 172.21.15.98:0/3578446745 >> v2:172.21.15.174:3300/0 conn(0x7f2e6c0b2620 0x7f2e6c0cccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:20.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.641+0000 7f2e63fff700 1 -- 172.21.15.98:0/3578446745 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e58003570 con 0x7f2e6c0b29f0 2023-10-06T05:07:20.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.641+0000 7f2e63fff700 1 -- 172.21.15.98:0/3578446745 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4096371276 0 0) 0x7f2e58003570 con 0x7f2e6c0b29f0 2023-10-06T05:07:20.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.642+0000 7f2e63fff700 1 -- 172.21.15.98:0/3578446745 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2e6c0cf1b0 msgr2=0x7f2e6c0cd1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:20.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.642+0000 7f2e63fff700 1 --2- 172.21.15.98:0/3578446745 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2e6c0cf1b0 0x7f2e6c0cd1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:20.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.642+0000 7f2e63fff700 1 -- 172.21.15.98:0/3578446745 >> v2:172.21.15.174:3300/0 conn(0x7f2e6c0b2620 msgr2=0x7f2e6c0cccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:20.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.642+0000 7f2e63fff700 1 --2- 172.21.15.98:0/3578446745 >> v2:172.21.15.174:3300/0 conn(0x7f2e6c0b2620 0x7f2e6c0cccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:20.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.642+0000 7f2e63fff700 1 -- 172.21.15.98:0/3578446745 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e6c0cc320 con 0x7f2e6c0b29f0 2023-10-06T05:07:20.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.642+0000 7f2e63fff700 1 -- 172.21.15.98:0/3578446745 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1543991876 0 0) 0x7f2e5c001470 con 0x7f2e6c0b29f0 2023-10-06T05:07:20.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.642+0000 7f2e63fff700 1 -- 172.21.15.98:0/3578446745 >> v1:172.21.15.98:6789/0 conn(0x7f2e6c0b29f0 legacy=0x7f2e6c0ceaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:20.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.642+0000 7f2e63fff700 1 --2- 172.21.15.98:0/3578446745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e58003a60 0x7f2e58003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:20.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.642+0000 7f2e63fff700 1 --2- 172.21.15.98:0/3578446745 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2e6c0cf1b0 0x7f2e58004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:20.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.642+0000 7f2e63fff700 1 --2- 172.21.15.98:0/3578446745 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2e58004d50 0x7f2e580051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:20.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.642+0000 7f2e63fff700 1 -- 172.21.15.98:0/3578446745 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e6c0cbd60 con 0x7f2e58003a60 2023-10-06T05:07:20.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.642+0000 7f2e63fff700 1 -- 172.21.15.98:0/3578446745 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e6c0cc040 con 0x7f2e58004d50 2023-10-06T05:07:20.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.642+0000 7f2e63fff700 1 -- 172.21.15.98:0/3578446745 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e5800ba60 con 0x7f2e6c0cf1b0 2023-10-06T05:07:20.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.642+0000 7f2e71776700 1 --2- 172.21.15.98:0/3578446745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e58003a60 0x7f2e58003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:20.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.642+0000 7f2e70f75700 1 --2- 172.21.15.98:0/3578446745 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2e6c0cf1b0 0x7f2e58004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:20.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.642+0000 7f2e71f77700 1 --2- 172.21.15.98:0/3578446745 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2e58004d50 0x7f2e580051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:20.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.642+0000 7f2e71776700 1 -- 172.21.15.98:0/3578446745 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2e6c0cf1b0 msgr2=0x7f2e58004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:20.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.642+0000 7f2e71776700 1 --2- 172.21.15.98:0/3578446745 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2e6c0cf1b0 0x7f2e58004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:20.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.642+0000 7f2e71776700 1 -- 172.21.15.98:0/3578446745 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2e58004d50 msgr2=0x7f2e580051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:20.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.642+0000 7f2e71776700 1 --2- 172.21.15.98:0/3578446745 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2e58004d50 0x7f2e580051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:20.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.642+0000 7f2e71776700 1 -- 172.21.15.98:0/3578446745 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e58003570 con 0x7f2e58003a60 2023-10-06T05:07:20.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.642+0000 7f2e71776700 1 --2- 172.21.15.98:0/3578446745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e58003a60 0x7f2e58003e70 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f2e6800e2d0 tx=0x7f2e6800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9459f2f3e9557a3f server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:20.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.642+0000 7f2e63fff700 1 -- 172.21.15.98:0/3578446745 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e6800eac0 con 0x7f2e58003a60 2023-10-06T05:07:20.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.642+0000 7f2e63fff700 1 -- 172.21.15.98:0/3578446745 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e6800f040 con 0x7f2e58003a60 2023-10-06T05:07:20.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.642+0000 7f2e63fff700 1 -- 172.21.15.98:0/3578446745 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e6800ee70 con 0x7f2e58003a60 2023-10-06T05:07:20.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.643+0000 7f2e72778700 1 -- 172.21.15.98:0/3578446745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e58003a60 msgr2=0x7f2e58003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:20.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.643+0000 7f2e72778700 1 --2- 172.21.15.98:0/3578446745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e58003a60 0x7f2e58003e70 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f2e6800e2d0 tx=0x7f2e6800b040 comp rx=0 tx=0).stop 2023-10-06T05:07:20.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.643+0000 7f2e71f77700 1 -- 172.21.15.98:0/3578446745 reap_dead start 2023-10-06T05:07:20.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.643+0000 7f2e72778700 1 -- 172.21.15.98:0/3578446745 shutdown_connections 2023-10-06T05:07:20.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.643+0000 7f2e72778700 1 -- 172.21.15.98:0/3578446745 >> 172.21.15.98:0/3578446745 conn(0x7f2e6c099520 msgr2=0x7f2e6c06f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:20.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.643+0000 7f2e72778700 1 -- 172.21.15.98:0/3578446745 shutdown_connections 2023-10-06T05:07:20.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.643+0000 7f2e72778700 1 -- 172.21.15.98:0/3578446745 wait complete. 2023-10-06T05:07:20.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.644+0000 7f2e72778700 1 Processor -- start 2023-10-06T05:07:20.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.644+0000 7f2e72778700 1 -- start start 2023-10-06T05:07:20.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.645+0000 7f2e72778700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2e58003a60 0x7f2e6c14d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:20.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.645+0000 7f2e72778700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2e6c1514a0 0x7f2e6c151a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:20.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.645+0000 7f2e72778700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e6c151f80 0x7f2e6c0b42a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:20.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.645+0000 7f2e72778700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e6c152420 con 0x7f2e6c151f80 2023-10-06T05:07:20.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.645+0000 7f2e72778700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e6c154420 con 0x7f2e58003a60 2023-10-06T05:07:20.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.645+0000 7f2e72778700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e6c153850 con 0x7f2e6c1514a0 2023-10-06T05:07:20.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.645+0000 7f2e71f77700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e6c151f80 0x7f2e6c0b42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:20.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.645+0000 7f2e71f77700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e6c151f80 0x7f2e6c0b42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46940/0 (socket says 172.21.15.98:46940) 2023-10-06T05:07:20.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.645+0000 7f2e71f77700 1 -- 172.21.15.98:0/3535249947 learned_addr learned my addr 172.21.15.98:0/3535249947 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:20.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.645+0000 7f2e70f75700 1 --2- 172.21.15.98:0/3535249947 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2e6c1514a0 0x7f2e6c151a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:20.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.645+0000 7f2e71776700 1 --2- 172.21.15.98:0/3535249947 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2e58003a60 0x7f2e6c14d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:20.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.645+0000 7f2e71f77700 1 -- 172.21.15.98:0/3535249947 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2e6c1514a0 msgr2=0x7f2e6c151a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:20.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.645+0000 7f2e71f77700 1 --2- 172.21.15.98:0/3535249947 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2e6c1514a0 0x7f2e6c151a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:20.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.645+0000 7f2e71f77700 1 -- 172.21.15.98:0/3535249947 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2e58003a60 msgr2=0x7f2e6c14d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:20.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.645+0000 7f2e71f77700 1 --2- 172.21.15.98:0/3535249947 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2e58003a60 0x7f2e6c14d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:20.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.645+0000 7f2e71f77700 1 -- 172.21.15.98:0/3535249947 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e6c153af0 con 0x7f2e6c151f80 2023-10-06T05:07:20.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.645+0000 7f2e71f77700 1 --2- 172.21.15.98:0/3535249947 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e6c151f80 0x7f2e6c0b42a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f2e64006f00 tx=0x7f2e640091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:20.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.646+0000 7f2e617fa700 1 -- 172.21.15.98:0/3535249947 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e6400f940 con 0x7f2e6c151f80 2023-10-06T05:07:20.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.646+0000 7f2e617fa700 1 -- 172.21.15.98:0/3535249947 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e6400faa0 con 0x7f2e6c151f80 2023-10-06T05:07:20.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.646+0000 7f2e72778700 1 -- 172.21.15.98:0/3535249947 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e6c153dd0 con 0x7f2e6c151f80 2023-10-06T05:07:20.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.646+0000 7f2e617fa700 1 -- 172.21.15.98:0/3535249947 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e6400fe40 con 0x7f2e6c151f80 2023-10-06T05:07:20.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.646+0000 7f2e72778700 1 -- 172.21.15.98:0/3535249947 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e6c0b47e0 con 0x7f2e6c151f80 2023-10-06T05:07:20.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.646+0000 7f2e72778700 1 -- 172.21.15.98:0/3535249947 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e6c05f230 con 0x7f2e6c151f80 2023-10-06T05:07:20.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.647+0000 7f2e617fa700 1 -- 172.21.15.98:0/3535249947 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f2e64014650 con 0x7f2e6c151f80 2023-10-06T05:07:20.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.647+0000 7f2e617fa700 1 --2- 172.21.15.98:0/3535249947 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f2e50066f30 0x7f2e500693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:20.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.647+0000 7f2e617fa700 1 -- 172.21.15.98:0/3535249947 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f2e64084fb0 con 0x7f2e6c151f80 2023-10-06T05:07:20.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.647+0000 7f2e71776700 1 --2- 172.21.15.98:0/3535249947 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f2e50066f30 0x7f2e500693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:20.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.648+0000 7f2e71776700 1 --2- 172.21.15.98:0/3535249947 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f2e50066f30 0x7f2e500693e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f2e6c151350 tx=0x7f2e6801a000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:20.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.651+0000 7f2e617fa700 1 -- 172.21.15.98:0/3535249947 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2e64051e50 con 0x7f2e6c151f80 2023-10-06T05:07:20.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.836+0000 7f2e72778700 1 -- 172.21.15.98:0/3535249947 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2e6c0cc600 con 0x7f2e6c151f80 2023-10-06T05:07:20.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.837+0000 7f2e617fa700 1 -- 172.21.15.98:0/3535249947 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v29) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2e64055470 con 0x7f2e6c151f80 2023-10-06T05:07:20.839 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2023-10-06T05:07:20.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.839+0000 7f2e72778700 1 -- 172.21.15.98:0/3535249947 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f2e50066f30 msgr2=0x7f2e500693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:20.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.839+0000 7f2e72778700 1 --2- 172.21.15.98:0/3535249947 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f2e50066f30 0x7f2e500693e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f2e6c151350 tx=0x7f2e6801a000 comp rx=0 tx=0).stop 2023-10-06T05:07:20.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.839+0000 7f2e72778700 1 -- 172.21.15.98:0/3535249947 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e6c151f80 msgr2=0x7f2e6c0b42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:20.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.839+0000 7f2e72778700 1 --2- 172.21.15.98:0/3535249947 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e6c151f80 0x7f2e6c0b42a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f2e64006f00 tx=0x7f2e640091a0 comp rx=0 tx=0).stop 2023-10-06T05:07:20.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.839+0000 7f2e72778700 1 -- 172.21.15.98:0/3535249947 shutdown_connections 2023-10-06T05:07:20.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.839+0000 7f2e72778700 1 --2- 172.21.15.98:0/3535249947 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f2e50066f30 0x7f2e500693e0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:20.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.839+0000 7f2e72778700 1 --2- 172.21.15.98:0/3535249947 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e6c151f80 0x7f2e6c0b42a0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:20.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.839+0000 7f2e72778700 1 --2- 172.21.15.98:0/3535249947 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2e58003a60 0x7f2e6c14d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:20.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.839+0000 7f2e72778700 1 --2- 172.21.15.98:0/3535249947 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2e6c1514a0 0x7f2e6c151a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:20.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.839+0000 7f2e72778700 1 -- 172.21.15.98:0/3535249947 >> 172.21.15.98:0/3535249947 conn(0x7f2e6c099520 msgr2=0x7f2e6c0c6550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:20.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.839+0000 7f2e72778700 1 -- 172.21.15.98:0/3535249947 shutdown_connections 2023-10-06T05:07:20.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.839+0000 7f2e72778700 1 -- 172.21.15.98:0/3535249947 wait complete. 2023-10-06T05:07:20.853 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-06T05:07:20.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.967+0000 7ff7816b3700 1 Processor -- start 2023-10-06T05:07:20.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.967+0000 7ff7816b3700 1 -- start start 2023-10-06T05:07:20.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.968+0000 7ff7816b3700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ff77c0b2620 0x7ff77c0cccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:20.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.968+0000 7ff7816b3700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff77c0b29f0 0x7ff77c0cd1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:20.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.968+0000 7ff7816b3700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff77c0cc970 con 0x7ff77c0ceaa0 2023-10-06T05:07:20.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.968+0000 7ff7816b3700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff77c0cbd60 con 0x7ff77c0b29f0 2023-10-06T05:07:20.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.968+0000 7ff7816b3700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff77c0cc040 con 0x7ff77c0b2620 2023-10-06T05:07:20.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.968+0000 7ff77b7fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff77c0b29f0 0x7ff77c0cd1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:20.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.968+0000 7ff77b7fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff77c0b29f0 0x7ff77c0cd1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46950/0 (socket says 172.21.15.98:46950) 2023-10-06T05:07:20.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.968+0000 7ff77b7fe700 1 -- 172.21.15.98:0/3918863079 learned_addr learned my addr 172.21.15.98:0/3918863079 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:20.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.968+0000 7ff77bfff700 1 --2- 172.21.15.98:0/3918863079 >> v2:172.21.15.174:3300/0 conn(0x7ff77c0b2620 0x7ff77c0cccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:20.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.968+0000 7ff77b7fe700 1 -- 172.21.15.98:0/3918863079 >> v2:172.21.15.174:3300/0 conn(0x7ff77c0b2620 msgr2=0x7ff77c0cccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:20.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.968+0000 7ff77b7fe700 1 --2- 172.21.15.98:0/3918863079 >> v2:172.21.15.174:3300/0 conn(0x7ff77c0b2620 0x7ff77c0cccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:20.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.968+0000 7ff77b7fe700 1 -- 172.21.15.98:0/3918863079 >> v1:172.21.15.174:6789/0 conn(0x7ff77c0ceaa0 legacy=0x7ff77c0cef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:20.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.968+0000 7ff77b7fe700 1 -- 172.21.15.98:0/3918863079 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff77c0cc320 con 0x7ff77c0b29f0 2023-10-06T05:07:20.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.968+0000 7ff77bfff700 1 --2- 172.21.15.98:0/3918863079 >> v2:172.21.15.174:3300/0 conn(0x7ff77c0b2620 0x7ff77c0cccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:07:20.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.968+0000 7ff77b7fe700 1 --2- 172.21.15.98:0/3918863079 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff77c0b29f0 0x7ff77c0cd1f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ff77c0894c0 tx=0x7ff770009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25b1e423ed2b085f server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:20.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.969+0000 7ff77affd700 1 -- 172.21.15.98:0/3918863079 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff77000b820 con 0x7ff77c0b29f0 2023-10-06T05:07:20.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.969+0000 7ff77affd700 1 -- 172.21.15.98:0/3918863079 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff77000b980 con 0x7ff77c0b29f0 2023-10-06T05:07:20.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.969+0000 7ff77affd700 1 -- 172.21.15.98:0/3918863079 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff77000bd20 con 0x7ff77c0b29f0 2023-10-06T05:07:20.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.969+0000 7ff7816b3700 1 -- 172.21.15.98:0/3918863079 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff77c0b29f0 msgr2=0x7ff77c0cd1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:20.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.969+0000 7ff7816b3700 1 --2- 172.21.15.98:0/3918863079 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff77c0b29f0 0x7ff77c0cd1f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ff77c0894c0 tx=0x7ff770009250 comp rx=0 tx=0).stop 2023-10-06T05:07:20.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.969+0000 7ff7816b3700 1 -- 172.21.15.98:0/3918863079 shutdown_connections 2023-10-06T05:07:20.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.969+0000 7ff7816b3700 1 --2- 172.21.15.98:0/3918863079 >> v2:172.21.15.174:3300/0 conn(0x7ff77c0b2620 0x7ff77c0cccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:20.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.969+0000 7ff7816b3700 1 --2- 172.21.15.98:0/3918863079 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff77c0b29f0 0x7ff77c0cd1f0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:20.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.969+0000 7ff7816b3700 1 -- 172.21.15.98:0/3918863079 >> 172.21.15.98:0/3918863079 conn(0x7ff77c099520 msgr2=0x7ff77c06f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:20.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.969+0000 7ff7816b3700 1 -- 172.21.15.98:0/3918863079 shutdown_connections 2023-10-06T05:07:20.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.969+0000 7ff7816b3700 1 -- 172.21.15.98:0/3918863079 wait complete. 2023-10-06T05:07:20.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.970+0000 7ff7816b3700 1 Processor -- start 2023-10-06T05:07:20.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.970+0000 7ff7816b3700 1 -- start start 2023-10-06T05:07:20.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.971+0000 7ff7816b3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff77c0b2620 0x7ff77c0c7770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:20.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.971+0000 7ff7816b3700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff77c0b29f0 0x7ff77c0c9110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:20.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.971+0000 7ff7816b3700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff77c0ceaa0 0x7ff77c0c9650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:20.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.971+0000 7ff7816b3700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff77c156430 con 0x7ff77c0b2620 2023-10-06T05:07:20.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.971+0000 7ff7816b3700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff77c0c9b90 con 0x7ff77c0b29f0 2023-10-06T05:07:20.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.971+0000 7ff7816b3700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff77c0c7cb0 con 0x7ff77c0ceaa0 2023-10-06T05:07:20.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.971+0000 7ff77bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff77c0b2620 0x7ff77c0c7770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:20.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.971+0000 7ff77bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff77c0b2620 0x7ff77c0c7770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46956/0 (socket says 172.21.15.98:46956) 2023-10-06T05:07:20.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.971+0000 7ff77bfff700 1 -- 172.21.15.98:0/240378849 learned_addr learned my addr 172.21.15.98:0/240378849 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:20.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.971+0000 7ff780eb2700 1 --2- 172.21.15.98:0/240378849 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff77c0ceaa0 0x7ff77c0c9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:20.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.971+0000 7ff77b7fe700 1 --2- 172.21.15.98:0/240378849 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff77c0b29f0 0x7ff77c0c9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:20.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.971+0000 7ff77bfff700 1 -- 172.21.15.98:0/240378849 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff77c0ceaa0 msgr2=0x7ff77c0c9650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:20.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.971+0000 7ff77bfff700 1 --2- 172.21.15.98:0/240378849 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff77c0ceaa0 0x7ff77c0c9650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:20.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.971+0000 7ff77bfff700 1 -- 172.21.15.98:0/240378849 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff77c0b29f0 msgr2=0x7ff77c0c9110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:20.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.971+0000 7ff77bfff700 1 --2- 172.21.15.98:0/240378849 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff77c0b29f0 0x7ff77c0c9110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:20.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.971+0000 7ff77bfff700 1 -- 172.21.15.98:0/240378849 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7700094f0 con 0x7ff77c0b2620 2023-10-06T05:07:20.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.972+0000 7ff77bfff700 1 --2- 172.21.15.98:0/240378849 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff77c0b2620 0x7ff77c0c7770 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ff76c00c880 tx=0x7ff76c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:20.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.972+0000 7ff763fff700 1 -- 172.21.15.98:0/240378849 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff76c005a00 con 0x7ff77c0b2620 2023-10-06T05:07:20.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.972+0000 7ff763fff700 1 -- 172.21.15.98:0/240378849 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff76c005b60 con 0x7ff77c0b2620 2023-10-06T05:07:20.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.972+0000 7ff7816b3700 1 -- 172.21.15.98:0/240378849 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff77c156150 con 0x7ff77c0b2620 2023-10-06T05:07:20.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.972+0000 7ff763fff700 1 -- 172.21.15.98:0/240378849 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff76c011600 con 0x7ff77c0b2620 2023-10-06T05:07:20.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.972+0000 7ff7816b3700 1 -- 172.21.15.98:0/240378849 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff77c0b3a40 con 0x7ff77c0b2620 2023-10-06T05:07:20.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.973+0000 7ff7816b3700 1 -- 172.21.15.98:0/240378849 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff77c05f230 con 0x7ff77c0b2620 2023-10-06T05:07:20.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.973+0000 7ff763fff700 1 -- 172.21.15.98:0/240378849 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7ff76c0119f0 con 0x7ff77c0b2620 2023-10-06T05:07:20.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.974+0000 7ff763fff700 1 --2- 172.21.15.98:0/240378849 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7ff768066f30 0x7ff7680693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:20.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.974+0000 7ff77b7fe700 1 --2- 172.21.15.98:0/240378849 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7ff768066f30 0x7ff7680693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:20.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.974+0000 7ff763fff700 1 -- 172.21.15.98:0/240378849 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7ff76c0847c0 con 0x7ff77c0b2620 2023-10-06T05:07:20.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.974+0000 7ff77b7fe700 1 --2- 172.21.15.98:0/240378849 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7ff768066f30 0x7ff7680693e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff77c0894c0 tx=0x7ff770016000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:20.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:20.978+0000 7ff763fff700 1 -- 172.21.15.98:0/240378849 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff76c0515e0 con 0x7ff77c0b2620 2023-10-06T05:07:21.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.161+0000 7ff7816b3700 1 -- 172.21.15.98:0/240378849 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff77c0cc600 con 0x7ff77c0b2620 2023-10-06T05:07:21.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.162+0000 7ff763fff700 1 -- 172.21.15.98:0/240378849 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v29) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff76c01e020 con 0x7ff77c0b2620 2023-10-06T05:07:21.165 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2023-10-06T05:07:21.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.164+0000 7ff7816b3700 1 -- 172.21.15.98:0/240378849 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7ff768066f30 msgr2=0x7ff7680693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:21.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.164+0000 7ff7816b3700 1 --2- 172.21.15.98:0/240378849 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7ff768066f30 0x7ff7680693e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff77c0894c0 tx=0x7ff770016000 comp rx=0 tx=0).stop 2023-10-06T05:07:21.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.164+0000 7ff7816b3700 1 -- 172.21.15.98:0/240378849 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff77c0b2620 msgr2=0x7ff77c0c7770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:21.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.164+0000 7ff7816b3700 1 --2- 172.21.15.98:0/240378849 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff77c0b2620 0x7ff77c0c7770 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ff76c00c880 tx=0x7ff76c008040 comp rx=0 tx=0).stop 2023-10-06T05:07:21.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.164+0000 7ff7816b3700 1 -- 172.21.15.98:0/240378849 shutdown_connections 2023-10-06T05:07:21.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.164+0000 7ff7816b3700 1 --2- 172.21.15.98:0/240378849 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff77c0ceaa0 0x7ff77c0c9650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:21.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.164+0000 7ff7816b3700 1 --2- 172.21.15.98:0/240378849 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7ff768066f30 0x7ff7680693e0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:21.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.164+0000 7ff7816b3700 1 --2- 172.21.15.98:0/240378849 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff77c0b2620 0x7ff77c0c7770 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:21.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.164+0000 7ff7816b3700 1 --2- 172.21.15.98:0/240378849 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff77c0b29f0 0x7ff77c0c9110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:21.168 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.164+0000 7ff7816b3700 1 -- 172.21.15.98:0/240378849 >> 172.21.15.98:0/240378849 conn(0x7ff77c099520 msgr2=0x7ff77c0690f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:21.168 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.164+0000 7ff7816b3700 1 -- 172.21.15.98:0/240378849 shutdown_connections 2023-10-06T05:07:21.168 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.164+0000 7ff7816b3700 1 -- 172.21.15.98:0/240378849 wait complete. 2023-10-06T05:07:21.178 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-06T05:07:21.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.295+0000 7f7a3c1f5700 1 Processor -- start 2023-10-06T05:07:21.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.296+0000 7f7a3c1f5700 1 -- start start 2023-10-06T05:07:21.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.296+0000 7f7a3c1f5700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7a340b0a10 0x7f7a340cbee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:21.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.296+0000 7f7a3c1f5700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f7a340cc420 0x7f7a340c9050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:21.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.296+0000 7f7a3c1f5700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a340c53c0 con 0x7f7a340b0640 2023-10-06T05:07:21.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.296+0000 7f7a3c1f5700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a340ca790 con 0x7f7a340cc420 2023-10-06T05:07:21.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.296+0000 7f7a3c1f5700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a340ccd00 con 0x7f7a340b0a10 2023-10-06T05:07:21.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.296+0000 7f7a39f91700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f7a340b0640 0x7f7a340cecd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:60906/0 (socket says 172.21.15.98:60906) 2023-10-06T05:07:21.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.296+0000 7f7a39f91700 1 -- 172.21.15.98:0/1861812005 learned_addr learned my addr 172.21.15.98:0/1861812005 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:07:21.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.297+0000 7f7a39790700 1 --2- 172.21.15.98:0/1861812005 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7a340b0a10 0x7f7a340cbee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:21.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.297+0000 7f7a3a792700 1 --2- 172.21.15.98:0/1861812005 >> v2:172.21.15.174:3300/0 conn(0x7f7a340cc420 0x7f7a340c9050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:21.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.297+0000 7f7a38f8f700 1 -- 172.21.15.98:0/1861812005 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2300983528 0 0) 0x7f7a340c53c0 con 0x7f7a340b0640 2023-10-06T05:07:21.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.297+0000 7f7a38f8f700 1 -- 172.21.15.98:0/1861812005 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a20003570 con 0x7f7a340b0640 2023-10-06T05:07:21.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.297+0000 7f7a38f8f700 1 -- 172.21.15.98:0/1861812005 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3704117865 0 0) 0x7f7a20003570 con 0x7f7a340b0640 2023-10-06T05:07:21.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.297+0000 7f7a38f8f700 1 -- 172.21.15.98:0/1861812005 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7a340b0a10 msgr2=0x7f7a340cbee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:21.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.297+0000 7f7a38f8f700 1 --2- 172.21.15.98:0/1861812005 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7a340b0a10 0x7f7a340cbee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:21.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.297+0000 7f7a38f8f700 1 -- 172.21.15.98:0/1861812005 >> v2:172.21.15.174:3300/0 conn(0x7f7a340cc420 msgr2=0x7f7a340c9050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:21.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.297+0000 7f7a38f8f700 1 --2- 172.21.15.98:0/1861812005 >> v2:172.21.15.174:3300/0 conn(0x7f7a340cc420 0x7f7a340c9050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:21.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.297+0000 7f7a38f8f700 1 -- 172.21.15.98:0/1861812005 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a340ccfe0 con 0x7f7a340b0640 2023-10-06T05:07:21.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.297+0000 7f7a38f8f700 1 -- 172.21.15.98:0/1861812005 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1543991876 0 0) 0x7f7a30001970 con 0x7f7a340b0640 2023-10-06T05:07:21.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.297+0000 7f7a38f8f700 1 -- 172.21.15.98:0/1861812005 >> v1:172.21.15.98:6789/0 conn(0x7f7a340b0640 legacy=0x7f7a340cecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:21.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.297+0000 7f7a38f8f700 1 --2- 172.21.15.98:0/1861812005 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a20003a60 0x7f7a20003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:21.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.297+0000 7f7a38f8f700 1 --2- 172.21.15.98:0/1861812005 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7a20004810 0x7f7a20004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:21.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.297+0000 7f7a38f8f700 1 --2- 172.21.15.98:0/1861812005 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7a340b0a10 0x7f7a200051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:21.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.297+0000 7f7a38f8f700 1 -- 172.21.15.98:0/1861812005 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a340ca790 con 0x7f7a20003a60 2023-10-06T05:07:21.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.297+0000 7f7a38f8f700 1 -- 172.21.15.98:0/1861812005 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a340ccd00 con 0x7f7a20004810 2023-10-06T05:07:21.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.297+0000 7f7a38f8f700 1 -- 172.21.15.98:0/1861812005 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2000ba60 con 0x7f7a340b0a10 2023-10-06T05:07:21.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.297+0000 7f7a39f91700 1 --2- 172.21.15.98:0/1861812005 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a20003a60 0x7f7a20003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:21.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.297+0000 7f7a39790700 1 --2- 172.21.15.98:0/1861812005 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7a20004810 0x7f7a20004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:21.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.297+0000 7f7a3a792700 1 --2- 172.21.15.98:0/1861812005 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7a340b0a10 0x7f7a200051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:21.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.297+0000 7f7a39f91700 1 -- 172.21.15.98:0/1861812005 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7a340b0a10 msgr2=0x7f7a200051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:21.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.297+0000 7f7a39f91700 1 --2- 172.21.15.98:0/1861812005 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7a340b0a10 0x7f7a200051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:21.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.297+0000 7f7a39f91700 1 -- 172.21.15.98:0/1861812005 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7a20004810 msgr2=0x7f7a20004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:21.312 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.297+0000 7f7a39f91700 1 --2- 172.21.15.98:0/1861812005 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7a20004810 0x7f7a20004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:21.312 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.297+0000 7f7a39f91700 1 -- 172.21.15.98:0/1861812005 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a20003570 con 0x7f7a20003a60 2023-10-06T05:07:21.312 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.298+0000 7f7a39f91700 1 --2- 172.21.15.98:0/1861812005 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a20003a60 0x7f7a20003e70 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f7a3000c6e0 tx=0x7f7a30007ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d61caca490c06e4 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:21.312 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.298+0000 7f7a38f8f700 1 -- 172.21.15.98:0/1861812005 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a30005b70 con 0x7f7a20003a60 2023-10-06T05:07:21.312 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.298+0000 7f7a38f8f700 1 -- 172.21.15.98:0/1861812005 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a30005cf0 con 0x7f7a20003a60 2023-10-06T05:07:21.312 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.298+0000 7f7a38f8f700 1 -- 172.21.15.98:0/1861812005 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a30007360 con 0x7f7a20003a60 2023-10-06T05:07:21.312 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.298+0000 7f7a3c1f5700 1 -- 172.21.15.98:0/1861812005 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a20003a60 msgr2=0x7f7a20003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:21.313 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.298+0000 7f7a3c1f5700 1 --2- 172.21.15.98:0/1861812005 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a20003a60 0x7f7a20003e70 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f7a3000c6e0 tx=0x7f7a30007ca0 comp rx=0 tx=0).stop 2023-10-06T05:07:21.313 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.298+0000 7f7a3a792700 1 -- 172.21.15.98:0/1861812005 reap_dead start 2023-10-06T05:07:21.313 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.298+0000 7f7a3c1f5700 1 -- 172.21.15.98:0/1861812005 shutdown_connections 2023-10-06T05:07:21.313 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.298+0000 7f7a3c1f5700 1 -- 172.21.15.98:0/1861812005 >> 172.21.15.98:0/1861812005 conn(0x7f7a34087fc0 msgr2=0x7f7a340ce310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:21.313 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.298+0000 7f7a3c1f5700 1 -- 172.21.15.98:0/1861812005 shutdown_connections 2023-10-06T05:07:21.313 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.298+0000 7f7a3c1f5700 1 -- 172.21.15.98:0/1861812005 wait complete. 2023-10-06T05:07:21.313 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.299+0000 7f7a3c1f5700 1 Processor -- start 2023-10-06T05:07:21.313 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.299+0000 7f7a3c1f5700 1 -- start start 2023-10-06T05:07:21.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.300+0000 7f7a3c1f5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a20003a60 0x7f7a3414d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:21.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.300+0000 7f7a3c1f5700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7a3415c3d0 0x7f7a3415c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:21.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.300+0000 7f7a3c1f5700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7a340b2750 0x7f7a340b2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:21.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.300+0000 7f7a3c1f5700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a3416c740 con 0x7f7a20003a60 2023-10-06T05:07:21.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.300+0000 7f7a3c1f5700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a3415cd80 con 0x7f7a340b2750 2023-10-06T05:07:21.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.300+0000 7f7a3c1f5700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a3414ddd0 con 0x7f7a3415c3d0 2023-10-06T05:07:21.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.300+0000 7f7a39f91700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a20003a60 0x7f7a3414d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:21.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.300+0000 7f7a39f91700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a20003a60 0x7f7a3414d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46974/0 (socket says 172.21.15.98:46974) 2023-10-06T05:07:21.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.300+0000 7f7a39f91700 1 -- 172.21.15.98:0/3047552712 learned_addr learned my addr 172.21.15.98:0/3047552712 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:21.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.300+0000 7f7a39790700 1 --2- 172.21.15.98:0/3047552712 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7a3415c3d0 0x7f7a3415c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:21.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.300+0000 7f7a3a792700 1 --2- 172.21.15.98:0/3047552712 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7a340b2750 0x7f7a340b2bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:21.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.300+0000 7f7a39f91700 1 -- 172.21.15.98:0/3047552712 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7a3415c3d0 msgr2=0x7f7a3415c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:21.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.301+0000 7f7a39f91700 1 --2- 172.21.15.98:0/3047552712 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7a3415c3d0 0x7f7a3415c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:21.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.301+0000 7f7a39f91700 1 -- 172.21.15.98:0/3047552712 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7a340b2750 msgr2=0x7f7a340b2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:21.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.301+0000 7f7a39f91700 1 --2- 172.21.15.98:0/3047552712 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7a340b2750 0x7f7a340b2bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:21.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.301+0000 7f7a39f91700 1 -- 172.21.15.98:0/3047552712 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a340b3480 con 0x7f7a20003a60 2023-10-06T05:07:21.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.301+0000 7f7a3a792700 1 --2- 172.21.15.98:0/3047552712 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7a340b2750 0x7f7a340b2bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:07:21.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.301+0000 7f7a39f91700 1 --2- 172.21.15.98:0/3047552712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a20003a60 0x7f7a3414d540 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f7a3000d010 tx=0x7f7a30015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:21.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.301+0000 7f7a29ffb700 1 -- 172.21.15.98:0/3047552712 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a30001b40 con 0x7f7a20003a60 2023-10-06T05:07:21.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.301+0000 7f7a29ffb700 1 -- 172.21.15.98:0/3047552712 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a30001cc0 con 0x7f7a20003a60 2023-10-06T05:07:21.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.301+0000 7f7a3c1f5700 1 -- 172.21.15.98:0/3047552712 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a340b3760 con 0x7f7a20003a60 2023-10-06T05:07:21.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.301+0000 7f7a29ffb700 1 -- 172.21.15.98:0/3047552712 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a30014670 con 0x7f7a20003a60 2023-10-06T05:07:21.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.301+0000 7f7a3c1f5700 1 -- 172.21.15.98:0/3047552712 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a3415a350 con 0x7f7a20003a60 2023-10-06T05:07:21.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.302+0000 7f7a3c1f5700 1 -- 172.21.15.98:0/3047552712 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a340cf290 con 0x7f7a20003a60 2023-10-06T05:07:21.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.302+0000 7f7a29ffb700 1 -- 172.21.15.98:0/3047552712 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f7a30014a60 con 0x7f7a20003a60 2023-10-06T05:07:21.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.303+0000 7f7a29ffb700 1 --2- 172.21.15.98:0/3047552712 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f7a18066f30 0x7f7a180693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:21.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.303+0000 7f7a29ffb700 1 -- 172.21.15.98:0/3047552712 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f7a30086a80 con 0x7f7a20003a60 2023-10-06T05:07:21.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.303+0000 7f7a39790700 1 --2- 172.21.15.98:0/3047552712 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f7a18066f30 0x7f7a180693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:21.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.303+0000 7f7a39790700 1 --2- 172.21.15.98:0/3047552712 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f7a18066f30 0x7f7a180693e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f7a340b0330 tx=0x7f7a240091a0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:21.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.307+0000 7f7a29ffb700 1 -- 172.21.15.98:0/3047552712 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7a30053800 con 0x7f7a20003a60 2023-10-06T05:07:21.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.490+0000 7f7a3c1f5700 1 -- 172.21.15.98:0/3047552712 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f7a340cf290 con 0x7f7a20003a60 2023-10-06T05:07:21.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.490+0000 7f7a29ffb700 1 -- 172.21.15.98:0/3047552712 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v29) v1 ==== 103+0+11 (secure 0 0 0) 0x7f7a30056e20 con 0x7f7a20003a60 2023-10-06T05:07:21.493 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2023-10-06T05:07:21.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.492+0000 7f7a3c1f5700 1 -- 172.21.15.98:0/3047552712 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f7a18066f30 msgr2=0x7f7a180693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:21.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.492+0000 7f7a3c1f5700 1 --2- 172.21.15.98:0/3047552712 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f7a18066f30 0x7f7a180693e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f7a340b0330 tx=0x7f7a240091a0 comp rx=0 tx=0).stop 2023-10-06T05:07:21.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.492+0000 7f7a3c1f5700 1 -- 172.21.15.98:0/3047552712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a20003a60 msgr2=0x7f7a3414d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:21.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.492+0000 7f7a3c1f5700 1 --2- 172.21.15.98:0/3047552712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a20003a60 0x7f7a3414d540 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f7a3000d010 tx=0x7f7a30015000 comp rx=0 tx=0).stop 2023-10-06T05:07:21.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.492+0000 7f7a3c1f5700 1 -- 172.21.15.98:0/3047552712 shutdown_connections 2023-10-06T05:07:21.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.492+0000 7f7a3c1f5700 1 --2- 172.21.15.98:0/3047552712 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7a340b2750 0x7f7a340b2bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:21.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.492+0000 7f7a3c1f5700 1 --2- 172.21.15.98:0/3047552712 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f7a18066f30 0x7f7a180693e0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:21.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.492+0000 7f7a3c1f5700 1 --2- 172.21.15.98:0/3047552712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a20003a60 0x7f7a3414d540 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:21.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.492+0000 7f7a3c1f5700 1 --2- 172.21.15.98:0/3047552712 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7a3415c3d0 0x7f7a3415c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:21.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.492+0000 7f7a3c1f5700 1 -- 172.21.15.98:0/3047552712 >> 172.21.15.98:0/3047552712 conn(0x7f7a34087fc0 msgr2=0x7f7a340c78a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:21.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.492+0000 7f7a3c1f5700 1 -- 172.21.15.98:0/3047552712 shutdown_connections 2023-10-06T05:07:21.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.492+0000 7f7a3c1f5700 1 -- 172.21.15.98:0/3047552712 wait complete. 2023-10-06T05:07:21.507 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-06T05:07:21.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.622+0000 7f847e395700 1 Processor -- start 2023-10-06T05:07:21.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.622+0000 7f847e395700 1 -- start start 2023-10-06T05:07:21.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.623+0000 7f847e395700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f8478067800 0x7f84780c7a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:21.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.623+0000 7f847e395700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f84780ca9f0 0x7f847806c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:21.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.623+0000 7f847e395700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f847806bfa0 con 0x7f84780c7f60 2023-10-06T05:07:21.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.623+0000 7f847e395700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8478162040 con 0x7f84780ca9f0 2023-10-06T05:07:21.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.623+0000 7f847e395700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8478162320 con 0x7f8478067800 2023-10-06T05:07:21.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.623+0000 7f84777fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f84780ca9f0 0x7f847806c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:21.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.623+0000 7f84777fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f84780ca9f0 0x7f847806c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46990/0 (socket says 172.21.15.98:46990) 2023-10-06T05:07:21.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.623+0000 7f84777fe700 1 -- 172.21.15.98:0/3930989040 learned_addr learned my addr 172.21.15.98:0/3930989040 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:21.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.623+0000 7f8477fff700 1 --2- 172.21.15.98:0/3930989040 >> v2:172.21.15.174:3300/0 conn(0x7f8478067800 0x7f84780c7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:21.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.623+0000 7f84777fe700 1 -- 172.21.15.98:0/3930989040 >> v2:172.21.15.174:3300/0 conn(0x7f8478067800 msgr2=0x7f84780c7a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:21.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.623+0000 7f84777fe700 1 --2- 172.21.15.98:0/3930989040 >> v2:172.21.15.174:3300/0 conn(0x7f8478067800 0x7f84780c7a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:21.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.623+0000 7f84777fe700 1 -- 172.21.15.98:0/3930989040 >> v1:172.21.15.174:6789/0 conn(0x7f84780c7f60 legacy=0x7f847806e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:21.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.623+0000 7f84777fe700 1 -- 172.21.15.98:0/3930989040 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8478162600 con 0x7f84780ca9f0 2023-10-06T05:07:21.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.624+0000 7f84777fe700 1 --2- 172.21.15.98:0/3930989040 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84780ca9f0 0x7f847806c430 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f84780712e0 tx=0x7f846c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db1e8781762de12f server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:21.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.624+0000 7f8476ffd700 1 -- 172.21.15.98:0/3930989040 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f846c00b820 con 0x7f84780ca9f0 2023-10-06T05:07:21.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.624+0000 7f8476ffd700 1 -- 172.21.15.98:0/3930989040 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f846c00b980 con 0x7f84780ca9f0 2023-10-06T05:07:21.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.624+0000 7f8476ffd700 1 -- 172.21.15.98:0/3930989040 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f846c00bd20 con 0x7f84780ca9f0 2023-10-06T05:07:21.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.624+0000 7f847e395700 1 -- 172.21.15.98:0/3930989040 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84780ca9f0 msgr2=0x7f847806c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:21.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.624+0000 7f847e395700 1 --2- 172.21.15.98:0/3930989040 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84780ca9f0 0x7f847806c430 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f84780712e0 tx=0x7f846c009250 comp rx=0 tx=0).stop 2023-10-06T05:07:21.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.624+0000 7f847e395700 1 -- 172.21.15.98:0/3930989040 shutdown_connections 2023-10-06T05:07:21.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.624+0000 7f847e395700 1 --2- 172.21.15.98:0/3930989040 >> v2:172.21.15.174:3300/0 conn(0x7f8478067800 0x7f84780c7a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:21.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.624+0000 7f847e395700 1 --2- 172.21.15.98:0/3930989040 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84780ca9f0 0x7f847806c430 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:21.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.624+0000 7f847e395700 1 -- 172.21.15.98:0/3930989040 >> 172.21.15.98:0/3930989040 conn(0x7f84780cdeb0 msgr2=0x7f84780c9e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:21.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.624+0000 7f847e395700 1 -- 172.21.15.98:0/3930989040 shutdown_connections 2023-10-06T05:07:21.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.624+0000 7f847e395700 1 -- 172.21.15.98:0/3930989040 wait complete. 2023-10-06T05:07:21.634 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.625+0000 7f847e395700 1 Processor -- start 2023-10-06T05:07:21.634 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.626+0000 7f847e395700 1 -- start start 2023-10-06T05:07:21.638 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.626+0000 7f847e395700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8478067800 0x7f847816e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:21.638 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.626+0000 7f847e395700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84780c7f60 0x7f847815f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:21.638 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.626+0000 7f847e395700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f84780ca9f0 0x7f84780b2190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:21.638 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.626+0000 7f847e395700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f847815ec70 con 0x7f84780c7f60 2023-10-06T05:07:21.638 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.626+0000 7f847e395700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f847816eeb0 con 0x7f84780ca9f0 2023-10-06T05:07:21.638 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.626+0000 7f847e395700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f847816f110 con 0x7f8478067800 2023-10-06T05:07:21.638 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.626+0000 7f84777fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84780c7f60 0x7f847815f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:21.638 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.626+0000 7f84777fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84780c7f60 0x7f847815f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47002/0 (socket says 172.21.15.98:47002) 2023-10-06T05:07:21.638 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.626+0000 7f84777fe700 1 -- 172.21.15.98:0/2296737600 learned_addr learned my addr 172.21.15.98:0/2296737600 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:21.638 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.627+0000 7f847c932700 1 --2- 172.21.15.98:0/2296737600 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f84780ca9f0 0x7f84780b2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:21.638 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.627+0000 7f8477fff700 1 --2- 172.21.15.98:0/2296737600 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8478067800 0x7f847816e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:21.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.627+0000 7f84777fe700 1 -- 172.21.15.98:0/2296737600 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8478067800 msgr2=0x7f847816e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:21.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.627+0000 7f84777fe700 1 --2- 172.21.15.98:0/2296737600 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8478067800 0x7f847816e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:21.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.627+0000 7f84777fe700 1 -- 172.21.15.98:0/2296737600 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f84780ca9f0 msgr2=0x7f84780b2190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:21.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.627+0000 7f84777fe700 1 --2- 172.21.15.98:0/2296737600 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f84780ca9f0 0x7f84780b2190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:21.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.627+0000 7f84777fe700 1 -- 172.21.15.98:0/2296737600 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f846c0094f0 con 0x7f84780c7f60 2023-10-06T05:07:21.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.627+0000 7f847c932700 1 --2- 172.21.15.98:0/2296737600 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f84780ca9f0 0x7f84780b2190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:07:21.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.627+0000 7f8477fff700 1 --2- 172.21.15.98:0/2296737600 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8478067800 0x7f847816e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:07:21.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.627+0000 7f84777fe700 1 --2- 172.21.15.98:0/2296737600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84780c7f60 0x7f847815f100 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f846c0030c0 tx=0x7f846c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:21.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.627+0000 7f845ffff700 1 -- 172.21.15.98:0/2296737600 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f846c011e60 con 0x7f84780c7f60 2023-10-06T05:07:21.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.627+0000 7f845ffff700 1 -- 172.21.15.98:0/2296737600 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f846c01b680 con 0x7f84780c7f60 2023-10-06T05:07:21.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.627+0000 7f845ffff700 1 -- 172.21.15.98:0/2296737600 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f846c01ba20 con 0x7f84780c7f60 2023-10-06T05:07:21.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.627+0000 7f847e395700 1 -- 172.21.15.98:0/2296737600 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f847815e340 con 0x7f84780c7f60 2023-10-06T05:07:21.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.627+0000 7f847e395700 1 -- 172.21.15.98:0/2296737600 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f847815e900 con 0x7f84780c7f60 2023-10-06T05:07:21.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.628+0000 7f847e395700 1 -- 172.21.15.98:0/2296737600 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f847815ea90 con 0x7f84780c7f60 2023-10-06T05:07:21.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.628+0000 7f845ffff700 1 -- 172.21.15.98:0/2296737600 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f846c01c450 con 0x7f84780c7f60 2023-10-06T05:07:21.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.629+0000 7f845ffff700 1 --2- 172.21.15.98:0/2296737600 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f8464066f30 0x7f84640693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:21.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.629+0000 7f845ffff700 1 -- 172.21.15.98:0/2296737600 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f846c086620 con 0x7f84780c7f60 2023-10-06T05:07:21.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.629+0000 7f8477fff700 1 --2- 172.21.15.98:0/2296737600 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f8464066f30 0x7f84640693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:21.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.629+0000 7f8477fff700 1 --2- 172.21.15.98:0/2296737600 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f8464066f30 0x7f84640693e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f8468006fd0 tx=0x7f8468008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:21.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.633+0000 7f845ffff700 1 -- 172.21.15.98:0/2296737600 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f846c053440 con 0x7f84780c7f60 2023-10-06T05:07:21.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.859+0000 7f847e395700 1 -- 172.21.15.98:0/2296737600 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f84780cbb60 con 0x7f84780c7f60 2023-10-06T05:07:21.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.860+0000 7f845ffff700 1 -- 172.21.15.98:0/2296737600 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4596 (secure 0 0 0) 0x7f846c056a60 con 0x7f84780c7f60 2023-10-06T05:07:21.862 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:07:21.862 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4285,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6834/925250632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":925250632},{"type":"v1","addr":"172.21.15.174:6834","nonce":925250632}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4288,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/1629449810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6833","nonce":1629449810},{"type":"v1","addr":"172.21.15.174:6835","nonce":1629449810}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4299,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/3961327990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3961327990},{"type":"v1","addr":"172.21.15.98:6836","nonce":3961327990}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-06T05:06:57.086683+0000","modified":"2023-10-06T05:07:09.773654+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.98:6837/2683239094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":2683239094},{"type":"v1","addr":"172.21.15.98:6837","nonce":2683239094}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-06T05:07:21.864 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.862+0000 7f847e395700 1 -- 172.21.15.98:0/2296737600 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f8464066f30 msgr2=0x7f84640693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:21.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.862+0000 7f847e395700 1 --2- 172.21.15.98:0/2296737600 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f8464066f30 0x7f84640693e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f8468006fd0 tx=0x7f8468008040 comp rx=0 tx=0).stop 2023-10-06T05:07:21.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.862+0000 7f847e395700 1 -- 172.21.15.98:0/2296737600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84780c7f60 msgr2=0x7f847815f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:21.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.862+0000 7f847e395700 1 --2- 172.21.15.98:0/2296737600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84780c7f60 0x7f847815f100 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f846c0030c0 tx=0x7f846c016000 comp rx=0 tx=0).stop 2023-10-06T05:07:21.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.862+0000 7f847e395700 1 -- 172.21.15.98:0/2296737600 shutdown_connections 2023-10-06T05:07:21.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.862+0000 7f847e395700 1 --2- 172.21.15.98:0/2296737600 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f84780ca9f0 0x7f84780b2190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:21.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.862+0000 7f847e395700 1 --2- 172.21.15.98:0/2296737600 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8478067800 0x7f847816e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:21.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.862+0000 7f847e395700 1 --2- 172.21.15.98:0/2296737600 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f8464066f30 0x7f84640693e0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:21.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.862+0000 7f847e395700 1 --2- 172.21.15.98:0/2296737600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84780c7f60 0x7f847815f100 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:21.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.862+0000 7f847e395700 1 -- 172.21.15.98:0/2296737600 >> 172.21.15.98:0/2296737600 conn(0x7f84780cdeb0 msgr2=0x7f847815fb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:21.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.862+0000 7f847e395700 1 -- 172.21.15.98:0/2296737600 shutdown_connections 2023-10-06T05:07:21.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.862+0000 7f847e395700 1 -- 172.21.15.98:0/2296737600 wait complete. 2023-10-06T05:07:21.866 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 8 2023-10-06T05:07:21.877 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T05:07:21.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.988+0000 7f23b0889700 1 Processor -- start 2023-10-06T05:07:21.991 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.988+0000 7f23b0889700 1 -- start start 2023-10-06T05:07:21.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.989+0000 7f23b0889700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f23ac0ca370 0x7f23ac158270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:21.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.989+0000 7f23b0889700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f23ac0adb60 0x7f23ac0ce190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:21.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.989+0000 7f23b0889700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23ac0ceb30 con 0x7f23ac0ce6d0 2023-10-06T05:07:21.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.989+0000 7f23b0889700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23ac0cbb90 con 0x7f23ac0ca370 2023-10-06T05:07:21.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.989+0000 7f23b0889700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23ac157de0 con 0x7f23ac0adb60 2023-10-06T05:07:21.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.989+0000 7f23aad9d700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f23ac0ce6d0 0x7f23ac15aed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:60920/0 (socket says 172.21.15.98:60920) 2023-10-06T05:07:21.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.989+0000 7f23aad9d700 1 -- 172.21.15.98:0/2103052218 learned_addr learned my addr 172.21.15.98:0/2103052218 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:07:21.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.989+0000 7f23aa59c700 1 --2- 172.21.15.98:0/2103052218 >> v2:172.21.15.174:3300/0 conn(0x7f23ac0ca370 0x7f23ac158270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:21.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.989+0000 7f23a1d9b700 1 --2- 172.21.15.98:0/2103052218 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f23ac0adb60 0x7f23ac0ce190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:21.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.989+0000 7f23a9d9b700 1 -- 172.21.15.98:0/2103052218 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2075497973 0 0) 0x7f23a4001490 con 0x7f23ac0ce6d0 2023-10-06T05:07:21.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.990+0000 7f23a9d9b700 1 -- 172.21.15.98:0/2103052218 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2398003570 con 0x7f23ac0ce6d0 2023-10-06T05:07:21.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.990+0000 7f23a9d9b700 1 -- 172.21.15.98:0/2103052218 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3086129040 0 0) 0x7f2398003570 con 0x7f23ac0ce6d0 2023-10-06T05:07:21.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.990+0000 7f23a9d9b700 1 -- 172.21.15.98:0/2103052218 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f23ac0adb60 msgr2=0x7f23ac0ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:21.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.990+0000 7f23a9d9b700 1 --2- 172.21.15.98:0/2103052218 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f23ac0adb60 0x7f23ac0ce190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:21.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.990+0000 7f23a9d9b700 1 -- 172.21.15.98:0/2103052218 >> v2:172.21.15.174:3300/0 conn(0x7f23ac0ca370 msgr2=0x7f23ac158270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:21.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.990+0000 7f23a9d9b700 1 --2- 172.21.15.98:0/2103052218 >> v2:172.21.15.174:3300/0 conn(0x7f23ac0ca370 0x7f23ac158270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:21.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.990+0000 7f23a9d9b700 1 -- 172.21.15.98:0/2103052218 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23ac0cb5d0 con 0x7f23ac0ce6d0 2023-10-06T05:07:21.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.990+0000 7f23a9d9b700 1 -- 172.21.15.98:0/2103052218 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1543991876 0 0) 0x7f23a4001980 con 0x7f23ac0ce6d0 2023-10-06T05:07:21.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.990+0000 7f23a9d9b700 1 -- 172.21.15.98:0/2103052218 >> v1:172.21.15.98:6789/0 conn(0x7f23ac0ce6d0 legacy=0x7f23ac15aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:21.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.990+0000 7f23a9d9b700 1 --2- 172.21.15.98:0/2103052218 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f23ac0adb60 0x7f2398003a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:21.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.990+0000 7f23a9d9b700 1 --2- 172.21.15.98:0/2103052218 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2398003fe0 0x7f2398004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:21.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.990+0000 7f23a9d9b700 1 --2- 172.21.15.98:0/2103052218 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2398004d50 0x7f23980051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:21.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.990+0000 7f23a9d9b700 1 -- 172.21.15.98:0/2103052218 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23ac0cbb90 con 0x7f2398003fe0 2023-10-06T05:07:21.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.990+0000 7f23a9d9b700 1 -- 172.21.15.98:0/2103052218 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23ac157de0 con 0x7f2398004d50 2023-10-06T05:07:21.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.990+0000 7f23a9d9b700 1 -- 172.21.15.98:0/2103052218 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f239800ba90 con 0x7f23ac0adb60 2023-10-06T05:07:21.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.990+0000 7f23a1d9b700 1 --2- 172.21.15.98:0/2103052218 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2398003fe0 0x7f2398004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:21.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.990+0000 7f23aa59c700 1 --2- 172.21.15.98:0/2103052218 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f23ac0adb60 0x7f2398003a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:21.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.990+0000 7f23aad9d700 1 --2- 172.21.15.98:0/2103052218 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2398004d50 0x7f23980051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:21.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.990+0000 7f23a1d9b700 1 -- 172.21.15.98:0/2103052218 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f23ac0adb60 msgr2=0x7f2398003a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:21.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.990+0000 7f23a1d9b700 1 --2- 172.21.15.98:0/2103052218 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f23ac0adb60 0x7f2398003a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:21.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.990+0000 7f23a1d9b700 1 -- 172.21.15.98:0/2103052218 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2398004d50 msgr2=0x7f23980051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:21.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.990+0000 7f23a1d9b700 1 --2- 172.21.15.98:0/2103052218 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2398004d50 0x7f23980051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:21.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.990+0000 7f23a1d9b700 1 -- 172.21.15.98:0/2103052218 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2398003570 con 0x7f2398003fe0 2023-10-06T05:07:21.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.990+0000 7f23a1d9b700 1 --2- 172.21.15.98:0/2103052218 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2398003fe0 0x7f2398004810 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f2394009c00 tx=0x7f23940091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9855c65eed505c43 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:21.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.990+0000 7f23a9d9b700 1 -- 172.21.15.98:0/2103052218 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f239400e6b0 con 0x7f2398003fe0 2023-10-06T05:07:22.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.990+0000 7f23a9d9b700 1 -- 172.21.15.98:0/2103052218 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f239400e810 con 0x7f2398003fe0 2023-10-06T05:07:22.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.990+0000 7f23a9d9b700 1 -- 172.21.15.98:0/2103052218 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f239400ebd0 con 0x7f2398003fe0 2023-10-06T05:07:22.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.991+0000 7f23b0889700 1 -- 172.21.15.98:0/2103052218 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2398003fe0 msgr2=0x7f2398004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:22.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.991+0000 7f23b0889700 1 --2- 172.21.15.98:0/2103052218 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2398003fe0 0x7f2398004810 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f2394009c00 tx=0x7f23940091a0 comp rx=0 tx=0).stop 2023-10-06T05:07:22.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.991+0000 7f23aad9d700 1 -- 172.21.15.98:0/2103052218 reap_dead start 2023-10-06T05:07:22.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.991+0000 7f23b0889700 1 -- 172.21.15.98:0/2103052218 shutdown_connections 2023-10-06T05:07:22.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.991+0000 7f23b0889700 1 -- 172.21.15.98:0/2103052218 >> 172.21.15.98:0/2103052218 conn(0x7f23ac0707d0 msgr2=0x7f23ac070bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:22.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.991+0000 7f23b0889700 1 -- 172.21.15.98:0/2103052218 shutdown_connections 2023-10-06T05:07:22.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.991+0000 7f23b0889700 1 -- 172.21.15.98:0/2103052218 wait complete. 2023-10-06T05:07:22.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.992+0000 7f23b0889700 1 Processor -- start 2023-10-06T05:07:22.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.992+0000 7f23b0889700 1 -- start start 2023-10-06T05:07:22.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.993+0000 7f23b0889700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2398003fe0 0x7f23ac153b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:22.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.993+0000 7f23b0889700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23ac0c38a0 0x7f23ac15c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:22.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.993+0000 7f23b0889700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f23ac0c3d10 0x7f23ac15c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:22.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.993+0000 7f23b0889700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23ac0b2d60 con 0x7f23ac0c38a0 2023-10-06T05:07:22.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.993+0000 7f23b0889700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23ac0b3040 con 0x7f2398003fe0 2023-10-06T05:07:22.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.993+0000 7f23b0889700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23ac0c1e90 con 0x7f23ac0c3d10 2023-10-06T05:07:22.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.993+0000 7f23a1d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23ac0c38a0 0x7f23ac15c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:22.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.993+0000 7f23a1d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23ac0c38a0 0x7f23ac15c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47008/0 (socket says 172.21.15.98:47008) 2023-10-06T05:07:22.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.993+0000 7f23a1d9b700 1 -- 172.21.15.98:0/994737436 learned_addr learned my addr 172.21.15.98:0/994737436 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:22.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.993+0000 7f23aad9d700 1 --2- 172.21.15.98:0/994737436 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f23ac0c3d10 0x7f23ac15c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:22.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.993+0000 7f23aa59c700 1 --2- 172.21.15.98:0/994737436 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2398003fe0 0x7f23ac153b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:22.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.993+0000 7f23a1d9b700 1 -- 172.21.15.98:0/994737436 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f23ac0c3d10 msgr2=0x7f23ac15c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:22.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.993+0000 7f23a1d9b700 1 --2- 172.21.15.98:0/994737436 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f23ac0c3d10 0x7f23ac15c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:22.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.993+0000 7f23a1d9b700 1 -- 172.21.15.98:0/994737436 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2398003fe0 msgr2=0x7f23ac153b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:22.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.993+0000 7f23a1d9b700 1 --2- 172.21.15.98:0/994737436 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2398003fe0 0x7f23ac153b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:22.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.993+0000 7f23a1d9b700 1 -- 172.21.15.98:0/994737436 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23ac0c2170 con 0x7f23ac0c38a0 2023-10-06T05:07:22.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.994+0000 7f23a1d9b700 1 --2- 172.21.15.98:0/994737436 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23ac0c38a0 0x7f23ac15c450 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f23940091e0 tx=0x7f239401a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:22.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.994+0000 7f23a2ffd700 1 -- 172.21.15.98:0/994737436 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2394022070 con 0x7f23ac0c38a0 2023-10-06T05:07:22.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.994+0000 7f23a2ffd700 1 -- 172.21.15.98:0/994737436 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f239400e980 con 0x7f23ac0c38a0 2023-10-06T05:07:22.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.994+0000 7f23b0889700 1 -- 172.21.15.98:0/994737436 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23ac1540c0 con 0x7f23ac0c38a0 2023-10-06T05:07:22.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.994+0000 7f23a2ffd700 1 -- 172.21.15.98:0/994737436 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2394027470 con 0x7f23ac0c38a0 2023-10-06T05:07:22.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.994+0000 7f23b0889700 1 -- 172.21.15.98:0/994737436 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23ac0c2850 con 0x7f23ac0c38a0 2023-10-06T05:07:22.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.995+0000 7f23b0889700 1 -- 172.21.15.98:0/994737436 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23ac064a90 con 0x7f23ac0c38a0 2023-10-06T05:07:22.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.995+0000 7f23a2ffd700 1 -- 172.21.15.98:0/994737436 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f239401c450 con 0x7f23ac0c38a0 2023-10-06T05:07:22.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.995+0000 7f23a2ffd700 1 --2- 172.21.15.98:0/994737436 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f238c066f30 0x7f238c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:22.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.995+0000 7f23a2ffd700 1 -- 172.21.15.98:0/994737436 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f2394086580 con 0x7f23ac0c38a0 2023-10-06T05:07:22.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.995+0000 7f23aa59c700 1 --2- 172.21.15.98:0/994737436 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f238c066f30 0x7f238c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:22.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:21.996+0000 7f23aa59c700 1 --2- 172.21.15.98:0/994737436 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f238c066f30 0x7f238c0693e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f239c006f00 tx=0x7f239c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:22.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.000+0000 7f23a2ffd700 1 -- 172.21.15.98:0/994737436 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2394053420 con 0x7f23ac0c38a0 2023-10-06T05:07:22.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.187+0000 7f23b0889700 1 -- 172.21.15.98:0/994737436 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f23ac0b2d60 con 0x7f23ac0c38a0 2023-10-06T05:07:22.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.187+0000 7f23a2ffd700 1 -- 172.21.15.98:0/994737436 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+18707 (secure 0 0 0) 0x7f23940569c0 con 0x7f23ac0c38a0 2023-10-06T05:07:22.190 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:07:22.190 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":29,"fsid":"f50c5df5-cad5-47a4-9bb1-8c791d4311ec","created":"2023-10-06T05:06:32.971175+0000","modified":"2023-10-06T05:07:02.131672+0000","last_up_change":"2023-10-06T05:06:44.016489+0000","last_in_change":"2023-10-06T05:06:35.544511+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-06T05:06:44.559671+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-06T05:06:51.432161+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-10-06T05:06:55.589057+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-10-06T05:06:56.041687+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-10-06T05:06:59.754099+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d9495d95-da75-4c3b-8927-e9bae6a8b093","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":90627},{"type":"v1","addr":"172.21.15.98:6801","nonce":90627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":90627},{"type":"v1","addr":"172.21.15.98:6803","nonce":90627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":90627},{"type":"v1","addr":"172.21.15.98:6807","nonce":90627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":90627},{"type":"v1","addr":"172.21.15.98:6805","nonce":90627}]},"public_addr":"172.21.15.98:6801/90627","cluster_addr":"172.21.15.98:6803/90627","heartbeat_back_addr":"172.21.15.98:6807/90627","heartbeat_front_addr":"172.21.15.98:6805/90627","state":["exists","up"]},{"osd":1,"uuid":"f25ba5be-d65e-4707-a134-154092deb754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":90628},{"type":"v1","addr":"172.21.15.98:6809","nonce":90628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":90628},{"type":"v1","addr":"172.21.15.98:6811","nonce":90628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":90628},{"type":"v1","addr":"172.21.15.98:6815","nonce":90628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":90628},{"type":"v1","addr":"172.21.15.98:6813","nonce":90628}]},"public_addr":"172.21.15.98:6809/90628","cluster_addr":"172.21.15.98:6811/90628","heartbeat_back_addr":"172.21.15.98:6815/90628","heartbeat_front_addr":"172.21.15.98:6813/90628","state":["exists","up"]},{"osd":2,"uuid":"5ab9e028-945c-43e2-8c95-3b0b4cf38e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":90662},{"type":"v1","addr":"172.21.15.98:6818","nonce":90662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":90662},{"type":"v1","addr":"172.21.15.98:6822","nonce":90662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":90662},{"type":"v1","addr":"172.21.15.98:6830","nonce":90662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":90662},{"type":"v1","addr":"172.21.15.98:6826","nonce":90662}]},"public_addr":"172.21.15.98:6818/90662","cluster_addr":"172.21.15.98:6822/90662","heartbeat_back_addr":"172.21.15.98:6830/90662","heartbeat_front_addr":"172.21.15.98:6826/90662","state":["exists","up"]},{"osd":3,"uuid":"3bb803ff-1738-46bf-be62-2267bf378822","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6817","nonce":90663},{"type":"v1","addr":"172.21.15.98:6819","nonce":90663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6821","nonce":90663},{"type":"v1","addr":"172.21.15.98:6823","nonce":90663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6829","nonce":90663},{"type":"v1","addr":"172.21.15.98:6831","nonce":90663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6825","nonce":90663},{"type":"v1","addr":"172.21.15.98:6827","nonce":90663}]},"public_addr":"172.21.15.98:6819/90663","cluster_addr":"172.21.15.98:6823/90663","heartbeat_back_addr":"172.21.15.98:6831/90663","heartbeat_front_addr":"172.21.15.98:6827/90663","state":["exists","up"]},{"osd":4,"uuid":"7d57a4c9-6462-43f6-b64a-7f9e9b4353d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":90847},{"type":"v1","addr":"172.21.15.174:6801","nonce":90847}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":90847},{"type":"v1","addr":"172.21.15.174:6803","nonce":90847}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":90847},{"type":"v1","addr":"172.21.15.174:6807","nonce":90847}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":90847},{"type":"v1","addr":"172.21.15.174:6805","nonce":90847}]},"public_addr":"172.21.15.174:6801/90847","cluster_addr":"172.21.15.174:6803/90847","heartbeat_back_addr":"172.21.15.174:6807/90847","heartbeat_front_addr":"172.21.15.174:6805/90847","state":["exists","up"]},{"osd":5,"uuid":"83287ea4-e44d-4af9-bc77-36d47c60d4a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":90848},{"type":"v1","addr":"172.21.15.174:6809","nonce":90848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":90848},{"type":"v1","addr":"172.21.15.174:6811","nonce":90848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":90848},{"type":"v1","addr":"172.21.15.174:6815","nonce":90848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":90848},{"type":"v1","addr":"172.21.15.174:6813","nonce":90848}]},"public_addr":"172.21.15.174:6809/90848","cluster_addr":"172.21.15.174:6811/90848","heartbeat_back_addr":"172.21.15.174:6815/90848","heartbeat_front_addr":"172.21.15.174:6813/90848","state":["exists","up"]},{"osd":6,"uuid":"496a6272-892e-43c4-a145-206623c22f99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":90877},{"type":"v1","addr":"172.21.15.174:6817","nonce":90877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":90877},{"type":"v1","addr":"172.21.15.174:6819","nonce":90877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":90877},{"type":"v1","addr":"172.21.15.174:6823","nonce":90877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":90877},{"type":"v1","addr":"172.21.15.174:6821","nonce":90877}]},"public_addr":"172.21.15.174:6817/90877","cluster_addr":"172.21.15.174:6819/90877","heartbeat_back_addr":"172.21.15.174:6823/90877","heartbeat_front_addr":"172.21.15.174:6821/90877","state":["exists","up"]},{"osd":7,"uuid":"49967e34-8b07-4bab-a13e-0c16d6a0f3a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":90878},{"type":"v1","addr":"172.21.15.174:6825","nonce":90878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":90878},{"type":"v1","addr":"172.21.15.174:6827","nonce":90878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":90878},{"type":"v1","addr":"172.21.15.174:6831","nonce":90878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":90878},{"type":"v1","addr":"172.21.15.174:6829","nonce":90878}]},"public_addr":"172.21.15.174:6825/90878","cluster_addr":"172.21.15.174:6827/90878","heartbeat_back_addr":"172.21.15.174:6831/90878","heartbeat_front_addr":"172.21.15.174:6829/90878","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.446479+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.514266+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.794108+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.810894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.731919+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.717756+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.737435+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.508682+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T05:07:22.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.190+0000 7f23b0889700 1 -- 172.21.15.98:0/994737436 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f238c066f30 msgr2=0x7f238c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:22.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.190+0000 7f23b0889700 1 --2- 172.21.15.98:0/994737436 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f238c066f30 0x7f238c0693e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f239c006f00 tx=0x7f239c008040 comp rx=0 tx=0).stop 2023-10-06T05:07:22.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.190+0000 7f23b0889700 1 -- 172.21.15.98:0/994737436 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23ac0c38a0 msgr2=0x7f23ac15c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:22.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.190+0000 7f23b0889700 1 --2- 172.21.15.98:0/994737436 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23ac0c38a0 0x7f23ac15c450 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f23940091e0 tx=0x7f239401a000 comp rx=0 tx=0).stop 2023-10-06T05:07:22.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.190+0000 7f23b0889700 1 -- 172.21.15.98:0/994737436 shutdown_connections 2023-10-06T05:07:22.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.190+0000 7f23b0889700 1 --2- 172.21.15.98:0/994737436 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f23ac0c3d10 0x7f23ac15c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:22.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.190+0000 7f23b0889700 1 --2- 172.21.15.98:0/994737436 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2398003fe0 0x7f23ac153b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:22.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.190+0000 7f23b0889700 1 --2- 172.21.15.98:0/994737436 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f238c066f30 0x7f238c0693e0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:22.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.190+0000 7f23b0889700 1 --2- 172.21.15.98:0/994737436 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23ac0c38a0 0x7f23ac15c450 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:22.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.190+0000 7f23b0889700 1 -- 172.21.15.98:0/994737436 >> 172.21.15.98:0/994737436 conn(0x7f23ac0707d0 msgr2=0x7f23ac156100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:22.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.190+0000 7f23b0889700 1 -- 172.21.15.98:0/994737436 shutdown_connections 2023-10-06T05:07:22.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.190+0000 7f23b0889700 1 -- 172.21.15.98:0/994737436 wait complete. 2023-10-06T05:07:22.204 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-06T05:07:22.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.315+0000 7f816ec80700 1 Processor -- start 2023-10-06T05:07:22.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.315+0000 7f816ec80700 1 -- start start 2023-10-06T05:07:22.320 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.316+0000 7f816ec80700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f81680b49a0 0x7f81680aac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:22.320 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.316+0000 7f816ec80700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81680b0c60 con 0x7f81680b0760 2023-10-06T05:07:22.320 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.316+0000 7f816ec80700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81680b0f40 con 0x7f8168159560 2023-10-06T05:07:22.320 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.316+0000 7f816ec80700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81680b1220 con 0x7f81680b49a0 2023-10-06T05:07:22.320 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.316+0000 7f816d21d700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f81680b0760 0x7f81680b1bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:60928/0 (socket says 172.21.15.98:60928) 2023-10-06T05:07:22.320 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.316+0000 7f816d21d700 1 -- 172.21.15.98:0/556112091 learned_addr learned my addr 172.21.15.98:0/556112091 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:07:22.321 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.317+0000 7f8167fff700 1 --2- 172.21.15.98:0/556112091 >> v2:172.21.15.174:3300/0 conn(0x7f81680b49a0 0x7f81680aac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:22.321 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.317+0000 7f81677fe700 1 -- 172.21.15.98:0/556112091 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3438183564 0 0) 0x7f81680b0c60 con 0x7f81680b0760 2023-10-06T05:07:22.321 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.317+0000 7f81677fe700 1 -- 172.21.15.98:0/556112091 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8150003570 con 0x7f81680b0760 2023-10-06T05:07:22.321 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.317+0000 7f81677fe700 1 -- 172.21.15.98:0/556112091 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2680107976 0 0) 0x7f81680b0f40 con 0x7f8168159560 2023-10-06T05:07:22.321 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.317+0000 7f81677fe700 1 -- 172.21.15.98:0/556112091 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81680b0c60 con 0x7f8168159560 2023-10-06T05:07:22.321 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.317+0000 7f8167fff700 1 -- 172.21.15.98:0/556112091 >> v1:172.21.15.174:6789/0 conn(0x7f8168159560 legacy=0x7f81680aa530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:22.321 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.317+0000 7f81677fe700 1 -- 172.21.15.98:0/556112091 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1253224806 0 0) 0x7f8150003570 con 0x7f81680b0760 2023-10-06T05:07:22.321 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.317+0000 7f8167fff700 1 -- 172.21.15.98:0/556112091 >> v1:172.21.15.98:6789/0 conn(0x7f81680b0760 legacy=0x7f81680b1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:22.321 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.317+0000 7f8167fff700 1 -- 172.21.15.98:0/556112091 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81680b1500 con 0x7f81680b49a0 2023-10-06T05:07:22.321 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.317+0000 7f8167fff700 1 --2- 172.21.15.98:0/556112091 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81680b49a0 0x7f81680aac40 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f815c009910 tx=0x7f815c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ac57f3f55345e370 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:22.321 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.317+0000 7f81677fe700 1 -- 172.21.15.98:0/556112091 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8160001470 con 0x7f81680b49a0 2023-10-06T05:07:22.322 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.317+0000 7f81677fe700 1 -- 172.21.15.98:0/556112091 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f815c011420 con 0x7f81680b49a0 2023-10-06T05:07:22.322 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.317+0000 7f81677fe700 1 -- 172.21.15.98:0/556112091 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f815c0117c0 con 0x7f81680b49a0 2023-10-06T05:07:22.322 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.318+0000 7f816ec80700 1 -- 172.21.15.98:0/556112091 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81680b49a0 msgr2=0x7f81680aac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:22.322 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.318+0000 7f816ec80700 1 --2- 172.21.15.98:0/556112091 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81680b49a0 0x7f81680aac40 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f815c009910 tx=0x7f815c0091a0 comp rx=0 tx=0).stop 2023-10-06T05:07:22.322 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.318+0000 7f816ec80700 1 -- 172.21.15.98:0/556112091 shutdown_connections 2023-10-06T05:07:22.322 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.318+0000 7f816ec80700 1 --2- 172.21.15.98:0/556112091 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81680b49a0 0x7f81680aac40 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:22.322 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.318+0000 7f816ec80700 1 -- 172.21.15.98:0/556112091 >> 172.21.15.98:0/556112091 conn(0x7f81680997f0 msgr2=0x7f816806f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:22.322 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.318+0000 7f816ec80700 1 -- 172.21.15.98:0/556112091 shutdown_connections 2023-10-06T05:07:22.322 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.318+0000 7f816ec80700 1 -- 172.21.15.98:0/556112091 wait complete. 2023-10-06T05:07:22.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.319+0000 7f816ec80700 1 Processor -- start 2023-10-06T05:07:22.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.319+0000 7f816ec80700 1 -- start start 2023-10-06T05:07:22.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.320+0000 7f816ec80700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81680b0760 0x7f81680ab290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:22.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.320+0000 7f816ec80700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f81680b49a0 0x7f8168156c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:22.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.320+0000 7f816ec80700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8168159560 0x7f81680ac970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:22.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.320+0000 7f816ec80700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f816815bab0 con 0x7f8168159560 2023-10-06T05:07:22.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.320+0000 7f816ec80700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8168157170 con 0x7f81680b0760 2023-10-06T05:07:22.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.320+0000 7f816ec80700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8168169830 con 0x7f81680b49a0 2023-10-06T05:07:22.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.320+0000 7f816d21d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8168159560 0x7f81680ac970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:22.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.320+0000 7f816d21d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8168159560 0x7f81680ac970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47022/0 (socket says 172.21.15.98:47022) 2023-10-06T05:07:22.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.320+0000 7f816d21d700 1 -- 172.21.15.98:0/184474185 learned_addr learned my addr 172.21.15.98:0/184474185 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:22.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.320+0000 7f816ca1c700 1 --2- 172.21.15.98:0/184474185 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81680b0760 0x7f81680ab290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:22.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.320+0000 7f8167fff700 1 --2- 172.21.15.98:0/184474185 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f81680b49a0 0x7f8168156c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:22.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.320+0000 7f816d21d700 1 -- 172.21.15.98:0/184474185 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f81680b49a0 msgr2=0x7f8168156c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:22.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.320+0000 7f816d21d700 1 --2- 172.21.15.98:0/184474185 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f81680b49a0 0x7f8168156c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:22.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.320+0000 7f816d21d700 1 -- 172.21.15.98:0/184474185 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81680b0760 msgr2=0x7f81680ab290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:22.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.320+0000 7f816d21d700 1 --2- 172.21.15.98:0/184474185 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81680b0760 0x7f81680ab290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:22.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.320+0000 7f816d21d700 1 -- 172.21.15.98:0/184474185 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f815c009420 con 0x7f8168159560 2023-10-06T05:07:22.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.320+0000 7f816d21d700 1 --2- 172.21.15.98:0/184474185 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8168159560 0x7f81680ac970 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f816000cb70 tx=0x7f8160001b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:22.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.320+0000 7f8164ff9700 1 -- 172.21.15.98:0/184474185 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8160007360 con 0x7f8168159560 2023-10-06T05:07:22.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.320+0000 7f8164ff9700 1 -- 172.21.15.98:0/184474185 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8160007500 con 0x7f8168159560 2023-10-06T05:07:22.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.320+0000 7f816ec80700 1 -- 172.21.15.98:0/184474185 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81680ab870 con 0x7f8168159560 2023-10-06T05:07:22.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.320+0000 7f8164ff9700 1 -- 172.21.15.98:0/184474185 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8160004a80 con 0x7f8168159560 2023-10-06T05:07:22.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.320+0000 7f816ec80700 1 -- 172.21.15.98:0/184474185 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81680abe30 con 0x7f8168159560 2023-10-06T05:07:22.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.321+0000 7f816ec80700 1 -- 172.21.15.98:0/184474185 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f816815bc20 con 0x7f8168159560 2023-10-06T05:07:22.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.322+0000 7f8164ff9700 1 -- 172.21.15.98:0/184474185 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f816000a590 con 0x7f8168159560 2023-10-06T05:07:22.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.322+0000 7f8164ff9700 1 --2- 172.21.15.98:0/184474185 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f8154066f30 0x7f81540693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:22.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.322+0000 7f8164ff9700 1 -- 172.21.15.98:0/184474185 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f8160084300 con 0x7f8168159560 2023-10-06T05:07:22.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.322+0000 7f816ca1c700 1 --2- 172.21.15.98:0/184474185 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f8154066f30 0x7f81540693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:22.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.323+0000 7f816ca1c700 1 --2- 172.21.15.98:0/184474185 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f8154066f30 0x7f81540693e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f815800e160 tx=0x7f8158009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:22.328 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.326+0000 7f8164ff9700 1 -- 172.21.15.98:0/184474185 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8160054c30 con 0x7f8168159560 2023-10-06T05:07:22.554 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.552+0000 7f816ec80700 1 -- 172.21.15.98:0/184474185 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8168064a90 con 0x7f8168159560 2023-10-06T05:07:22.555 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.553+0000 7f8164ff9700 1 -- 172.21.15.98:0/184474185 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4596 (secure 0 0 0) 0x7f81600547c0 con 0x7f8168159560 2023-10-06T05:07:22.555 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:07:22.555 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4285,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6834/925250632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":925250632},{"type":"v1","addr":"172.21.15.174:6834","nonce":925250632}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4288,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/1629449810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6833","nonce":1629449810},{"type":"v1","addr":"172.21.15.174:6835","nonce":1629449810}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4299,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/3961327990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3961327990},{"type":"v1","addr":"172.21.15.98:6836","nonce":3961327990}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-06T05:06:57.086683+0000","modified":"2023-10-06T05:07:09.773654+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.98:6837/2683239094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":2683239094},{"type":"v1","addr":"172.21.15.98:6837","nonce":2683239094}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-06T05:07:22.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.555+0000 7f816ec80700 1 -- 172.21.15.98:0/184474185 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f8154066f30 msgr2=0x7f81540693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:22.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.555+0000 7f816ec80700 1 --2- 172.21.15.98:0/184474185 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f8154066f30 0x7f81540693e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f815800e160 tx=0x7f8158009040 comp rx=0 tx=0).stop 2023-10-06T05:07:22.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.555+0000 7f816ec80700 1 -- 172.21.15.98:0/184474185 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8168159560 msgr2=0x7f81680ac970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:22.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.555+0000 7f816ec80700 1 --2- 172.21.15.98:0/184474185 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8168159560 0x7f81680ac970 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f816000cb70 tx=0x7f8160001b00 comp rx=0 tx=0).stop 2023-10-06T05:07:22.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.555+0000 7f816ec80700 1 -- 172.21.15.98:0/184474185 shutdown_connections 2023-10-06T05:07:22.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.555+0000 7f816ec80700 1 --2- 172.21.15.98:0/184474185 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f8154066f30 0x7f81540693e0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:22.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.555+0000 7f816ec80700 1 --2- 172.21.15.98:0/184474185 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8168159560 0x7f81680ac970 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:22.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.555+0000 7f816ec80700 1 --2- 172.21.15.98:0/184474185 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81680b0760 0x7f81680ab290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:22.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.555+0000 7f816ec80700 1 --2- 172.21.15.98:0/184474185 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f81680b49a0 0x7f8168156c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:22.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.555+0000 7f816ec80700 1 -- 172.21.15.98:0/184474185 >> 172.21.15.98:0/184474185 conn(0x7f81680997f0 msgr2=0x7f8168158a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:22.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.555+0000 7f816ec80700 1 -- 172.21.15.98:0/184474185 shutdown_connections 2023-10-06T05:07:22.558 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:22.555+0000 7f816ec80700 1 -- 172.21.15.98:0/184474185 wait complete. 2023-10-06T05:07:22.559 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 8 2023-10-06T05:07:22.973 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-06T05:06:57.086683+0000', 'modified': '2023-10-06T05:07:09.773654+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4302}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4302': {'gid': 4302, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.98:6837/2683239094', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 2683239094}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 2683239094}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [4, 5], 'metadata_pool': 3, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2023-10-06T05:07:22.973 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-06T05:07:22.973 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-10-06T05:07:23.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.087+0000 7efc45a71700 1 Processor -- start 2023-10-06T05:07:23.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.087+0000 7efc45a71700 1 -- start start 2023-10-06T05:07:23.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.088+0000 7efc45a71700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7efc40159560 0x7efc400aa530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:23.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.088+0000 7efc45a71700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efc400b49a0 0x7efc400aaa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:23.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.088+0000 7efc45a71700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7efc400b0760 0x7efc400b3890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:23.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.088+0000 7efc45a71700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc400b0c60 con 0x7efc40159560 2023-10-06T05:07:23.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.088+0000 7efc45a71700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc400b0f40 con 0x7efc400b0760 2023-10-06T05:07:23.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.088+0000 7efc45a71700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc400b1220 con 0x7efc400b49a0 2023-10-06T05:07:23.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.088+0000 7efc3f7fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7efc40159560 0x7efc400aa530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:23.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.088+0000 7efc3f7fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7efc40159560 0x7efc400aa530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47038/0 (socket says 172.21.15.98:47038) 2023-10-06T05:07:23.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.088+0000 7efc3f7fe700 1 -- 172.21.15.98:0/4072173362 learned_addr learned my addr 172.21.15.98:0/4072173362 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:23.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.088+0000 7efc3ffff700 1 --2- 172.21.15.98:0/4072173362 >> v2:172.21.15.174:3300/0 conn(0x7efc400b0760 0x7efc400b3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:23.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.088+0000 7efc3effd700 1 --2- 172.21.15.98:0/4072173362 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efc400b49a0 0x7efc400aaa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:23.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.088+0000 7efc3f7fe700 1 -- 172.21.15.98:0/4072173362 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efc400b49a0 msgr2=0x7efc400aaa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:23.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.088+0000 7efc3f7fe700 1 --2- 172.21.15.98:0/4072173362 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efc400b49a0 0x7efc400aaa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:23.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.088+0000 7efc3f7fe700 1 -- 172.21.15.98:0/4072173362 >> v2:172.21.15.174:3300/0 conn(0x7efc400b0760 msgr2=0x7efc400b3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:23.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.088+0000 7efc3f7fe700 1 --2- 172.21.15.98:0/4072173362 >> v2:172.21.15.174:3300/0 conn(0x7efc400b0760 0x7efc400b3890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:23.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.088+0000 7efc3f7fe700 1 -- 172.21.15.98:0/4072173362 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc400b1500 con 0x7efc40159560 2023-10-06T05:07:23.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.088+0000 7efc3effd700 1 --2- 172.21.15.98:0/4072173362 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efc400b49a0 0x7efc400aaa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:07:23.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.089+0000 7efc3f7fe700 1 --2- 172.21.15.98:0/4072173362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc40159560 0x7efc400aa530 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7efc400d0460 tx=0x7efc30009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c632035fe23233e4 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:23.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.089+0000 7efc3e7fc700 1 -- 172.21.15.98:0/4072173362 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc30005c10 con 0x7efc40159560 2023-10-06T05:07:23.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.089+0000 7efc3e7fc700 1 -- 172.21.15.98:0/4072173362 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc30005d70 con 0x7efc40159560 2023-10-06T05:07:23.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.089+0000 7efc3e7fc700 1 -- 172.21.15.98:0/4072173362 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc3000b880 con 0x7efc40159560 2023-10-06T05:07:23.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.089+0000 7efc45a71700 1 -- 172.21.15.98:0/4072173362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc40159560 msgr2=0x7efc400aa530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:23.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.089+0000 7efc45a71700 1 --2- 172.21.15.98:0/4072173362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc40159560 0x7efc400aa530 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7efc400d0460 tx=0x7efc30009220 comp rx=0 tx=0).stop 2023-10-06T05:07:23.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.089+0000 7efc45a71700 1 -- 172.21.15.98:0/4072173362 shutdown_connections 2023-10-06T05:07:23.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.089+0000 7efc45a71700 1 --2- 172.21.15.98:0/4072173362 >> v2:172.21.15.174:3300/0 conn(0x7efc400b0760 0x7efc400b3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:23.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.089+0000 7efc45a71700 1 --2- 172.21.15.98:0/4072173362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc40159560 0x7efc400aa530 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:23.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.089+0000 7efc45a71700 1 --2- 172.21.15.98:0/4072173362 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efc400b49a0 0x7efc400aaa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:23.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.089+0000 7efc45a71700 1 -- 172.21.15.98:0/4072173362 >> 172.21.15.98:0/4072173362 conn(0x7efc400997f0 msgr2=0x7efc4006f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:23.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.089+0000 7efc45a71700 1 -- 172.21.15.98:0/4072173362 shutdown_connections 2023-10-06T05:07:23.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.089+0000 7efc45a71700 1 -- 172.21.15.98:0/4072173362 wait complete. 2023-10-06T05:07:23.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.090+0000 7efc45a71700 1 Processor -- start 2023-10-06T05:07:23.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.090+0000 7efc45a71700 1 -- start start 2023-10-06T05:07:23.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.091+0000 7efc45a71700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efc400b0760 0x7efc4016a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:23.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.091+0000 7efc45a71700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efc400b49a0 0x7efc4015b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:23.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.091+0000 7efc45a71700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc40159560 0x7efc400695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:23.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.091+0000 7efc45a71700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc4015be70 con 0x7efc40159560 2023-10-06T05:07:23.102 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.091+0000 7efc45a71700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc40169f30 con 0x7efc400b49a0 2023-10-06T05:07:23.102 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.091+0000 7efc45a71700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc4009dae0 con 0x7efc400b0760 2023-10-06T05:07:23.102 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.091+0000 7efc3ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc40159560 0x7efc400695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:23.102 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.091+0000 7efc3ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc40159560 0x7efc400695e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47054/0 (socket says 172.21.15.98:47054) 2023-10-06T05:07:23.102 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.091+0000 7efc3ffff700 1 -- 172.21.15.98:0/2064216499 learned_addr learned my addr 172.21.15.98:0/2064216499 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:23.102 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.091+0000 7efc3f7fe700 1 --2- 172.21.15.98:0/2064216499 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efc400b0760 0x7efc4016a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:23.102 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.091+0000 7efc3effd700 1 --2- 172.21.15.98:0/2064216499 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efc400b49a0 0x7efc4015b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:23.102 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.092+0000 7efc3ffff700 1 -- 172.21.15.98:0/2064216499 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efc400b0760 msgr2=0x7efc4016a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:23.102 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.092+0000 7efc3ffff700 1 --2- 172.21.15.98:0/2064216499 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efc400b0760 0x7efc4016a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:23.102 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.092+0000 7efc3ffff700 1 -- 172.21.15.98:0/2064216499 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efc400b49a0 msgr2=0x7efc4015b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:23.102 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.092+0000 7efc3ffff700 1 --2- 172.21.15.98:0/2064216499 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efc400b49a0 0x7efc4015b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:23.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.092+0000 7efc3ffff700 1 -- 172.21.15.98:0/2064216499 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc30009480 con 0x7efc40159560 2023-10-06T05:07:23.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.092+0000 7efc3ffff700 1 --2- 172.21.15.98:0/2064216499 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc40159560 0x7efc400695e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7efc38009fa0 tx=0x7efc38009900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:23.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.092+0000 7efc277fe700 1 -- 172.21.15.98:0/2064216499 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc3800ebc0 con 0x7efc40159560 2023-10-06T05:07:23.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.092+0000 7efc277fe700 1 -- 172.21.15.98:0/2064216499 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc3800ed20 con 0x7efc40159560 2023-10-06T05:07:23.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.092+0000 7efc45a71700 1 -- 172.21.15.98:0/2064216499 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc4009ddc0 con 0x7efc40159560 2023-10-06T05:07:23.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.092+0000 7efc277fe700 1 -- 172.21.15.98:0/2064216499 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc38014670 con 0x7efc40159560 2023-10-06T05:07:23.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.092+0000 7efc45a71700 1 -- 172.21.15.98:0/2064216499 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc4009d760 con 0x7efc40159560 2023-10-06T05:07:23.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.093+0000 7efc277fe700 1 -- 172.21.15.98:0/2064216499 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7efc380149e0 con 0x7efc40159560 2023-10-06T05:07:23.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.094+0000 7efc277fe700 1 --2- 172.21.15.98:0/2064216499 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7efc2c066fb0 0x7efc2c069460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:23.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.094+0000 7efc277fe700 1 -- 172.21.15.98:0/2064216499 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7efc38084fd0 con 0x7efc40159560 2023-10-06T05:07:23.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.094+0000 7efc3f7fe700 1 --2- 172.21.15.98:0/2064216499 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7efc2c066fb0 0x7efc2c069460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:23.104 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.094+0000 7efc3f7fe700 1 --2- 172.21.15.98:0/2064216499 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7efc2c066fb0 0x7efc2c069460 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7efc30009260 tx=0x7efc30016000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:23.104 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.096+0000 7efc4ed69b80 1 Processor -- start 2023-10-06T05:07:23.104 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.096+0000 7efc4ed69b80 1 -- start start 2023-10-06T05:07:23.104 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.096+0000 7efc4ed69b80 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55b737fb8a90 0x55b737fb8e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:23.104 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.096+0000 7efc4ed69b80 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b737f749a0 0x55b737f71580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:23.104 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.096+0000 7efc4ed69b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b737f72bd0 0x55b737f71ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:23.104 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.096+0000 7efc4ed69b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b737ef1c80 con 0x55b737f72bd0 2023-10-06T05:07:23.104 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.096+0000 7efc4ed69b80 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b737ec8a70 con 0x55b737f749a0 2023-10-06T05:07:23.104 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.096+0000 7efc4ed69b80 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b737ec6270 con 0x55b737fb8a90 2023-10-06T05:07:23.104 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.097+0000 7efc3ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b737f72bd0 0x55b737f71ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:23.104 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.097+0000 7efc3ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b737f72bd0 0x55b737f71ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47060/0 (socket says 172.21.15.98:47060) 2023-10-06T05:07:23.105 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.097+0000 7efc3ffff700 1 -- 172.21.15.98:0/2723738758 learned_addr learned my addr 172.21.15.98:0/2723738758 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:23.105 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.097+0000 7efc3f7fe700 1 --2- 172.21.15.98:0/2723738758 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55b737fb8a90 0x55b737fb8e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:23.105 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.097+0000 7efc3effd700 1 --2- 172.21.15.98:0/2723738758 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b737f749a0 0x55b737f71580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:23.105 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.097+0000 7efc3ffff700 1 -- 172.21.15.98:0/2723738758 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55b737fb8a90 msgr2=0x55b737fb8e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:23.105 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.097+0000 7efc3ffff700 1 --2- 172.21.15.98:0/2723738758 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55b737fb8a90 0x55b737fb8e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:23.105 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.097+0000 7efc3ffff700 1 -- 172.21.15.98:0/2723738758 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b737f749a0 msgr2=0x55b737f71580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:23.105 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.097+0000 7efc3ffff700 1 --2- 172.21.15.98:0/2723738758 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b737f749a0 0x55b737f71580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:23.105 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.097+0000 7efc3ffff700 1 -- 172.21.15.98:0/2723738758 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b737d5c270 con 0x55b737f72bd0 2023-10-06T05:07:23.105 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.097+0000 7efc3effd700 1 --2- 172.21.15.98:0/2723738758 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b737f749a0 0x55b737f71580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:07:23.105 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.097+0000 7efc3ffff700 1 --2- 172.21.15.98:0/2723738758 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b737f72bd0 0x55b737f71ac0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7efc38082fd0 tx=0x7efc38082f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7928fb00fe7625d server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:23.105 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.097+0000 7efc24ff9700 1 -- 172.21.15.98:0/2723738758 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc38058c60 con 0x55b737f72bd0 2023-10-06T05:07:23.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.097+0000 7efc24ff9700 1 -- 172.21.15.98:0/2723738758 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc3805c9b0 con 0x55b737f72bd0 2023-10-06T05:07:23.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.097+0000 7efc24ff9700 1 -- 172.21.15.98:0/2723738758 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc38083110 con 0x55b737f72bd0 2023-10-06T05:07:23.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.098+0000 7efc4ed69b80 1 -- 172.21.15.98:0/2723738758 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b737f72bd0 msgr2=0x55b737f71ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:23.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.098+0000 7efc4ed69b80 1 --2- 172.21.15.98:0/2723738758 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b737f72bd0 0x55b737f71ac0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7efc38082fd0 tx=0x7efc38082f60 comp rx=0 tx=0).stop 2023-10-06T05:07:23.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.098+0000 7efc4ed69b80 1 -- 172.21.15.98:0/2723738758 shutdown_connections 2023-10-06T05:07:23.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.098+0000 7efc4ed69b80 1 --2- 172.21.15.98:0/2723738758 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b737f72bd0 0x55b737f71ac0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:23.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.098+0000 7efc4ed69b80 1 --2- 172.21.15.98:0/2723738758 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55b737fb8a90 0x55b737fb8e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:23.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.098+0000 7efc4ed69b80 1 --2- 172.21.15.98:0/2723738758 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b737f749a0 0x55b737f71580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:23.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.098+0000 7efc4ed69b80 1 -- 172.21.15.98:0/2723738758 >> 172.21.15.98:0/2723738758 conn(0x55b737f73070 msgr2=0x55b737f73440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:23.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.098+0000 7efc4ed69b80 1 -- 172.21.15.98:0/2723738758 shutdown_connections 2023-10-06T05:07:23.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.098+0000 7efc4ed69b80 1 -- 172.21.15.98:0/2723738758 wait complete. 2023-10-06T05:07:23.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.098+0000 7efc4ed69b80 1 -- start start 2023-10-06T05:07:23.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.098+0000 7efc4ed69b80 1 Processor -- start 2023-10-06T05:07:23.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.099+0000 7efc4ed69b80 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b737f72bd0 0x55b737f9a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:23.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.099+0000 7efc4ed69b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b737f749a0 0x55b737f9c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:23.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.099+0000 7efc4ed69b80 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55b737fb8a90 0x55b737f94670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:23.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.099+0000 7efc4ed69b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b737ec2060 con 0x55b737f749a0 2023-10-06T05:07:23.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.099+0000 7efc4ed69b80 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b737e6d890 con 0x55b737f72bd0 2023-10-06T05:07:23.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.099+0000 7efc4ed69b80 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b737e88ea0 con 0x55b737fb8a90 2023-10-06T05:07:23.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.099+0000 7efc4ed69b80 10 client.0 _ll_register_callbacks cb 0x55b737d5ef30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-06T05:07:23.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.099+0000 7efc3effd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b737f749a0 0x55b737f9c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:23.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.099+0000 7efc3effd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b737f749a0 0x55b737f9c080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47062/0 (socket says 172.21.15.98:47062) 2023-10-06T05:07:23.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.099+0000 7efc3effd700 1 -- 172.21.15.98:0/1651086486 learned_addr learned my addr 172.21.15.98:0/1651086486 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:23.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.099+0000 7efc3ffff700 1 --2- 172.21.15.98:0/1651086486 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55b737fb8a90 0x55b737f94670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:23.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.099+0000 7efc3f7fe700 1 --2- 172.21.15.98:0/1651086486 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b737f72bd0 0x55b737f9a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:23.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.100+0000 7efc3effd700 1 -- 172.21.15.98:0/1651086486 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55b737fb8a90 msgr2=0x55b737f94670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:23.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.100+0000 7efc3effd700 1 --2- 172.21.15.98:0/1651086486 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55b737fb8a90 0x55b737f94670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:23.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.100+0000 7efc3effd700 1 -- 172.21.15.98:0/1651086486 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b737f72bd0 msgr2=0x55b737f9a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:23.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.100+0000 7efc3effd700 1 --2- 172.21.15.98:0/1651086486 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b737f72bd0 0x55b737f9a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:23.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.100+0000 7efc3effd700 1 -- 172.21.15.98:0/1651086486 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc4009d760 con 0x55b737f749a0 2023-10-06T05:07:23.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.100+0000 7efc3effd700 1 --2- 172.21.15.98:0/1651086486 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b737f749a0 0x55b737f9c080 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7efc340117c0 tx=0x7efc34006f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=694eba303a0a1623 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:23.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.100+0000 7efc1dffb700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2023-10-06T05:07:23.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.100+0000 7efc1dffb700 1 -- 172.21.15.98:0/1651086486 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc34013680 con 0x55b737f749a0 2023-10-06T05:07:23.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.100+0000 7efc1dffb700 1 -- 172.21.15.98:0/1651086486 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7efc340137e0 con 0x55b737f749a0 2023-10-06T05:07:23.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.100+0000 7efc4ed69b80 1 -- 172.21.15.98:0/1651086486 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b737f66690 con 0x55b737f749a0 2023-10-06T05:07:23.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.100+0000 7efc1dffb700 1 -- 172.21.15.98:0/1651086486 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7efc34013bb0 con 0x55b737f749a0 2023-10-06T05:07:23.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.100+0000 7efc4ed69b80 10 client.4773 fetch_fsmap learned FSMap version 8 2023-10-06T05:07:23.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.100+0000 7efc4ed69b80 1 -- 172.21.15.98:0/1651086486 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x55b737cfefe0 con 0x55b737f749a0 2023-10-06T05:07:23.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.100+0000 7efc1dffb700 1 -- 172.21.15.98:0/1651086486 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55b737f66690 con 0x55b737f749a0 2023-10-06T05:07:23.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.100+0000 7efc1dffb700 1 -- 172.21.15.98:0/1651086486 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 8) v1 ==== 3108+0+0 (secure 0 0 0) 0x7efc34013d10 con 0x55b737f749a0 2023-10-06T05:07:23.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.100+0000 7efc4ed69b80 10 client.4773 fetch_fsmap finished waiting for FSMap version 8 2023-10-06T05:07:23.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.100+0000 7efc4ed69b80 10 client.4773 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-06T05:07:23.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.100+0000 7efc4ed69b80 20 client.4773 populate_metadata read hostname 'smithi098' 2023-10-06T05:07:23.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.100+0000 7efc4ed69b80 1 --2- 172.21.15.98:0/1651086486 >> [v2:172.21.15.98:6835/2683239094,v1:172.21.15.98:6837/2683239094] conn(0x55b737fcbb60 0x55b737fcbf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:23.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.100+0000 7efc4ed69b80 4 client.4773 mds_command: new command op to 4302 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-06T05:07:23.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.100+0000 7efc4ed69b80 1 -- 172.21.15.98:0/1651086486 --> [v2:172.21.15.98:6835/2683239094,v1:172.21.15.98:6837/2683239094] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b737f435c0 con 0x55b737fcbb60 2023-10-06T05:07:23.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.101+0000 7efc3f7fe700 1 --2- 172.21.15.98:0/1651086486 >> [v2:172.21.15.98:6835/2683239094,v1:172.21.15.98:6837/2683239094] conn(0x55b737fcbb60 0x55b737fcbf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:23.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.101+0000 7efc1dffb700 10 client.4773 ms_handle_connect on v2:172.21.15.98:6835/2683239094 2023-10-06T05:07:23.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.101+0000 7efc3f7fe700 1 --2- 172.21.15.98:0/1651086486 >> [v2:172.21.15.98:6835/2683239094,v1:172.21.15.98:6837/2683239094] conn(0x55b737fcbb60 0x55b737fcbf30 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=32dad1d72ca2649c server_cookie=d5cf402ddbc3831a in_seq=0 out_seq=0 2023-10-06T05:07:23.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.101+0000 7efc1dffb700 1 -- 172.21.15.98:0/1651086486 <== mds.0 v2:172.21.15.98:6835/2683239094 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+16954 (crc 0 0 0) 0x55b737f435c0 con 0x55b737fcbb60 2023-10-06T05:07:23.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.101+0000 7efc1dffb700 10 client.4773 handle_command_reply: tid=0 2023-10-06T05:07:23.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.101+0000 7efc4ed69b80 1 client.4773 shutdown 2023-10-06T05:07:23.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.101+0000 7efc4ed69b80 1 -- 172.21.15.98:0/1651086486 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b737f749a0 msgr2=0x55b737f9c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:23.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.101+0000 7efc4ed69b80 1 --2- 172.21.15.98:0/1651086486 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b737f749a0 0x55b737f9c080 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7efc340117c0 tx=0x7efc34006f90 comp rx=0 tx=0).stop 2023-10-06T05:07:23.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.101+0000 7efc4ed69b80 1 -- 172.21.15.98:0/1651086486 shutdown_connections 2023-10-06T05:07:23.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.101+0000 7efc4ed69b80 1 --2- 172.21.15.98:0/1651086486 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55b737fb8a90 0x55b737f94670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:23.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.101+0000 7efc4ed69b80 1 --2- 172.21.15.98:0/1651086486 >> [v2:172.21.15.98:6835/2683239094,v1:172.21.15.98:6837/2683239094] conn(0x55b737fcbb60 0x55b737fcbf30 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:23.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.102+0000 7efc4ed69b80 1 --2- 172.21.15.98:0/1651086486 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b737f72bd0 0x55b737f9a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:23.115 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.102+0000 7efc1dffb700 0 client.4773 ms_handle_reset on v2:172.21.15.98:6835/2683239094 2023-10-06T05:07:23.115 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.102+0000 7efc4ed69b80 1 --2- 172.21.15.98:0/1651086486 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b737f749a0 0x55b737f9c080 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:23.115 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.102+0000 7efc4ed69b80 1 -- 172.21.15.98:0/1651086486 >> 172.21.15.98:0/1651086486 conn(0x55b737f73070 msgr2=0x55b737fb6560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:23.115 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.102+0000 7efc4ed69b80 1 -- 172.21.15.98:0/1651086486 shutdown_connections 2023-10-06T05:07:23.115 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.102+0000 7efc4ed69b80 1 -- 172.21.15.98:0/1651086486 wait complete. 2023-10-06T05:07:23.115 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.102+0000 7efc4ed69b80 20 client.4773 trim_cache size 0 max 16384 2023-10-06T05:07:23.120 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.118+0000 7efc4ed69b80 1 Processor -- start 2023-10-06T05:07:23.120 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.118+0000 7efc4ed69b80 1 -- start start 2023-10-06T05:07:23.122 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.119+0000 7efc4ed69b80 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b737f72bd0 0x55b737fb4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:23.123 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.119+0000 7efc4ed69b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b737f749a0 0x55b737f9d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:23.123 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.119+0000 7efc4ed69b80 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55b737fb8a90 0x55b737f9d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:23.123 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.119+0000 7efc4ed69b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b737d44e30 con 0x55b737f749a0 2023-10-06T05:07:23.123 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.119+0000 7efc4ed69b80 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b737f39b90 con 0x55b737f72bd0 2023-10-06T05:07:23.123 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.119+0000 7efc4ed69b80 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b737f9c990 con 0x55b737fb8a90 2023-10-06T05:07:23.123 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.119+0000 7efc3effd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b737f749a0 0x55b737f9d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:23.123 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.119+0000 7efc3effd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b737f749a0 0x55b737f9d160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47068/0 (socket says 172.21.15.98:47068) 2023-10-06T05:07:23.123 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.119+0000 7efc3effd700 1 -- 172.21.15.98:0/1009939870 learned_addr learned my addr 172.21.15.98:0/1009939870 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:23.123 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.119+0000 7efc3effd700 1 -- 172.21.15.98:0/1009939870 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55b737fb8a90 msgr2=0x55b737f9d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:23.124 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.119+0000 7efc3f7fe700 1 --2- 172.21.15.98:0/1009939870 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b737f72bd0 0x55b737fb4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:23.124 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.119+0000 7efc3effd700 1 --2- 172.21.15.98:0/1009939870 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55b737fb8a90 0x55b737f9d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:23.124 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.119+0000 7efc3effd700 1 -- 172.21.15.98:0/1009939870 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b737f72bd0 msgr2=0x55b737fb4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:23.124 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.119+0000 7efc3effd700 1 --2- 172.21.15.98:0/1009939870 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b737f72bd0 0x55b737fb4ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:23.124 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.119+0000 7efc3effd700 1 -- 172.21.15.98:0/1009939870 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b737e90780 con 0x55b737f749a0 2023-10-06T05:07:23.124 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.119+0000 7efc3effd700 1 --2- 172.21.15.98:0/1009939870 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b737f749a0 0x55b737f9d160 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7efc3400a920 tx=0x7efc3401d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4e54de10126e65 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:23.124 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.119+0000 7efc257fa700 1 -- 172.21.15.98:0/1009939870 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc34013680 con 0x55b737f749a0 2023-10-06T05:07:23.124 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.120+0000 7efc257fa700 1 -- 172.21.15.98:0/1009939870 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc3400c040 con 0x55b737f749a0 2023-10-06T05:07:23.124 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.120+0000 7efc257fa700 1 -- 172.21.15.98:0/1009939870 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc34000e30 con 0x55b737f749a0 2023-10-06T05:07:23.125 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.120+0000 7efc4ed69b80 1 -- 172.21.15.98:0/1009939870 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b737f749a0 msgr2=0x55b737f9d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:23.125 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.120+0000 7efc4ed69b80 1 --2- 172.21.15.98:0/1009939870 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b737f749a0 0x55b737f9d160 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7efc3400a920 tx=0x7efc3401d000 comp rx=0 tx=0).stop 2023-10-06T05:07:23.125 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.120+0000 7efc4ed69b80 1 -- 172.21.15.98:0/1009939870 shutdown_connections 2023-10-06T05:07:23.125 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.120+0000 7efc4ed69b80 1 --2- 172.21.15.98:0/1009939870 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55b737fb8a90 0x55b737f9d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:23.125 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.120+0000 7efc4ed69b80 1 --2- 172.21.15.98:0/1009939870 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b737f72bd0 0x55b737fb4ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:23.125 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.120+0000 7efc4ed69b80 1 --2- 172.21.15.98:0/1009939870 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b737f749a0 0x55b737f9d160 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:23.125 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.120+0000 7efc4ed69b80 1 -- 172.21.15.98:0/1009939870 >> 172.21.15.98:0/1009939870 conn(0x55b737f73070 msgr2=0x55b737f73440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:23.125 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.120+0000 7efc4ed69b80 1 -- 172.21.15.98:0/1009939870 shutdown_connections 2023-10-06T05:07:23.125 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.120+0000 7efc4ed69b80 1 -- 172.21.15.98:0/1009939870 wait complete. 2023-10-06T05:07:23.125 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.121+0000 7efc4ed69b80 1 -- start start 2023-10-06T05:07:23.125 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.121+0000 7efc4ed69b80 1 Processor -- start 2023-10-06T05:07:23.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.121+0000 7efc4ed69b80 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b737f72bd0 0x55b738018380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:23.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.121+0000 7efc4ed69b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b737f749a0 0x55b73801e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:23.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.121+0000 7efc4ed69b80 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55b737fb8a90 0x55b737fb33c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:23.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.121+0000 7efc4ed69b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b73801f8a0 con 0x55b737f749a0 2023-10-06T05:07:23.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.121+0000 7efc4ed69b80 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b73801f410 con 0x55b737f72bd0 2023-10-06T05:07:23.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.121+0000 7efc4ed69b80 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b737fc9720 con 0x55b737fb8a90 2023-10-06T05:07:23.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.121+0000 7efc4ed69b80 10 client.0 _ll_register_callbacks cb 0x55b737d5ef30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-06T05:07:23.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.121+0000 7efc3effd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b737f749a0 0x55b73801e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:23.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.121+0000 7efc3effd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b737f749a0 0x55b73801e1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47084/0 (socket says 172.21.15.98:47084) 2023-10-06T05:07:23.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.121+0000 7efc3effd700 1 -- 172.21.15.98:0/2819445792 learned_addr learned my addr 172.21.15.98:0/2819445792 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:23.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.122+0000 7efc3effd700 1 -- 172.21.15.98:0/2819445792 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55b737fb8a90 msgr2=0x55b737fb33c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:23.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.122+0000 7efc3effd700 1 --2- 172.21.15.98:0/2819445792 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55b737fb8a90 0x55b737fb33c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:23.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.122+0000 7efc3effd700 1 -- 172.21.15.98:0/2819445792 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b737f72bd0 msgr2=0x55b738018380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:23.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.122+0000 7efc3effd700 1 --2- 172.21.15.98:0/2819445792 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b737f72bd0 0x55b738018380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:23.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.122+0000 7efc3effd700 1 -- 172.21.15.98:0/2819445792 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b737cfefe0 con 0x55b737f749a0 2023-10-06T05:07:23.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.122+0000 7efc3effd700 1 --2- 172.21.15.98:0/2819445792 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b737f749a0 0x55b73801e1f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7efc3402d000 tx=0x7efc3402a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56f076ab2e3d6fda server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:23.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.122+0000 7efc1dffb700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2023-10-06T05:07:23.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.122+0000 7efc1dffb700 1 -- 172.21.15.98:0/2819445792 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc34013680 con 0x55b737f749a0 2023-10-06T05:07:23.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.122+0000 7efc1dffb700 1 -- 172.21.15.98:0/2819445792 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7efc340137e0 con 0x55b737f749a0 2023-10-06T05:07:23.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.122+0000 7efc4ed69b80 1 -- 172.21.15.98:0/2819445792 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efc3805c9b0 con 0x55b737f749a0 2023-10-06T05:07:23.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.122+0000 7efc1dffb700 1 -- 172.21.15.98:0/2819445792 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7efc3401f6f0 con 0x55b737f749a0 2023-10-06T05:07:23.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.122+0000 7efc4ed69b80 10 client.4779 fetch_fsmap learned FSMap version 8 2023-10-06T05:07:23.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.122+0000 7efc4ed69b80 1 -- 172.21.15.98:0/2819445792 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x55b737e77470 con 0x55b737f749a0 2023-10-06T05:07:23.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.122+0000 7efc1dffb700 1 -- 172.21.15.98:0/2819445792 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc3805c9b0 con 0x55b737f749a0 2023-10-06T05:07:23.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.122+0000 7efc1dffb700 1 -- 172.21.15.98:0/2819445792 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 8) v1 ==== 3108+0+0 (secure 0 0 0) 0x7efc3402f020 con 0x55b737f749a0 2023-10-06T05:07:23.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.122+0000 7efc4ed69b80 10 client.4779 fetch_fsmap finished waiting for FSMap version 8 2023-10-06T05:07:23.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.122+0000 7efc4ed69b80 10 client.4779 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-06T05:07:23.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.122+0000 7efc4ed69b80 20 client.4779 populate_metadata read hostname 'smithi098' 2023-10-06T05:07:23.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.122+0000 7efc4ed69b80 1 --2- 172.21.15.98:0/2819445792 >> [v2:172.21.15.98:6835/2683239094,v1:172.21.15.98:6837/2683239094] conn(0x55b737fc9c20 0x55b737fca010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:23.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.122+0000 7efc4ed69b80 4 client.4779 mds_command: new command op to 4302 tid=0[{"prefix": "status"}] 2023-10-06T05:07:23.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.122+0000 7efc4ed69b80 1 -- 172.21.15.98:0/2819445792 --> [v2:172.21.15.98:6835/2683239094,v1:172.21.15.98:6837/2683239094] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55b737e64e00 con 0x55b737fc9c20 2023-10-06T05:07:23.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.122+0000 7efc3f7fe700 1 --2- 172.21.15.98:0/2819445792 >> [v2:172.21.15.98:6835/2683239094,v1:172.21.15.98:6837/2683239094] conn(0x55b737fc9c20 0x55b737fca010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:23.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.123+0000 7efc1dffb700 10 client.4779 ms_handle_connect on v2:172.21.15.98:6835/2683239094 2023-10-06T05:07:23.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.123+0000 7efc3f7fe700 1 --2- 172.21.15.98:0/2819445792 >> [v2:172.21.15.98:6835/2683239094,v1:172.21.15.98:6837/2683239094] conn(0x55b737fc9c20 0x55b737fca010 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=884e16fa5c56e99f server_cookie=81cf34db8df3eaa5 in_seq=0 out_seq=0 2023-10-06T05:07:23.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.123+0000 7efc1dffb700 1 -- 172.21.15.98:0/2819445792 <== mds.0 v2:172.21.15.98:6835/2683239094 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+317 (crc 0 0 0) 0x55b737e64e00 con 0x55b737fc9c20 2023-10-06T05:07:23.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.123+0000 7efc1dffb700 10 client.4779 handle_command_reply: tid=0 2023-10-06T05:07:23.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.123+0000 7efc4ed69b80 1 client.4779 shutdown 2023-10-06T05:07:23.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.123+0000 7efc4ed69b80 1 -- 172.21.15.98:0/2819445792 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b737f749a0 msgr2=0x55b73801e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:23.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.123+0000 7efc4ed69b80 1 --2- 172.21.15.98:0/2819445792 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b737f749a0 0x55b73801e1f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7efc3402d000 tx=0x7efc3402a000 comp rx=0 tx=0).stop 2023-10-06T05:07:23.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.123+0000 7efc4ed69b80 1 -- 172.21.15.98:0/2819445792 shutdown_connections 2023-10-06T05:07:23.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.123+0000 7efc4ed69b80 1 --2- 172.21.15.98:0/2819445792 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55b737fb8a90 0x55b737fb33c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:23.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.123+0000 7efc4ed69b80 1 --2- 172.21.15.98:0/2819445792 >> [v2:172.21.15.98:6835/2683239094,v1:172.21.15.98:6837/2683239094] conn(0x55b737fc9c20 0x55b737fca010 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:23.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.123+0000 7efc4ed69b80 1 --2- 172.21.15.98:0/2819445792 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b737f72bd0 0x55b738018380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:23.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.123+0000 7efc1dffb700 0 client.4779 ms_handle_reset on v2:172.21.15.98:6835/2683239094 2023-10-06T05:07:23.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.123+0000 7efc4ed69b80 1 --2- 172.21.15.98:0/2819445792 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b737f749a0 0x55b73801e1f0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:23.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.123+0000 7efc4ed69b80 1 -- 172.21.15.98:0/2819445792 >> 172.21.15.98:0/2819445792 conn(0x55b737f73070 msgr2=0x55b737faeb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:23.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.123+0000 7efc4ed69b80 1 -- 172.21.15.98:0/2819445792 shutdown_connections 2023-10-06T05:07:23.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.123+0000 7efc4ed69b80 1 -- 172.21.15.98:0/2819445792 wait complete. 2023-10-06T05:07:23.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.123+0000 7efc4ed69b80 20 client.4779 trim_cache size 0 max 16384 2023-10-06T05:07:23.134 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-10-06T05:07:23.134 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "f50c5df5-cad5-47a4-9bb1-8c791d4311ec", 2023-10-06T05:07:23.134 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 0, 2023-10-06T05:07:23.134 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 4302, 2023-10-06T05:07:23.134 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2023-10-06T05:07:23.134 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2023-10-06T05:07:23.134 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2023-10-06T05:07:23.134 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 25.42610264, 2023-10-06T05:07:23.135 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 8, 2023-10-06T05:07:23.135 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 29, 2023-10-06T05:07:23.135 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 29, 2023-10-06T05:07:23.135 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 30.441950744 2023-10-06T05:07:23.135 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-10-06T05:07:23.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.124+0000 7efc45a71700 1 -- 172.21.15.98:0/2064216499 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7efc2c066fb0 msgr2=0x7efc2c069460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:23.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.124+0000 7efc45a71700 1 --2- 172.21.15.98:0/2064216499 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7efc2c066fb0 0x7efc2c069460 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7efc30009260 tx=0x7efc30016000 comp rx=0 tx=0).stop 2023-10-06T05:07:23.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.124+0000 7efc45a71700 1 -- 172.21.15.98:0/2064216499 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc40159560 msgr2=0x7efc400695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:23.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.124+0000 7efc45a71700 1 --2- 172.21.15.98:0/2064216499 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc40159560 0x7efc400695e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7efc38009fa0 tx=0x7efc38009900 comp rx=0 tx=0).stop 2023-10-06T05:07:23.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.124+0000 7efc45a71700 1 -- 172.21.15.98:0/2064216499 shutdown_connections 2023-10-06T05:07:23.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.124+0000 7efc45a71700 1 --2- 172.21.15.98:0/2064216499 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7efc2c066fb0 0x7efc2c069460 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:23.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.124+0000 7efc45a71700 1 --2- 172.21.15.98:0/2064216499 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc40159560 0x7efc400695e0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:23.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.124+0000 7efc45a71700 1 --2- 172.21.15.98:0/2064216499 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efc400b0760 0x7efc4016a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:23.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.124+0000 7efc45a71700 1 --2- 172.21.15.98:0/2064216499 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efc400b49a0 0x7efc4015b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:23.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.124+0000 7efc45a71700 1 -- 172.21.15.98:0/2064216499 >> 172.21.15.98:0/2064216499 conn(0x7efc400997f0 msgr2=0x7efc40158770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:23.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.124+0000 7efc45a71700 1 -- 172.21.15.98:0/2064216499 shutdown_connections 2023-10-06T05:07:23.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.124+0000 7efc45a71700 1 -- 172.21.15.98:0/2064216499 wait complete. 2023-10-06T05:07:23.138 INFO:teuthology.run_tasks:Running task kclient... 2023-10-06T05:07:23.150 INFO:tasks.kclient:Mounting kernel clients... 2023-10-06T05:07:23.150 INFO:tasks.kclient:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2023-10-06T05:07:23.150 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:07:23.150 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list 2023-10-06T05:07:23.165 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2023-10-06T05:07:23.166 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:07:23.166 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-10-06T05:07:23.241 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:07:23.241 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link delete ceph-brx 2023-10-06T05:07:23.338 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:07:23.339 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list 2023-10-06T05:07:23.353 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:07:23.353 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link delete ceph-brx 2023-10-06T05:07:23.422 INFO:teuthology.orchestra.run.smithi098.stderr:Cannot find device "ceph-brx" 2023-10-06T05:07:23.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-06T05:07:23.423 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:07:23.424 DEBUG:teuthology.orchestra.run.smithi174:> ip netns list 2023-10-06T05:07:23.442 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:07:23.443 DEBUG:teuthology.orchestra.run.smithi174:> sudo ip link delete ceph-brx 2023-10-06T05:07:23.515 INFO:teuthology.orchestra.run.smithi174.stderr:Cannot find device "ceph-brx" 2023-10-06T05:07:23.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-06T05:07:23.516 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:07:23.516 DEBUG:teuthology.orchestra.run.smithi174:> ip netns list 2023-10-06T05:07:23.573 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:07:23.573 DEBUG:teuthology.orchestra.run.smithi174:> sudo ip link delete ceph-brx 2023-10-06T05:07:23.642 INFO:teuthology.orchestra.run.smithi174.stderr:Cannot find device "ceph-brx" 2023-10-06T05:07:23.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-06T05:07:23.643 INFO:tasks.kclient:client.0 config is {'syntax': 'v2'} 2023-10-06T05:07:23.643 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-06T05:07:23.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.757+0000 7ff1acb92700 1 Processor -- start 2023-10-06T05:07:23.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.757+0000 7ff1acb92700 1 -- start start 2023-10-06T05:07:23.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.758+0000 7ff1acb92700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff1a8159560 0x7ff1a80aa530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:23.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.758+0000 7ff1acb92700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff1a80b0760 0x7ff1a80b1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:23.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.758+0000 7ff1acb92700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1a80b0c60 con 0x7ff1a80b49a0 2023-10-06T05:07:23.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.758+0000 7ff1acb92700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1a80b0f40 con 0x7ff1a8159560 2023-10-06T05:07:23.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.758+0000 7ff1acb92700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a80b1220 con 0x7ff1a80b0760 2023-10-06T05:07:23.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.758+0000 7ff1a659c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff1a8159560 0x7ff1a80aa530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:23.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.758+0000 7ff19ffff700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7ff1a80b49a0 0x7ff1a80aaa70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:60932/0 (socket says 172.21.15.98:60932) 2023-10-06T05:07:23.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.758+0000 7ff19ffff700 1 -- 172.21.15.98:0/2784999295 learned_addr learned my addr 172.21.15.98:0/2784999295 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:07:23.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.758+0000 7ff1a6d9d700 1 --2- 172.21.15.98:0/2784999295 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff1a80b0760 0x7ff1a80b1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:23.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.759+0000 7ff1a659c700 1 -- 172.21.15.98:0/2784999295 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff1a80b0760 msgr2=0x7ff1a80b1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:23.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.759+0000 7ff1a659c700 1 --2- 172.21.15.98:0/2784999295 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff1a80b0760 0x7ff1a80b1bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:23.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.759+0000 7ff1a659c700 1 -- 172.21.15.98:0/2784999295 >> v1:172.21.15.98:6789/0 conn(0x7ff1a80b49a0 legacy=0x7ff1a80aaa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:23.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.759+0000 7ff1a659c700 1 -- 172.21.15.98:0/2784999295 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1a80b1500 con 0x7ff1a8159560 2023-10-06T05:07:23.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.759+0000 7ff1a5d9b700 1 -- 172.21.15.98:0/2784999295 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 43673233 0 0) 0x7ff1a80b0c60 con 0x7ff1a80b49a0 2023-10-06T05:07:23.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.759+0000 7ff1a659c700 1 --2- 172.21.15.98:0/2784999295 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1a8159560 0x7ff1a80aa530 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7ff198009900 tx=0x7ff198009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94cfa4e6456a0251 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:23.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.759+0000 7ff1a5d9b700 1 -- 172.21.15.98:0/2784999295 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff198005c10 con 0x7ff1a8159560 2023-10-06T05:07:23.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.759+0000 7ff1a5d9b700 1 -- 172.21.15.98:0/2784999295 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff198005d70 con 0x7ff1a8159560 2023-10-06T05:07:23.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.759+0000 7ff1a5d9b700 1 -- 172.21.15.98:0/2784999295 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff198006260 con 0x7ff1a8159560 2023-10-06T05:07:23.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.759+0000 7ff1acb92700 1 -- 172.21.15.98:0/2784999295 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1a8159560 msgr2=0x7ff1a80aa530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:23.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.759+0000 7ff1acb92700 1 --2- 172.21.15.98:0/2784999295 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1a8159560 0x7ff1a80aa530 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7ff198009900 tx=0x7ff198009120 comp rx=0 tx=0).stop 2023-10-06T05:07:23.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.759+0000 7ff1acb92700 1 -- 172.21.15.98:0/2784999295 shutdown_connections 2023-10-06T05:07:23.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.759+0000 7ff1acb92700 1 --2- 172.21.15.98:0/2784999295 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff1a80b0760 0x7ff1a80b1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:23.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.759+0000 7ff1acb92700 1 --2- 172.21.15.98:0/2784999295 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1a8159560 0x7ff1a80aa530 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:23.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.759+0000 7ff1acb92700 1 -- 172.21.15.98:0/2784999295 >> 172.21.15.98:0/2784999295 conn(0x7ff1a80997f0 msgr2=0x7ff1a806f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:23.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.759+0000 7ff1acb92700 1 -- 172.21.15.98:0/2784999295 shutdown_connections 2023-10-06T05:07:23.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.759+0000 7ff1acb92700 1 -- 172.21.15.98:0/2784999295 wait complete. 2023-10-06T05:07:23.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.760+0000 7ff1acb92700 1 Processor -- start 2023-10-06T05:07:23.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.761+0000 7ff1acb92700 1 -- start start 2023-10-06T05:07:23.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.761+0000 7ff1acb92700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1a80b0760 0x7ff1a8169be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:23.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.761+0000 7ff1acb92700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff1a80b49a0 0x7ff1a809c170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:23.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.761+0000 7ff1acb92700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1a8159560 0x7ff1a81584a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:23.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.761+0000 7ff1acb92700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a8159d40 con 0x7ff1a8159560 2023-10-06T05:07:23.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.761+0000 7ff1acb92700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a8159fc0 con 0x7ff1a80b0760 2023-10-06T05:07:23.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.761+0000 7ff1acb92700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a815a2a0 con 0x7ff1a80b49a0 2023-10-06T05:07:23.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.761+0000 7ff1a6d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1a8159560 0x7ff1a81584a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:23.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.761+0000 7ff1a6d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1a8159560 0x7ff1a81584a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47100/0 (socket says 172.21.15.98:47100) 2023-10-06T05:07:23.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.761+0000 7ff1a6d9d700 1 -- 172.21.15.98:0/3137301722 learned_addr learned my addr 172.21.15.98:0/3137301722 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:23.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.761+0000 7ff19ffff700 1 --2- 172.21.15.98:0/3137301722 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff1a80b49a0 0x7ff1a809c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:23.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.761+0000 7ff1a659c700 1 --2- 172.21.15.98:0/3137301722 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1a80b0760 0x7ff1a8169be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:23.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.762+0000 7ff1a6d9d700 1 -- 172.21.15.98:0/3137301722 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff1a80b49a0 msgr2=0x7ff1a809c170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:23.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.762+0000 7ff1a6d9d700 1 --2- 172.21.15.98:0/3137301722 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff1a80b49a0 0x7ff1a809c170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:23.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.762+0000 7ff1a6d9d700 1 -- 172.21.15.98:0/3137301722 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1a80b0760 msgr2=0x7ff1a8169be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:23.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.762+0000 7ff1a6d9d700 1 --2- 172.21.15.98:0/3137301722 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1a80b0760 0x7ff1a8169be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:23.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.762+0000 7ff1a6d9d700 1 -- 172.21.15.98:0/3137301722 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1980093c0 con 0x7ff1a8159560 2023-10-06T05:07:23.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.762+0000 7ff1a6d9d700 1 --2- 172.21.15.98:0/3137301722 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1a8159560 0x7ff1a81584a0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff1a0009ef0 tx=0x7ff1a000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:23.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.762+0000 7ff19e7fc700 1 -- 172.21.15.98:0/3137301722 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1a000cb70 con 0x7ff1a8159560 2023-10-06T05:07:23.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.762+0000 7ff19e7fc700 1 -- 172.21.15.98:0/3137301722 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1a000f040 con 0x7ff1a8159560 2023-10-06T05:07:23.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.762+0000 7ff1acb92700 1 -- 172.21.15.98:0/3137301722 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1a815a580 con 0x7ff1a8159560 2023-10-06T05:07:23.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.762+0000 7ff19e7fc700 1 -- 172.21.15.98:0/3137301722 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1a0014480 con 0x7ff1a8159560 2023-10-06T05:07:23.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.762+0000 7ff1acb92700 1 -- 172.21.15.98:0/3137301722 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1a816a120 con 0x7ff1a8159560 2023-10-06T05:07:23.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.763+0000 7ff1acb92700 1 -- 172.21.15.98:0/3137301722 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1a8064a90 con 0x7ff1a8159560 2023-10-06T05:07:23.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.763+0000 7ff19e7fc700 1 -- 172.21.15.98:0/3137301722 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7ff1a0014830 con 0x7ff1a8159560 2023-10-06T05:07:23.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.764+0000 7ff19e7fc700 1 --2- 172.21.15.98:0/3137301722 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7ff188066f30 0x7ff1880693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:23.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.764+0000 7ff19e7fc700 1 -- 172.21.15.98:0/3137301722 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7ff1a00855a0 con 0x7ff1a8159560 2023-10-06T05:07:23.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.764+0000 7ff1a659c700 1 --2- 172.21.15.98:0/3137301722 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7ff188066f30 0x7ff1880693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:23.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.764+0000 7ff1a659c700 1 --2- 172.21.15.98:0/3137301722 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7ff188066f30 0x7ff1880693e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff198009160 tx=0x7ff198016000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:23.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.768+0000 7ff19e7fc700 1 -- 172.21.15.98:0/3137301722 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff1a00523c0 con 0x7ff1a8159560 2023-10-06T05:07:23.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.995+0000 7ff1acb92700 1 -- 172.21.15.98:0/3137301722 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7ff1a80b17e0 con 0x7ff1a8159560 2023-10-06T05:07:23.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.995+0000 7ff19e7fc700 1 -- 172.21.15.98:0/3137301722 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+88 (secure 0 0 0) 0x7ff1a00559e0 con 0x7ff1a8159560 2023-10-06T05:07:23.997 INFO:teuthology.orchestra.run.smithi098.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-10-06T05:07:24.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.997+0000 7ff1acb92700 1 -- 172.21.15.98:0/3137301722 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7ff188066f30 msgr2=0x7ff1880693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:24.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.997+0000 7ff1acb92700 1 --2- 172.21.15.98:0/3137301722 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7ff188066f30 0x7ff1880693e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff198009160 tx=0x7ff198016000 comp rx=0 tx=0).stop 2023-10-06T05:07:24.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.997+0000 7ff1acb92700 1 -- 172.21.15.98:0/3137301722 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1a8159560 msgr2=0x7ff1a81584a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:24.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.997+0000 7ff1acb92700 1 --2- 172.21.15.98:0/3137301722 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1a8159560 0x7ff1a81584a0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff1a0009ef0 tx=0x7ff1a000b040 comp rx=0 tx=0).stop 2023-10-06T05:07:24.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.997+0000 7ff1acb92700 1 -- 172.21.15.98:0/3137301722 shutdown_connections 2023-10-06T05:07:24.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.997+0000 7ff1acb92700 1 --2- 172.21.15.98:0/3137301722 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7ff188066f30 0x7ff1880693e0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:24.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.997+0000 7ff1acb92700 1 --2- 172.21.15.98:0/3137301722 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1a8159560 0x7ff1a81584a0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:24.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.997+0000 7ff1acb92700 1 --2- 172.21.15.98:0/3137301722 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1a80b0760 0x7ff1a8169be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:24.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.997+0000 7ff1acb92700 1 --2- 172.21.15.98:0/3137301722 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff1a80b49a0 0x7ff1a809c170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:24.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.997+0000 7ff1acb92700 1 -- 172.21.15.98:0/3137301722 >> 172.21.15.98:0/3137301722 conn(0x7ff1a80997f0 msgr2=0x7ff1a80b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:24.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.997+0000 7ff1acb92700 1 -- 172.21.15.98:0/3137301722 shutdown_connections 2023-10-06T05:07:24.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:23.997+0000 7ff1acb92700 1 -- 172.21.15.98:0/3137301722 wait complete. 2023-10-06T05:07:24.011 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-10-06T05:07:24.011 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-06T05:07:24.011 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi098.front.sepia.ceph.com 2023-10-06T05:07:24.012 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-06T05:07:24.012 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-06T05:07:24.012 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-10-06T05:07:24.012 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-06T05:07:24.012 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-06T05:07:24.012 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-06T05:07:24.012 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:07:24.012 DEBUG:teuthology.orchestra.run.smithi098:> ip addr 2023-10-06T05:07:24.027 INFO:teuthology.orchestra.run.smithi098.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-06T05:07:24.027 INFO:teuthology.orchestra.run.smithi098.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-06T05:07:24.027 INFO:teuthology.orchestra.run.smithi098.stdout: inet 127.0.0.1/8 scope host lo 2023-10-06T05:07:24.027 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-10-06T05:07:24.027 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 ::1/128 scope host 2023-10-06T05:07:24.027 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-10-06T05:07:24.027 INFO:teuthology.orchestra.run.smithi098.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-06T05:07:24.028 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:d8:d9:b8 brd ff:ff:ff:ff:ff:ff 2023-10-06T05:07:24.028 INFO:teuthology.orchestra.run.smithi098.stdout: altname enp7s0f0 2023-10-06T05:07:24.028 INFO:teuthology.orchestra.run.smithi098.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-06T05:07:24.028 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:d8:d9:b9 brd ff:ff:ff:ff:ff:ff 2023-10-06T05:07:24.028 INFO:teuthology.orchestra.run.smithi098.stdout: altname enp7s0f1 2023-10-06T05:07:24.028 INFO:teuthology.orchestra.run.smithi098.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-06T05:07:24.028 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:8f:d2:6a brd ff:ff:ff:ff:ff:ff 2023-10-06T05:07:24.028 INFO:teuthology.orchestra.run.smithi098.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-06T05:07:24.029 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:8f:d2:6b brd ff:ff:ff:ff:ff:ff 2023-10-06T05:07:24.029 INFO:teuthology.orchestra.run.smithi098.stdout: inet 172.21.15.98/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-10-06T05:07:24.029 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft 42481sec preferred_lft 42481sec 2023-10-06T05:07:24.029 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::ec4:7aff:fe8f:d26b/64 scope link 2023-10-06T05:07:24.029 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-10-06T05:07:24.029 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-10-06T05:07:24.029 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:07:24.030 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec bash -c ' 2023-10-06T05:07:24.030 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2023-10-06T05:07:24.030 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link add name ceph-brx type bridge 2023-10-06T05:07:24.030 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip addr flush dev ceph-brx 2023-10-06T05:07:24.030 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set ceph-brx up 2023-10-06T05:07:24.030 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-10-06T05:07:24.030 DEBUG:teuthology.orchestra.run.smithi098:> ') 2023-10-06T05:07:24.137 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:07:24.137 DEBUG:teuthology.orchestra.run.smithi098:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-06T05:07:24.161 INFO:teuthology.orchestra.run.smithi098.stdout:1 2023-10-06T05:07:24.162 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:07:24.162 DEBUG:teuthology.orchestra.run.smithi098:> route 2023-10-06T05:07:24.219 INFO:teuthology.orchestra.run.smithi098.stdout:Kernel IP routing table 2023-10-06T05:07:24.219 INFO:teuthology.orchestra.run.smithi098.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-06T05:07:24.219 INFO:teuthology.orchestra.run.smithi098.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-10-06T05:07:24.219 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-10-06T05:07:24.219 INFO:teuthology.orchestra.run.smithi098.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-06T05:07:24.220 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:07:24.220 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec bash -c ' 2023-10-06T05:07:24.220 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2023-10-06T05:07:24.220 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-10-06T05:07:24.220 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-10-06T05:07:24.221 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-10-06T05:07:24.221 DEBUG:teuthology.orchestra.run.smithi098:> ') 2023-10-06T05:07:24.319 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:07:24.319 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list 2023-10-06T05:07:24.334 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:07:24.334 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list-id 2023-10-06T05:07:24.388 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:07:24.388 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec bash -c ' 2023-10-06T05:07:24.388 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2023-10-06T05:07:24.389 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-06T05:07:24.389 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2023-10-06T05:07:24.389 DEBUG:teuthology.orchestra.run.smithi098:> ') 2023-10-06T05:07:24.477 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2023-10-06T05:07:24.478 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:07:24.478 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec bash -c ' 2023-10-06T05:07:24.478 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2023-10-06T05:07:24.478 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2023-10-06T05:07:24.478 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-10-06T05:07:24.478 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2023-10-06T05:07:24.478 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2023-10-06T05:07:24.478 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2023-10-06T05:07:24.478 DEBUG:teuthology.orchestra.run.smithi098:> ') 2023-10-06T05:07:24.657 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:07:24.658 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec bash -c ' 2023-10-06T05:07:24.658 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2023-10-06T05:07:24.658 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set brx.0 up 2023-10-06T05:07:24.658 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set dev brx.0 master ceph-brx 2023-10-06T05:07:24.658 DEBUG:teuthology.orchestra.run.smithi098:> ') 2023-10-06T05:07:24.749 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-06T05:07:24.750 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-06T05:07:24.764 INFO:teuthology.orchestra.run.smithi098.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-06T05:07:24.764 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-06T05:07:24.765 DEBUG:teuthology.orchestra.run.smithi098:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-06T05:07:24.819 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2023-10-06T05:07:24.819 INFO:teuthology.orchestra.run:Running command with timeout 1800 2023-10-06T05:07:24.820 DEBUG:teuthology.orchestra.run.smithi098:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-06T05:07:24.900 INFO:teuthology.orchestra.run.smithi098.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-06T05:07:24.900 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-06T05:07:24.993 INFO:teuthology.orchestra.run.smithi098.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-06T05:07:24.993 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-06T05:07:24.993 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: resolved to: "172.21.15.98:3300,172.21.15.174:3300,172.21.15.174:3301" 2023-10-06T05:07:24.993 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: trying mount with new device syntax: 0@f50c5df5-cad5-47a4-9bb1-8c791d4311ec.cephfs=/ 2023-10-06T05:07:24.994 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.98:3300/172.21.15.174:3300/172.21.15.174:3301" will pass to kernel 2023-10-06T05:07:24.994 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: v2 mount support check returned 2 2023-10-06T05:07:24.994 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: kernel does not support v2 syntax 2023-10-06T05:07:24.994 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-06T05:07:25.016 INFO:teuthology.orchestra.run.smithi098.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-06T05:07:25.016 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-06T05:07:25.016 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: resolved to: "172.21.15.98:3300,172.21.15.174:3300,172.21.15.174:3301" 2023-10-06T05:07:25.016 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: trying mount with new device syntax: 0@f50c5df5-cad5-47a4-9bb1-8c791d4311ec.cephfs=/ 2023-10-06T05:07:25.016 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.98:3300/172.21.15.174:3300/172.21.15.174:3301" will pass to kernel 2023-10-06T05:07:25.016 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: v2 mount support check returned 2 2023-10-06T05:07:25.016 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: kernel does not support v2 syntax 2023-10-06T05:07:25.017 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-06T05:07:25.017 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: resolved to: "172.21.15.98:6789,172.21.15.174:6789,172.21.15.174:6790" 2023-10-06T05:07:25.017 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.98:6789,172.21.15.174:6789,172.21.15.174:6790:/ 2023-10-06T05:07:25.017 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,key=0,mds_namespace=cephfs,fsid=f50c5df5-cad5-47a4-9bb1-8c791d4311ec" will pass to kernel 2023-10-06T05:07:25.018 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-06T05:07:25.019 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:07:25.019 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-06T05:07:25.083 INFO:tasks.kclient:client.2 config is {'syntax': 'v2'} 2023-10-06T05:07:25.083 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-06T05:07:25.238 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.236+0000 7f3dc6ea2700 1 Processor -- start 2023-10-06T05:07:25.238 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.236+0000 7f3dc6ea2700 1 -- start start 2023-10-06T05:07:25.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.237+0000 7f3dc6ea2700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3dc00cd140 0x7f3dc00cd5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:25.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.237+0000 7f3dc6ea2700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f3dc00cb3c0 0x7f3dc00cb800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:25.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.237+0000 7f3dc6ea2700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dc00cdaf0 con 0x7f3dc0159df0 2023-10-06T05:07:25.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.237+0000 7f3dc6ea2700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3dc015e020 con 0x7f3dc00cb3c0 2023-10-06T05:07:25.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.237+0000 7f3dc6ea2700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc015d4a0 con 0x7f3dc00cd140 2023-10-06T05:07:25.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.237+0000 7f3dc4c3e700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f3dc0159df0 0x7f3dc015a1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:60942/0 (socket says 172.21.15.98:60942) 2023-10-06T05:07:25.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.237+0000 7f3dc4c3e700 1 -- 172.21.15.98:0/1427180752 learned_addr learned my addr 172.21.15.98:0/1427180752 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:07:25.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.237+0000 7f3dbffff700 1 --2- 172.21.15.98:0/1427180752 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3dc00cd140 0x7f3dc00cd5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:25.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.237+0000 7f3dc543f700 1 --2- 172.21.15.98:0/1427180752 >> v2:172.21.15.174:3300/0 conn(0x7f3dc00cb3c0 0x7f3dc00cb800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:25.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.237+0000 7f3dbf7fe700 1 -- 172.21.15.98:0/1427180752 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3041360651 0 0) 0x7f3dc00cdaf0 con 0x7f3dc0159df0 2023-10-06T05:07:25.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.237+0000 7f3dbf7fe700 1 -- 172.21.15.98:0/1427180752 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3dac003570 con 0x7f3dc0159df0 2023-10-06T05:07:25.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.237+0000 7f3dbf7fe700 1 -- 172.21.15.98:0/1427180752 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1192711728 0 0) 0x7f3dac003570 con 0x7f3dc0159df0 2023-10-06T05:07:25.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.237+0000 7f3dbf7fe700 1 -- 172.21.15.98:0/1427180752 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3dc00cd140 msgr2=0x7f3dc00cd5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:25.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.237+0000 7f3dbf7fe700 1 --2- 172.21.15.98:0/1427180752 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3dc00cd140 0x7f3dc00cd5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:25.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.237+0000 7f3dbf7fe700 1 -- 172.21.15.98:0/1427180752 >> v2:172.21.15.174:3300/0 conn(0x7f3dc00cb3c0 msgr2=0x7f3dc00cb800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:25.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.237+0000 7f3dbf7fe700 1 --2- 172.21.15.98:0/1427180752 >> v2:172.21.15.174:3300/0 conn(0x7f3dc00cb3c0 0x7f3dc00cb800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:25.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.237+0000 7f3dbf7fe700 1 -- 172.21.15.98:0/1427180752 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dc015d780 con 0x7f3dc0159df0 2023-10-06T05:07:25.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.237+0000 7f3dbf7fe700 1 -- 172.21.15.98:0/1427180752 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1543991876 0 0) 0x7f3db8001470 con 0x7f3dc0159df0 2023-10-06T05:07:25.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.237+0000 7f3dbf7fe700 1 -- 172.21.15.98:0/1427180752 >> v1:172.21.15.98:6789/0 conn(0x7f3dc0159df0 legacy=0x7f3dc015a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:25.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.238+0000 7f3dbf7fe700 1 --2- 172.21.15.98:0/1427180752 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3dc00cd140 0x7f3dac003a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:25.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.238+0000 7f3dbf7fe700 1 --2- 172.21.15.98:0/1427180752 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dac003fe0 0x7f3dac004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:25.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.238+0000 7f3dbf7fe700 1 --2- 172.21.15.98:0/1427180752 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3dac004d50 0x7f3dac0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:25.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.238+0000 7f3dbf7fe700 1 -- 172.21.15.98:0/1427180752 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc015e020 con 0x7f3dac003fe0 2023-10-06T05:07:25.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.238+0000 7f3dbf7fe700 1 -- 172.21.15.98:0/1427180752 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc015d4a0 con 0x7f3dac004d50 2023-10-06T05:07:25.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.238+0000 7f3dbf7fe700 1 -- 172.21.15.98:0/1427180752 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dac00bb40 con 0x7f3dc00cd140 2023-10-06T05:07:25.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.238+0000 7f3dbffff700 1 --2- 172.21.15.98:0/1427180752 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dac003fe0 0x7f3dac004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:25.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.238+0000 7f3dc4c3e700 1 --2- 172.21.15.98:0/1427180752 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3dc00cd140 0x7f3dac003a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:25.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.238+0000 7f3dc543f700 1 --2- 172.21.15.98:0/1427180752 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3dac004d50 0x7f3dac0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:25.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.238+0000 7f3dbffff700 1 -- 172.21.15.98:0/1427180752 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3dc00cd140 msgr2=0x7f3dac003a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:25.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.238+0000 7f3dbffff700 1 --2- 172.21.15.98:0/1427180752 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3dc00cd140 0x7f3dac003a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:25.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.238+0000 7f3dbffff700 1 -- 172.21.15.98:0/1427180752 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3dac004d50 msgr2=0x7f3dac0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:25.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.238+0000 7f3dbffff700 1 --2- 172.21.15.98:0/1427180752 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3dac004d50 0x7f3dac0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:25.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.238+0000 7f3dbffff700 1 -- 172.21.15.98:0/1427180752 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dac003570 con 0x7f3dac003fe0 2023-10-06T05:07:25.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.238+0000 7f3dc543f700 1 --2- 172.21.15.98:0/1427180752 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3dac004d50 0x7f3dac0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:07:25.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.238+0000 7f3dbffff700 1 --2- 172.21.15.98:0/1427180752 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dac003fe0 0x7f3dac004810 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f3db00094d0 tx=0x7f3db000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb1aca476b446ffe server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:25.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.238+0000 7f3dbf7fe700 1 -- 172.21.15.98:0/1427180752 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3db000ea90 con 0x7f3dac003fe0 2023-10-06T05:07:25.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.238+0000 7f3dbf7fe700 1 -- 172.21.15.98:0/1427180752 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3db000f040 con 0x7f3dac003fe0 2023-10-06T05:07:25.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.238+0000 7f3dbf7fe700 1 -- 172.21.15.98:0/1427180752 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3db000ee40 con 0x7f3dac003fe0 2023-10-06T05:07:25.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.239+0000 7f3dc6ea2700 1 -- 172.21.15.98:0/1427180752 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dac003fe0 msgr2=0x7f3dac004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:25.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.239+0000 7f3dc6ea2700 1 --2- 172.21.15.98:0/1427180752 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dac003fe0 0x7f3dac004810 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f3db00094d0 tx=0x7f3db000b040 comp rx=0 tx=0).stop 2023-10-06T05:07:25.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.239+0000 7f3dc543f700 1 -- 172.21.15.98:0/1427180752 reap_dead start 2023-10-06T05:07:25.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.239+0000 7f3dc6ea2700 1 -- 172.21.15.98:0/1427180752 shutdown_connections 2023-10-06T05:07:25.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.239+0000 7f3dc6ea2700 1 -- 172.21.15.98:0/1427180752 >> 172.21.15.98:0/1427180752 conn(0x7f3dc00707d0 msgr2=0x7f3dc0070bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:25.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.239+0000 7f3dc6ea2700 1 -- 172.21.15.98:0/1427180752 shutdown_connections 2023-10-06T05:07:25.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.239+0000 7f3dc6ea2700 1 -- 172.21.15.98:0/1427180752 wait complete. 2023-10-06T05:07:25.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.240+0000 7f3dc6ea2700 1 Processor -- start 2023-10-06T05:07:25.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.240+0000 7f3dc6ea2700 1 -- start start 2023-10-06T05:07:25.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.241+0000 7f3dc6ea2700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3dac003fe0 0x7f3dc015b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:25.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.241+0000 7f3dc6ea2700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3dc00c5c60 0x7f3dc00c60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:25.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.241+0000 7f3dc6ea2700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dc00c3e30 0x7f3dc00c4260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:25.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.241+0000 7f3dc6ea2700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc0144ab0 con 0x7f3dc00c3e30 2023-10-06T05:07:25.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.241+0000 7f3dc6ea2700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc00c6610 con 0x7f3dc00c5c60 2023-10-06T05:07:25.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.241+0000 7f3dc6ea2700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc0155d50 con 0x7f3dac003fe0 2023-10-06T05:07:25.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.241+0000 7f3dc543f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dc00c3e30 0x7f3dc00c4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:25.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.241+0000 7f3dc543f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dc00c3e30 0x7f3dc00c4260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47118/0 (socket says 172.21.15.98:47118) 2023-10-06T05:07:25.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.241+0000 7f3dc543f700 1 -- 172.21.15.98:0/407816791 learned_addr learned my addr 172.21.15.98:0/407816791 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:25.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.241+0000 7f3dc4c3e700 1 --2- 172.21.15.98:0/407816791 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3dac003fe0 0x7f3dc015b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:25.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.241+0000 7f3dbffff700 1 --2- 172.21.15.98:0/407816791 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3dc00c5c60 0x7f3dc00c60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:25.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.241+0000 7f3dc543f700 1 -- 172.21.15.98:0/407816791 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3dac003fe0 msgr2=0x7f3dc015b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:25.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.241+0000 7f3dc543f700 1 --2- 172.21.15.98:0/407816791 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3dac003fe0 0x7f3dc015b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:25.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.241+0000 7f3dc543f700 1 -- 172.21.15.98:0/407816791 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3dc00c5c60 msgr2=0x7f3dc00c60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:25.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.241+0000 7f3dc543f700 1 --2- 172.21.15.98:0/407816791 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3dc00c5c60 0x7f3dc00c60d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:25.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.241+0000 7f3dc543f700 1 -- 172.21.15.98:0/407816791 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dc00c68b0 con 0x7f3dc00c3e30 2023-10-06T05:07:25.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.241+0000 7f3dc543f700 1 --2- 172.21.15.98:0/407816791 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dc00c3e30 0x7f3dc00c4260 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f3dc00d1410 tx=0x7f3da8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:25.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.241+0000 7f3dbcff9700 1 -- 172.21.15.98:0/407816791 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3da8011ed0 con 0x7f3dc00c3e30 2023-10-06T05:07:25.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.242+0000 7f3dbcff9700 1 -- 172.21.15.98:0/407816791 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3da8015670 con 0x7f3dc00c3e30 2023-10-06T05:07:25.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.242+0000 7f3dc6ea2700 1 -- 172.21.15.98:0/407816791 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dc00c6b90 con 0x7f3dc00c3e30 2023-10-06T05:07:25.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.242+0000 7f3dbcff9700 1 -- 172.21.15.98:0/407816791 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3da8015a30 con 0x7f3dc00c3e30 2023-10-06T05:07:25.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.242+0000 7f3dc6ea2700 1 -- 172.21.15.98:0/407816791 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dc00c7150 con 0x7f3dc00c3e30 2023-10-06T05:07:25.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.243+0000 7f3dc6ea2700 1 -- 172.21.15.98:0/407816791 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3dc0064a90 con 0x7f3dc00c3e30 2023-10-06T05:07:25.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.243+0000 7f3dbcff9700 1 -- 172.21.15.98:0/407816791 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f3da8020460 con 0x7f3dc00c3e30 2023-10-06T05:07:25.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.243+0000 7f3dbcff9700 1 --2- 172.21.15.98:0/407816791 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f3da0066f30 0x7f3da00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:25.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.243+0000 7f3dbcff9700 1 -- 172.21.15.98:0/407816791 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f3da80849c0 con 0x7f3dc00c3e30 2023-10-06T05:07:25.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.243+0000 7f3dc4c3e700 1 --2- 172.21.15.98:0/407816791 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f3da0066f30 0x7f3da00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:25.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.244+0000 7f3dc4c3e700 1 --2- 172.21.15.98:0/407816791 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f3da0066f30 0x7f3da00693e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f3db8000fd0 tx=0x7f3db8001e90 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:25.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.248+0000 7f3dbcff9700 1 -- 172.21.15.98:0/407816791 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3da8055270 con 0x7f3dc00c3e30 2023-10-06T05:07:25.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.474+0000 7f3dc6ea2700 1 -- 172.21.15.98:0/407816791 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f3dc015da60 con 0x7f3dc00c3e30 2023-10-06T05:07:25.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.474+0000 7f3dbcff9700 1 -- 172.21.15.98:0/407816791 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+88 (secure 0 0 0) 0x7f3da8054e00 con 0x7f3dc00c3e30 2023-10-06T05:07:25.477 INFO:teuthology.orchestra.run.smithi098.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-10-06T05:07:25.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.476+0000 7f3dc6ea2700 1 -- 172.21.15.98:0/407816791 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f3da0066f30 msgr2=0x7f3da00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:25.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.476+0000 7f3dc6ea2700 1 --2- 172.21.15.98:0/407816791 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f3da0066f30 0x7f3da00693e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f3db8000fd0 tx=0x7f3db8001e90 comp rx=0 tx=0).stop 2023-10-06T05:07:25.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.476+0000 7f3dc6ea2700 1 -- 172.21.15.98:0/407816791 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dc00c3e30 msgr2=0x7f3dc00c4260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:25.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.476+0000 7f3dc6ea2700 1 --2- 172.21.15.98:0/407816791 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dc00c3e30 0x7f3dc00c4260 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f3dc00d1410 tx=0x7f3da8008040 comp rx=0 tx=0).stop 2023-10-06T05:07:25.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.476+0000 7f3dc6ea2700 1 -- 172.21.15.98:0/407816791 shutdown_connections 2023-10-06T05:07:25.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.476+0000 7f3dc6ea2700 1 --2- 172.21.15.98:0/407816791 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f3da0066f30 0x7f3da00693e0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:25.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.476+0000 7f3dc6ea2700 1 --2- 172.21.15.98:0/407816791 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dc00c3e30 0x7f3dc00c4260 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:25.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.476+0000 7f3dc6ea2700 1 --2- 172.21.15.98:0/407816791 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3dac003fe0 0x7f3dc015b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:25.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.476+0000 7f3dc6ea2700 1 --2- 172.21.15.98:0/407816791 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3dc00c5c60 0x7f3dc00c60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:25.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.476+0000 7f3dc6ea2700 1 -- 172.21.15.98:0/407816791 >> 172.21.15.98:0/407816791 conn(0x7f3dc00707d0 msgr2=0x7f3dc01568b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:25.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.476+0000 7f3dc6ea2700 1 -- 172.21.15.98:0/407816791 shutdown_connections 2023-10-06T05:07:25.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:25.476+0000 7f3dc6ea2700 1 -- 172.21.15.98:0/407816791 wait complete. 2023-10-06T05:07:25.490 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-10-06T05:07:25.490 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-06T05:07:25.490 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi098.front.sepia.ceph.com 2023-10-06T05:07:25.490 INFO:tasks.cephfs.mount:self.client.name = client.2 2023-10-06T05:07:25.490 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2023-10-06T05:07:25.490 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-10-06T05:07:25.491 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-06T05:07:25.491 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-06T05:07:25.491 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2023-10-06T05:07:25.491 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:07:25.491 DEBUG:teuthology.orchestra.run.smithi098:> ip addr 2023-10-06T05:07:25.506 INFO:teuthology.orchestra.run.smithi098.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-06T05:07:25.506 INFO:teuthology.orchestra.run.smithi098.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-06T05:07:25.506 INFO:teuthology.orchestra.run.smithi098.stdout: inet 127.0.0.1/8 scope host lo 2023-10-06T05:07:25.507 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-10-06T05:07:25.507 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 ::1/128 scope host 2023-10-06T05:07:25.507 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-10-06T05:07:25.508 INFO:teuthology.orchestra.run.smithi098.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-06T05:07:25.508 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:d8:d9:b8 brd ff:ff:ff:ff:ff:ff 2023-10-06T05:07:25.508 INFO:teuthology.orchestra.run.smithi098.stdout: altname enp7s0f0 2023-10-06T05:07:25.508 INFO:teuthology.orchestra.run.smithi098.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-06T05:07:25.508 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:d8:d9:b9 brd ff:ff:ff:ff:ff:ff 2023-10-06T05:07:25.509 INFO:teuthology.orchestra.run.smithi098.stdout: altname enp7s0f1 2023-10-06T05:07:25.509 INFO:teuthology.orchestra.run.smithi098.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-06T05:07:25.509 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:8f:d2:6a brd ff:ff:ff:ff:ff:ff 2023-10-06T05:07:25.509 INFO:teuthology.orchestra.run.smithi098.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-06T05:07:25.510 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:8f:d2:6b brd ff:ff:ff:ff:ff:ff 2023-10-06T05:07:25.510 INFO:teuthology.orchestra.run.smithi098.stdout: inet 172.21.15.98/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-10-06T05:07:25.510 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft 42480sec preferred_lft 42480sec 2023-10-06T05:07:25.510 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::ec4:7aff:fe8f:d26b/64 scope link 2023-10-06T05:07:25.511 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-10-06T05:07:25.511 INFO:teuthology.orchestra.run.smithi098.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-06T05:07:25.511 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether fa:c2:06:14:b9:4f brd ff:ff:ff:ff:ff:ff 2023-10-06T05:07:25.511 INFO:teuthology.orchestra.run.smithi098.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-06T05:07:25.511 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-10-06T05:07:25.512 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::f8f9:dcff:feb4:5363/64 scope link tentative 2023-10-06T05:07:25.512 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-10-06T05:07:25.512 INFO:teuthology.orchestra.run.smithi098.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-06T05:07:25.512 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether fa:c2:06:14:b9:4f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-06T05:07:25.513 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::f8c2:6ff:fe14:b94f/64 scope link tentative 2023-10-06T05:07:25.513 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-10-06T05:07:25.513 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:07:25.514 DEBUG:teuthology.orchestra.run.smithi098:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-06T05:07:25.569 INFO:teuthology.orchestra.run.smithi098.stdout:1 2023-10-06T05:07:25.570 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:07:25.570 DEBUG:teuthology.orchestra.run.smithi098:> route 2023-10-06T05:07:25.626 INFO:teuthology.orchestra.run.smithi098.stdout:Kernel IP routing table 2023-10-06T05:07:25.626 INFO:teuthology.orchestra.run.smithi098.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-06T05:07:25.626 INFO:teuthology.orchestra.run.smithi098.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-10-06T05:07:25.626 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-10-06T05:07:25.626 INFO:teuthology.orchestra.run.smithi098.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-06T05:07:25.626 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:07:25.627 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec bash -c ' 2023-10-06T05:07:25.627 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2023-10-06T05:07:25.627 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-10-06T05:07:25.627 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-10-06T05:07:25.627 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-10-06T05:07:25.627 DEBUG:teuthology.orchestra.run.smithi098:> ') 2023-10-06T05:07:25.726 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:07:25.726 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list 2023-10-06T05:07:25.741 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-06T05:07:25.741 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:07:25.741 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list-id 2023-10-06T05:07:25.796 INFO:teuthology.orchestra.run.smithi098.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2023-10-06T05:07:25.796 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:07:25.796 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec bash -c ' 2023-10-06T05:07:25.796 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2023-10-06T05:07:25.796 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-06T05:07:25.797 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2023-10-06T05:07:25.797 DEBUG:teuthology.orchestra.run.smithi098:> ') 2023-10-06T05:07:25.884 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:07:25.884 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2023-10-06T05:07:25.917 INFO:teuthology.orchestra.run.smithi098.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-06T05:07:25.917 INFO:teuthology.orchestra.run.smithi098.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-06T05:07:25.917 INFO:teuthology.orchestra.run.smithi098.stdout: inet 127.0.0.1/8 scope host lo 2023-10-06T05:07:25.917 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-10-06T05:07:25.917 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 ::1/128 scope host 2023-10-06T05:07:25.918 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-10-06T05:07:25.918 INFO:teuthology.orchestra.run.smithi098.stdout:2: veth0@if9: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-06T05:07:25.918 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 72:07:1a:c9:1f:9d brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-10-06T05:07:25.918 INFO:teuthology.orchestra.run.smithi098.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-10-06T05:07:25.918 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-10-06T05:07:25.918 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::7007:1aff:fec9:1f9d/64 scope link tentative 2023-10-06T05:07:25.918 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-10-06T05:07:25.931 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:07:25.931 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2023-10-06T05:07:26.005 INFO:teuthology.orchestra.run.smithi098.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-06T05:07:26.005 INFO:teuthology.orchestra.run.smithi098.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-06T05:07:26.005 INFO:teuthology.orchestra.run.smithi098.stdout: inet 127.0.0.1/8 scope host lo 2023-10-06T05:07:26.005 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-10-06T05:07:26.005 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 ::1/128 scope host 2023-10-06T05:07:26.005 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-10-06T05:07:26.005 INFO:teuthology.orchestra.run.smithi098.stdout:2: veth0@if9: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-06T05:07:26.005 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 72:07:1a:c9:1f:9d brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-10-06T05:07:26.005 INFO:teuthology.orchestra.run.smithi098.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-10-06T05:07:26.005 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-10-06T05:07:26.006 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::7007:1aff:fec9:1f9d/64 scope link 2023-10-06T05:07:26.006 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-10-06T05:07:26.014 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2023-10-06T05:07:26.014 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:07:26.015 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec bash -c ' 2023-10-06T05:07:26.015 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2023-10-06T05:07:26.015 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2023-10-06T05:07:26.015 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2023-10-06T05:07:26.015 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2023-10-06T05:07:26.015 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2023-10-06T05:07:26.015 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2023-10-06T05:07:26.015 DEBUG:teuthology.orchestra.run.smithi098:> ') 2023-10-06T05:07:26.245 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:07:26.246 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec bash -c ' 2023-10-06T05:07:26.246 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2023-10-06T05:07:26.246 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set brx.1 up 2023-10-06T05:07:26.246 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set dev brx.1 master ceph-brx 2023-10-06T05:07:26.247 DEBUG:teuthology.orchestra.run.smithi098:> ') 2023-10-06T05:07:26.326 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-06T05:07:26.326 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2023-10-06T05:07:26.340 INFO:teuthology.orchestra.run.smithi098.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2023-10-06T05:07:26.340 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-06T05:07:26.341 DEBUG:teuthology.orchestra.run.smithi098:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2023-10-06T05:07:26.395 INFO:tasks.cephfs.kernel_mount:mounting using device: 2@.cephfs=/ 2023-10-06T05:07:26.396 INFO:teuthology.orchestra.run:Running command with timeout 1800 2023-10-06T05:07:26.396 DEBUG:teuthology.orchestra.run.smithi098:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 2@.cephfs=/ /home/ubuntu/cephtest/mnt.2 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-06T05:07:26.476 INFO:teuthology.orchestra.run.smithi098.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-06T05:07:26.476 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-06T05:07:26.494 INFO:teuthology.orchestra.run.smithi098.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-06T05:07:26.494 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-06T05:07:26.494 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: resolved to: "172.21.15.98:3300,172.21.15.174:3300,172.21.15.174:3301" 2023-10-06T05:07:26.495 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: trying mount with new device syntax: 2@f50c5df5-cad5-47a4-9bb1-8c791d4311ec.cephfs=/ 2023-10-06T05:07:26.495 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,ms_mode=prefer-crc,key=2,mon_addr=172.21.15.98:3300/172.21.15.174:3300/172.21.15.174:3301" will pass to kernel 2023-10-06T05:07:26.495 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: v2 mount support check returned 2 2023-10-06T05:07:26.495 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: kernel does not support v2 syntax 2023-10-06T05:07:26.495 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-06T05:07:26.516 INFO:teuthology.orchestra.run.smithi098.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-06T05:07:26.516 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-06T05:07:26.516 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: resolved to: "172.21.15.98:3300,172.21.15.174:3300,172.21.15.174:3301" 2023-10-06T05:07:26.516 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: trying mount with new device syntax: 2@f50c5df5-cad5-47a4-9bb1-8c791d4311ec.cephfs=/ 2023-10-06T05:07:26.517 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,ms_mode=prefer-crc,key=2,mon_addr=172.21.15.98:3300/172.21.15.174:3300/172.21.15.174:3301" will pass to kernel 2023-10-06T05:07:26.517 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: v2 mount support check returned 2 2023-10-06T05:07:26.517 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: kernel does not support v2 syntax 2023-10-06T05:07:26.517 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-06T05:07:26.517 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: resolved to: "172.21.15.98:6789,172.21.15.174:6789,172.21.15.174:6790" 2023-10-06T05:07:26.517 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.98:6789,172.21.15.174:6789,172.21.15.174:6790:/ 2023-10-06T05:07:26.517 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,key=2,mds_namespace=cephfs,fsid=f50c5df5-cad5-47a4-9bb1-8c791d4311ec" will pass to kernel 2023-10-06T05:07:26.519 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-06T05:07:26.519 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:07:26.519 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2023-10-06T05:07:26.583 INFO:tasks.kclient:client.1 config is {'syntax': 'v2'} 2023-10-06T05:07:26.583 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-06T05:07:26.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.736+0000 7fccd24d0700 1 Processor -- start 2023-10-06T05:07:26.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.736+0000 7fccd24d0700 1 -- start start 2023-10-06T05:07:26.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.736+0000 7fccd24d0700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fcccc156350 0x7fcccc0cc1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:26.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.736+0000 7fccd24d0700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcccc0cc710 0x7fcccc06bc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:26.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.736+0000 7fccd24d0700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fcccc06c160 0x7fcccc0cafb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:26.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.736+0000 7fccd24d0700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcccc0ccb60 con 0x7fcccc06c160 2023-10-06T05:07:26.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.736+0000 7fccd24d0700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcccc06c9a0 con 0x7fcccc156350 2023-10-06T05:07:26.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.736+0000 7fccd24d0700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcccc156b20 con 0x7fcccc0cc710 2023-10-06T05:07:26.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.737+0000 7fccd0a6d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fcccc06c160 0x7fcccc0cafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:26.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.737+0000 7fccd0a6d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fcccc06c160 0x7fcccc0cafb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47126/0 (socket says 172.21.15.98:47126) 2023-10-06T05:07:26.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.737+0000 7fccd0a6d700 1 -- 172.21.15.98:0/578458658 learned_addr learned my addr 172.21.15.98:0/578458658 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:26.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.737+0000 7fcccb7fe700 1 --2- 172.21.15.98:0/578458658 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcccc0cc710 0x7fcccc06bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:26.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.737+0000 7fcccbfff700 1 --2- 172.21.15.98:0/578458658 >> v2:172.21.15.174:3300/0 conn(0x7fcccc156350 0x7fcccc0cc1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:26.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.737+0000 7fccd0a6d700 1 -- 172.21.15.98:0/578458658 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcccc0cc710 msgr2=0x7fcccc06bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:26.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.737+0000 7fccd0a6d700 1 --2- 172.21.15.98:0/578458658 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcccc0cc710 0x7fcccc06bc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:26.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.737+0000 7fccd0a6d700 1 -- 172.21.15.98:0/578458658 >> v2:172.21.15.174:3300/0 conn(0x7fcccc156350 msgr2=0x7fcccc0cc1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:26.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.737+0000 7fccd0a6d700 1 --2- 172.21.15.98:0/578458658 >> v2:172.21.15.174:3300/0 conn(0x7fcccc156350 0x7fcccc0cc1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:26.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.737+0000 7fccd0a6d700 1 -- 172.21.15.98:0/578458658 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcccc156e00 con 0x7fcccc06c160 2023-10-06T05:07:26.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.737+0000 7fccd0a6d700 1 --2- 172.21.15.98:0/578458658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcccc06c160 0x7fcccc0cafb0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fcccc069f50 tx=0x7fccc4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd3cedec21603f4a server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:26.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.737+0000 7fcccaffd700 1 -- 172.21.15.98:0/578458658 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc400e070 con 0x7fcccc06c160 2023-10-06T05:07:26.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.737+0000 7fcccaffd700 1 -- 172.21.15.98:0/578458658 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccc400ba50 con 0x7fcccc06c160 2023-10-06T05:07:26.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.737+0000 7fcccaffd700 1 -- 172.21.15.98:0/578458658 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc40092e0 con 0x7fcccc06c160 2023-10-06T05:07:26.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.738+0000 7fccd24d0700 1 -- 172.21.15.98:0/578458658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcccc06c160 msgr2=0x7fcccc0cafb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:26.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.738+0000 7fccd24d0700 1 --2- 172.21.15.98:0/578458658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcccc06c160 0x7fcccc0cafb0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fcccc069f50 tx=0x7fccc4009840 comp rx=0 tx=0).stop 2023-10-06T05:07:26.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.738+0000 7fccd24d0700 1 -- 172.21.15.98:0/578458658 shutdown_connections 2023-10-06T05:07:26.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.738+0000 7fccd24d0700 1 --2- 172.21.15.98:0/578458658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcccc06c160 0x7fcccc0cafb0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:26.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.738+0000 7fccd24d0700 1 --2- 172.21.15.98:0/578458658 >> v2:172.21.15.174:3300/0 conn(0x7fcccc156350 0x7fcccc0cc1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:26.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.738+0000 7fccd24d0700 1 --2- 172.21.15.98:0/578458658 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcccc0cc710 0x7fcccc06bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:26.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.738+0000 7fccd24d0700 1 -- 172.21.15.98:0/578458658 >> 172.21.15.98:0/578458658 conn(0x7fcccc0cdeb0 msgr2=0x7fcccc0c9e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:26.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.738+0000 7fccd24d0700 1 -- 172.21.15.98:0/578458658 shutdown_connections 2023-10-06T05:07:26.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.738+0000 7fccd24d0700 1 -- 172.21.15.98:0/578458658 wait complete. 2023-10-06T05:07:26.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.739+0000 7fccd24d0700 1 Processor -- start 2023-10-06T05:07:26.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.739+0000 7fccd24d0700 1 -- start start 2023-10-06T05:07:26.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.740+0000 7fccd24d0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcccc06c160 0x7fcccc0b0180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:26.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.740+0000 7fccd24d0700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcccc0cc710 0x7fcccc1673d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:26.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.740+0000 7fccd24d0700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcccc156350 0x7fcccc151b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:26.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.740+0000 7fccd24d0700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcccc167910 con 0x7fcccc06c160 2023-10-06T05:07:26.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.740+0000 7fccd24d0700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcccc0a5490 con 0x7fcccc156350 2023-10-06T05:07:26.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.740+0000 7fccd24d0700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcccc0b0860 con 0x7fcccc0cc710 2023-10-06T05:07:26.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.740+0000 7fcccbfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcccc06c160 0x7fcccc0b0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:26.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.740+0000 7fcccbfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcccc06c160 0x7fcccc0b0180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47130/0 (socket says 172.21.15.98:47130) 2023-10-06T05:07:26.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.740+0000 7fcccbfff700 1 -- 172.21.15.98:0/2455402504 learned_addr learned my addr 172.21.15.98:0/2455402504 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:26.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.740+0000 7fcccb7fe700 1 --2- 172.21.15.98:0/2455402504 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcccc0cc710 0x7fcccc1673d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:26.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.740+0000 7fccd0a6d700 1 --2- 172.21.15.98:0/2455402504 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcccc156350 0x7fcccc151b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:26.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.740+0000 7fcccbfff700 1 -- 172.21.15.98:0/2455402504 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcccc0cc710 msgr2=0x7fcccc1673d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:26.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.740+0000 7fcccbfff700 1 --2- 172.21.15.98:0/2455402504 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcccc0cc710 0x7fcccc1673d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:26.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.740+0000 7fcccbfff700 1 -- 172.21.15.98:0/2455402504 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcccc156350 msgr2=0x7fcccc151b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:26.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.740+0000 7fcccbfff700 1 --2- 172.21.15.98:0/2455402504 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcccc156350 0x7fcccc151b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:26.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.740+0000 7fcccbfff700 1 -- 172.21.15.98:0/2455402504 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccc4009ae0 con 0x7fcccc06c160 2023-10-06T05:07:26.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.740+0000 7fcccbfff700 1 --2- 172.21.15.98:0/2455402504 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcccc06c160 0x7fcccc0b0180 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fccbc00d860 tx=0x7fccbc008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:26.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.740+0000 7fccb3fff700 1 -- 172.21.15.98:0/2455402504 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccbc0125f0 con 0x7fcccc06c160 2023-10-06T05:07:26.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.740+0000 7fccb3fff700 1 -- 172.21.15.98:0/2455402504 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccbc012750 con 0x7fcccc06c160 2023-10-06T05:07:26.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.740+0000 7fccd24d0700 1 -- 172.21.15.98:0/2455402504 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcccc0b0b40 con 0x7fcccc06c160 2023-10-06T05:07:26.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.740+0000 7fccb3fff700 1 -- 172.21.15.98:0/2455402504 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccbc012b10 con 0x7fcccc06c160 2023-10-06T05:07:26.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.741+0000 7fccd24d0700 1 -- 172.21.15.98:0/2455402504 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcccc0b1100 con 0x7fcccc06c160 2023-10-06T05:07:26.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.741+0000 7fccd24d0700 1 -- 172.21.15.98:0/2455402504 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcccc0a5770 con 0x7fcccc06c160 2023-10-06T05:07:26.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.742+0000 7fccb3fff700 1 -- 172.21.15.98:0/2455402504 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fccbc013480 con 0x7fcccc06c160 2023-10-06T05:07:26.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.742+0000 7fccb3fff700 1 --2- 172.21.15.98:0/2455402504 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fccb8066f30 0x7fccb80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:26.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.742+0000 7fcccb7fe700 1 --2- 172.21.15.98:0/2455402504 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fccb8066f30 0x7fccb80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:26.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.742+0000 7fccb3fff700 1 -- 172.21.15.98:0/2455402504 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fccbc084c80 con 0x7fcccc06c160 2023-10-06T05:07:26.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.743+0000 7fcccb7fe700 1 --2- 172.21.15.98:0/2455402504 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fccb8066f30 0x7fccb80693e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fcccc169ee0 tx=0x7fccc0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:26.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.747+0000 7fccb3fff700 1 -- 172.21.15.98:0/2455402504 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fccbc055530 con 0x7fcccc06c160 2023-10-06T05:07:26.937 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-06T05:07:26.939 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-06T05:07:26.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.978+0000 7fccd24d0700 1 -- 172.21.15.98:0/2455402504 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fcccc064a90 con 0x7fcccc06c160 2023-10-06T05:07:26.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.978+0000 7fccb3fff700 1 -- 172.21.15.98:0/2455402504 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+88 (secure 0 0 0) 0x7fccbc084070 con 0x7fcccc06c160 2023-10-06T05:07:26.981 INFO:teuthology.orchestra.run.smithi098.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-10-06T05:07:26.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.980+0000 7fccd24d0700 1 -- 172.21.15.98:0/2455402504 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fccb8066f30 msgr2=0x7fccb80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:26.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.980+0000 7fccd24d0700 1 --2- 172.21.15.98:0/2455402504 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fccb8066f30 0x7fccb80693e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fcccc169ee0 tx=0x7fccc0008040 comp rx=0 tx=0).stop 2023-10-06T05:07:26.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.980+0000 7fccd24d0700 1 -- 172.21.15.98:0/2455402504 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcccc06c160 msgr2=0x7fcccc0b0180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:26.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.980+0000 7fccd24d0700 1 --2- 172.21.15.98:0/2455402504 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcccc06c160 0x7fcccc0b0180 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fccbc00d860 tx=0x7fccbc008040 comp rx=0 tx=0).stop 2023-10-06T05:07:26.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.981+0000 7fccd24d0700 1 -- 172.21.15.98:0/2455402504 shutdown_connections 2023-10-06T05:07:26.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.981+0000 7fccd24d0700 1 --2- 172.21.15.98:0/2455402504 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcccc156350 0x7fcccc151b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:26.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.981+0000 7fccd24d0700 1 --2- 172.21.15.98:0/2455402504 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fccb8066f30 0x7fccb80693e0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:26.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.981+0000 7fccd24d0700 1 --2- 172.21.15.98:0/2455402504 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcccc06c160 0x7fcccc0b0180 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:26.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.981+0000 7fccd24d0700 1 --2- 172.21.15.98:0/2455402504 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcccc0cc710 0x7fcccc1673d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:26.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.981+0000 7fccd24d0700 1 -- 172.21.15.98:0/2455402504 >> 172.21.15.98:0/2455402504 conn(0x7fcccc0cdeb0 msgr2=0x7fcccc0c9450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:26.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.981+0000 7fccd24d0700 1 -- 172.21.15.98:0/2455402504 shutdown_connections 2023-10-06T05:07:26.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:26.981+0000 7fccd24d0700 1 -- 172.21.15.98:0/2455402504 wait complete. 2023-10-06T05:07:26.995 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-10-06T05:07:26.995 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-06T05:07:26.995 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi174.front.sepia.ceph.com 2023-10-06T05:07:26.995 INFO:tasks.cephfs.mount:self.client.name = client.1 2023-10-06T05:07:26.995 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-10-06T05:07:26.995 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-10-06T05:07:26.995 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-06T05:07:26.995 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-06T05:07:26.995 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2023-10-06T05:07:26.995 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:07:26.996 DEBUG:teuthology.orchestra.run.smithi174:> ip addr 2023-10-06T05:07:27.013 INFO:teuthology.orchestra.run.smithi174.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-06T05:07:27.013 INFO:teuthology.orchestra.run.smithi174.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-06T05:07:27.013 INFO:teuthology.orchestra.run.smithi174.stdout: inet 127.0.0.1/8 scope host lo 2023-10-06T05:07:27.014 INFO:teuthology.orchestra.run.smithi174.stdout: valid_lft forever preferred_lft forever 2023-10-06T05:07:27.014 INFO:teuthology.orchestra.run.smithi174.stdout: inet6 ::1/128 scope host 2023-10-06T05:07:27.014 INFO:teuthology.orchestra.run.smithi174.stdout: valid_lft forever preferred_lft forever 2023-10-06T05:07:27.014 INFO:teuthology.orchestra.run.smithi174.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-06T05:07:27.014 INFO:teuthology.orchestra.run.smithi174.stdout: link/ether 0c:c4:7a:d9:36:e0 brd ff:ff:ff:ff:ff:ff 2023-10-06T05:07:27.014 INFO:teuthology.orchestra.run.smithi174.stdout: altname enp7s0f0 2023-10-06T05:07:27.014 INFO:teuthology.orchestra.run.smithi174.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-06T05:07:27.014 INFO:teuthology.orchestra.run.smithi174.stdout: link/ether 0c:c4:7a:d9:36:e1 brd ff:ff:ff:ff:ff:ff 2023-10-06T05:07:27.014 INFO:teuthology.orchestra.run.smithi174.stdout: altname enp7s0f1 2023-10-06T05:07:27.014 INFO:teuthology.orchestra.run.smithi174.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-06T05:07:27.014 INFO:teuthology.orchestra.run.smithi174.stdout: link/ether 0c:c4:7a:88:77:1a brd ff:ff:ff:ff:ff:ff 2023-10-06T05:07:27.015 INFO:teuthology.orchestra.run.smithi174.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-06T05:07:27.015 INFO:teuthology.orchestra.run.smithi174.stdout: link/ether 0c:c4:7a:88:77:1b brd ff:ff:ff:ff:ff:ff 2023-10-06T05:07:27.015 INFO:teuthology.orchestra.run.smithi174.stdout: inet 172.21.15.174/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-10-06T05:07:27.015 INFO:teuthology.orchestra.run.smithi174.stdout: valid_lft 42478sec preferred_lft 42478sec 2023-10-06T05:07:27.015 INFO:teuthology.orchestra.run.smithi174.stdout: inet6 fe80::ec4:7aff:fe88:771b/64 scope link 2023-10-06T05:07:27.015 INFO:teuthology.orchestra.run.smithi174.stdout: valid_lft forever preferred_lft forever 2023-10-06T05:07:27.015 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-10-06T05:07:27.015 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:07:27.015 DEBUG:teuthology.orchestra.run.smithi174:> (cd / && exec bash -c ' 2023-10-06T05:07:27.016 DEBUG:teuthology.orchestra.run.smithi174:> set -e 2023-10-06T05:07:27.016 DEBUG:teuthology.orchestra.run.smithi174:> sudo ip link add name ceph-brx type bridge 2023-10-06T05:07:27.016 DEBUG:teuthology.orchestra.run.smithi174:> sudo ip addr flush dev ceph-brx 2023-10-06T05:07:27.016 DEBUG:teuthology.orchestra.run.smithi174:> sudo ip link set ceph-brx up 2023-10-06T05:07:27.016 DEBUG:teuthology.orchestra.run.smithi174:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-10-06T05:07:27.016 DEBUG:teuthology.orchestra.run.smithi174:> ') 2023-10-06T05:07:27.186 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:07:27.186 DEBUG:teuthology.orchestra.run.smithi174:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-06T05:07:27.229 INFO:teuthology.orchestra.run.smithi174.stdout:1 2023-10-06T05:07:27.230 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:07:27.230 DEBUG:teuthology.orchestra.run.smithi174:> route 2023-10-06T05:07:27.297 INFO:teuthology.orchestra.run.smithi174.stdout:Kernel IP routing table 2023-10-06T05:07:27.297 INFO:teuthology.orchestra.run.smithi174.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-06T05:07:27.298 INFO:teuthology.orchestra.run.smithi174.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-10-06T05:07:27.298 INFO:teuthology.orchestra.run.smithi174.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-10-06T05:07:27.298 INFO:teuthology.orchestra.run.smithi174.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-06T05:07:27.298 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:07:27.298 DEBUG:teuthology.orchestra.run.smithi174:> (cd / && exec bash -c ' 2023-10-06T05:07:27.298 DEBUG:teuthology.orchestra.run.smithi174:> set -e 2023-10-06T05:07:27.298 DEBUG:teuthology.orchestra.run.smithi174:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-10-06T05:07:27.298 DEBUG:teuthology.orchestra.run.smithi174:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-10-06T05:07:27.299 DEBUG:teuthology.orchestra.run.smithi174:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-10-06T05:07:27.299 DEBUG:teuthology.orchestra.run.smithi174:> ') 2023-10-06T05:07:27.536 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:07:27.537 DEBUG:teuthology.orchestra.run.smithi174:> ip netns list 2023-10-06T05:07:27.551 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:07:27.551 DEBUG:teuthology.orchestra.run.smithi174:> ip netns list-id 2023-10-06T05:07:27.605 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:07:27.605 DEBUG:teuthology.orchestra.run.smithi174:> (cd / && exec bash -c ' 2023-10-06T05:07:27.606 DEBUG:teuthology.orchestra.run.smithi174:> set -e 2023-10-06T05:07:27.606 DEBUG:teuthology.orchestra.run.smithi174:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2023-10-06T05:07:27.606 DEBUG:teuthology.orchestra.run.smithi174:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2023-10-06T05:07:27.606 DEBUG:teuthology.orchestra.run.smithi174:> ') 2023-10-06T05:07:27.694 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2023-10-06T05:07:27.695 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:07:27.695 DEBUG:teuthology.orchestra.run.smithi174:> (cd / && exec bash -c ' 2023-10-06T05:07:27.695 DEBUG:teuthology.orchestra.run.smithi174:> set -e 2023-10-06T05:07:27.695 DEBUG:teuthology.orchestra.run.smithi174:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2023-10-06T05:07:27.695 DEBUG:teuthology.orchestra.run.smithi174:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-10-06T05:07:27.695 DEBUG:teuthology.orchestra.run.smithi174:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2023-10-06T05:07:27.695 DEBUG:teuthology.orchestra.run.smithi174:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2023-10-06T05:07:27.695 DEBUG:teuthology.orchestra.run.smithi174:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2023-10-06T05:07:27.695 DEBUG:teuthology.orchestra.run.smithi174:> ') 2023-10-06T05:07:27.896 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:07:27.896 DEBUG:teuthology.orchestra.run.smithi174:> (cd / && exec bash -c ' 2023-10-06T05:07:27.897 DEBUG:teuthology.orchestra.run.smithi174:> set -e 2023-10-06T05:07:27.897 DEBUG:teuthology.orchestra.run.smithi174:> sudo ip link set brx.0 up 2023-10-06T05:07:27.897 DEBUG:teuthology.orchestra.run.smithi174:> sudo ip link set dev brx.0 master ceph-brx 2023-10-06T05:07:27.897 DEBUG:teuthology.orchestra.run.smithi174:> ') 2023-10-06T05:07:27.988 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-06T05:07:27.989 DEBUG:teuthology.orchestra.run.smithi174:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-10-06T05:07:28.004 INFO:teuthology.orchestra.run.smithi174.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-10-06T05:07:28.004 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-06T05:07:28.004 DEBUG:teuthology.orchestra.run.smithi174:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-10-06T05:07:28.058 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2023-10-06T05:07:28.058 INFO:teuthology.orchestra.run:Running command with timeout 1800 2023-10-06T05:07:28.059 DEBUG:teuthology.orchestra.run.smithi174:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-06T05:07:28.156 INFO:teuthology.orchestra.run.smithi174.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-06T05:07:28.156 INFO:teuthology.orchestra.run.smithi174.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-06T05:07:28.243 INFO:teuthology.orchestra.run.smithi174.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-06T05:07:28.243 INFO:teuthology.orchestra.run.smithi174.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-06T05:07:28.243 INFO:teuthology.orchestra.run.smithi174.stdout:mount.ceph: resolved to: "172.21.15.98:3300,172.21.15.174:3300,172.21.15.174:3301" 2023-10-06T05:07:28.244 INFO:teuthology.orchestra.run.smithi174.stdout:mount.ceph: trying mount with new device syntax: 1@f50c5df5-cad5-47a4-9bb1-8c791d4311ec.cephfs=/ 2023-10-06T05:07:28.244 INFO:teuthology.orchestra.run.smithi174.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.98:3300/172.21.15.174:3300/172.21.15.174:3301" will pass to kernel 2023-10-06T05:07:28.244 INFO:teuthology.orchestra.run.smithi174.stdout:mount.ceph: v2 mount support check returned 2 2023-10-06T05:07:28.244 INFO:teuthology.orchestra.run.smithi174.stdout:mount.ceph: kernel does not support v2 syntax 2023-10-06T05:07:28.244 INFO:teuthology.orchestra.run.smithi174.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-06T05:07:28.265 INFO:teuthology.orchestra.run.smithi174.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-06T05:07:28.266 INFO:teuthology.orchestra.run.smithi174.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-06T05:07:28.266 INFO:teuthology.orchestra.run.smithi174.stdout:mount.ceph: resolved to: "172.21.15.98:3300,172.21.15.174:3300,172.21.15.174:3301" 2023-10-06T05:07:28.266 INFO:teuthology.orchestra.run.smithi174.stdout:mount.ceph: trying mount with new device syntax: 1@f50c5df5-cad5-47a4-9bb1-8c791d4311ec.cephfs=/ 2023-10-06T05:07:28.266 INFO:teuthology.orchestra.run.smithi174.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.98:3300/172.21.15.174:3300/172.21.15.174:3301" will pass to kernel 2023-10-06T05:07:28.266 INFO:teuthology.orchestra.run.smithi174.stdout:mount.ceph: v2 mount support check returned 2 2023-10-06T05:07:28.266 INFO:teuthology.orchestra.run.smithi174.stdout:mount.ceph: kernel does not support v2 syntax 2023-10-06T05:07:28.266 INFO:teuthology.orchestra.run.smithi174.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-06T05:07:28.266 INFO:teuthology.orchestra.run.smithi174.stdout:mount.ceph: resolved to: "172.21.15.98:6789,172.21.15.174:6789,172.21.15.174:6790" 2023-10-06T05:07:28.266 INFO:teuthology.orchestra.run.smithi174.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.98:6789,172.21.15.174:6789,172.21.15.174:6790:/ 2023-10-06T05:07:28.266 INFO:teuthology.orchestra.run.smithi174.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,key=1,mds_namespace=cephfs,fsid=f50c5df5-cad5-47a4-9bb1-8c791d4311ec" will pass to kernel 2023-10-06T05:07:28.268 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-06T05:07:28.268 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:07:28.268 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-10-06T05:07:28.292 INFO:tasks.kclient:client.3 config is {'syntax': 'v2'} 2023-10-06T05:07:28.292 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-06T05:07:28.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.416+0000 7fe1643cf700 1 Processor -- start 2023-10-06T05:07:28.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.416+0000 7fe1643cf700 1 -- start start 2023-10-06T05:07:28.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.417+0000 7fe1643cf700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fe15c0ac000 0x7fe15c0adf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:28.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.417+0000 7fe1643cf700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe15c0cc7f0 0x7fe15c06e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:28.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.417+0000 7fe1643cf700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe15c0ccc60 0x7fe15c069520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:28.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.417+0000 7fe1643cf700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe15c156110 con 0x7fe15c0cc7f0 2023-10-06T05:07:28.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.417+0000 7fe1643cf700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe15c0c7330 con 0x7fe15c0ac000 2023-10-06T05:07:28.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.417+0000 7fe1643cf700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe15c0ae460 con 0x7fe15c0ccc60 2023-10-06T05:07:28.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.417+0000 7fe16196a700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe15c0cc7f0 0x7fe15c06e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:28.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.417+0000 7fe16196a700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe15c0cc7f0 0x7fe15c06e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47134/0 (socket says 172.21.15.98:47134) 2023-10-06T05:07:28.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.417+0000 7fe16196a700 1 -- 172.21.15.98:0/566969992 learned_addr learned my addr 172.21.15.98:0/566969992 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:28.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.417+0000 7fe16216b700 1 --2- 172.21.15.98:0/566969992 >> v2:172.21.15.174:3300/0 conn(0x7fe15c0ac000 0x7fe15c0adf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:28.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.417+0000 7fe16296c700 1 --2- 172.21.15.98:0/566969992 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe15c0ccc60 0x7fe15c069520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:28.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.417+0000 7fe16196a700 1 -- 172.21.15.98:0/566969992 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe15c0ccc60 msgr2=0x7fe15c069520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:28.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.417+0000 7fe16196a700 1 --2- 172.21.15.98:0/566969992 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe15c0ccc60 0x7fe15c069520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:28.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.417+0000 7fe16196a700 1 -- 172.21.15.98:0/566969992 >> v2:172.21.15.174:3300/0 conn(0x7fe15c0ac000 msgr2=0x7fe15c0adf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:28.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.417+0000 7fe16196a700 1 --2- 172.21.15.98:0/566969992 >> v2:172.21.15.174:3300/0 conn(0x7fe15c0ac000 0x7fe15c0adf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:28.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.417+0000 7fe16196a700 1 -- 172.21.15.98:0/566969992 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe15c0c6720 con 0x7fe15c0cc7f0 2023-10-06T05:07:28.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.417+0000 7fe16216b700 1 --2- 172.21.15.98:0/566969992 >> v2:172.21.15.174:3300/0 conn(0x7fe15c0ac000 0x7fe15c0adf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:07:28.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.417+0000 7fe16296c700 1 --2- 172.21.15.98:0/566969992 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe15c0ccc60 0x7fe15c069520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:07:28.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.417+0000 7fe16196a700 1 --2- 172.21.15.98:0/566969992 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe15c0cc7f0 0x7fe15c06e240 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fe15c064f70 tx=0x7fe150009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4638da3ee41e6031 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:28.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.417+0000 7fe161169700 1 -- 172.21.15.98:0/566969992 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe15000b860 con 0x7fe15c0cc7f0 2023-10-06T05:07:28.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.418+0000 7fe161169700 1 -- 172.21.15.98:0/566969992 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe15000b9c0 con 0x7fe15c0cc7f0 2023-10-06T05:07:28.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.418+0000 7fe161169700 1 -- 172.21.15.98:0/566969992 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe15000bd60 con 0x7fe15c0cc7f0 2023-10-06T05:07:28.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.418+0000 7fe1643cf700 1 -- 172.21.15.98:0/566969992 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe15c0cc7f0 msgr2=0x7fe15c06e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:28.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.418+0000 7fe1643cf700 1 --2- 172.21.15.98:0/566969992 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe15c0cc7f0 0x7fe15c06e240 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fe15c064f70 tx=0x7fe150009250 comp rx=0 tx=0).stop 2023-10-06T05:07:28.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.418+0000 7fe1643cf700 1 -- 172.21.15.98:0/566969992 shutdown_connections 2023-10-06T05:07:28.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.418+0000 7fe1643cf700 1 --2- 172.21.15.98:0/566969992 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe15c0ccc60 0x7fe15c069520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:28.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.418+0000 7fe1643cf700 1 --2- 172.21.15.98:0/566969992 >> v2:172.21.15.174:3300/0 conn(0x7fe15c0ac000 0x7fe15c0adf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:28.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.418+0000 7fe1643cf700 1 --2- 172.21.15.98:0/566969992 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe15c0cc7f0 0x7fe15c06e240 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:28.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.418+0000 7fe1643cf700 1 -- 172.21.15.98:0/566969992 >> 172.21.15.98:0/566969992 conn(0x7fe15c068080 msgr2=0x7fe15c0cc0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:28.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.418+0000 7fe1643cf700 1 -- 172.21.15.98:0/566969992 shutdown_connections 2023-10-06T05:07:28.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.418+0000 7fe1643cf700 1 -- 172.21.15.98:0/566969992 wait complete. 2023-10-06T05:07:28.427 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.419+0000 7fe1643cf700 1 Processor -- start 2023-10-06T05:07:28.427 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.419+0000 7fe1643cf700 1 -- start start 2023-10-06T05:07:28.427 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.420+0000 7fe1643cf700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe15c0ac000 0x7fe15c14a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:28.427 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.420+0000 7fe1643cf700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe15c0cc7f0 0x7fe15c0bf9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:28.428 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.420+0000 7fe1643cf700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe15c0ccc60 0x7fe15c154910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:28.428 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.420+0000 7fe1643cf700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe15c0c5150 con 0x7fe15c0ac000 2023-10-06T05:07:28.428 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.420+0000 7fe1643cf700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe15c14c9e0 con 0x7fe15c0ccc60 2023-10-06T05:07:28.428 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.420+0000 7fe1643cf700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe15c0c6180 con 0x7fe15c0cc7f0 2023-10-06T05:07:28.428 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.420+0000 7fe16216b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe15c0ac000 0x7fe15c14a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:28.428 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.420+0000 7fe16216b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe15c0ac000 0x7fe15c14a700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47150/0 (socket says 172.21.15.98:47150) 2023-10-06T05:07:28.428 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.420+0000 7fe16216b700 1 -- 172.21.15.98:0/4179955541 learned_addr learned my addr 172.21.15.98:0/4179955541 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:28.428 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.420+0000 7fe16296c700 1 --2- 172.21.15.98:0/4179955541 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe15c0ccc60 0x7fe15c154910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:28.428 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.420+0000 7fe16216b700 1 -- 172.21.15.98:0/4179955541 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe15c0cc7f0 msgr2=0x7fe15c0bf9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:28.428 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.420+0000 7fe16216b700 1 --2- 172.21.15.98:0/4179955541 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe15c0cc7f0 0x7fe15c0bf9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:28.428 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.420+0000 7fe16216b700 1 -- 172.21.15.98:0/4179955541 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe15c0ccc60 msgr2=0x7fe15c154910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:28.429 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.420+0000 7fe16216b700 1 --2- 172.21.15.98:0/4179955541 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe15c0ccc60 0x7fe15c154910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:28.429 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.420+0000 7fe16216b700 1 -- 172.21.15.98:0/4179955541 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1500094f0 con 0x7fe15c0ac000 2023-10-06T05:07:28.429 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.420+0000 7fe16296c700 1 --2- 172.21.15.98:0/4179955541 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe15c0ccc60 0x7fe15c154910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:07:28.429 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.421+0000 7fe16216b700 1 --2- 172.21.15.98:0/4179955541 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe15c0ac000 0x7fe15c14a700 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fe14c00c8b0 tx=0x7fe14c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:28.429 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.421+0000 7fe15a7fc700 1 -- 172.21.15.98:0/4179955541 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe14c011660 con 0x7fe15c0ac000 2023-10-06T05:07:28.429 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.421+0000 7fe15a7fc700 1 -- 172.21.15.98:0/4179955541 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe14c0117c0 con 0x7fe15c0ac000 2023-10-06T05:07:28.429 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.421+0000 7fe15a7fc700 1 -- 172.21.15.98:0/4179955541 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe14c011b40 con 0x7fe15c0ac000 2023-10-06T05:07:28.429 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.421+0000 7fe1643cf700 1 -- 172.21.15.98:0/4179955541 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe15c0c4b90 con 0x7fe15c0ac000 2023-10-06T05:07:28.429 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.421+0000 7fe1643cf700 1 -- 172.21.15.98:0/4179955541 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe15c0bfef0 con 0x7fe15c0ac000 2023-10-06T05:07:28.429 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.422+0000 7fe1643cf700 1 -- 172.21.15.98:0/4179955541 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe15c064a90 con 0x7fe15c0ac000 2023-10-06T05:07:28.429 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.422+0000 7fe15a7fc700 1 -- 172.21.15.98:0/4179955541 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fe14c010350 con 0x7fe15c0ac000 2023-10-06T05:07:28.429 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.422+0000 7fe15a7fc700 1 --2- 172.21.15.98:0/4179955541 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fe148066f30 0x7fe1480693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:28.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.422+0000 7fe15a7fc700 1 -- 172.21.15.98:0/4179955541 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fe14c084910 con 0x7fe15c0ac000 2023-10-06T05:07:28.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.422+0000 7fe16196a700 1 --2- 172.21.15.98:0/4179955541 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fe148066f30 0x7fe1480693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:28.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.423+0000 7fe16196a700 1 --2- 172.21.15.98:0/4179955541 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fe148066f30 0x7fe1480693e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fe15c064f70 tx=0x7fe150017000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:28.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.428+0000 7fe15a7fc700 1 -- 172.21.15.98:0/4179955541 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe14c0551c0 con 0x7fe15c0ac000 2023-10-06T05:07:28.659 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.656+0000 7fe1643cf700 1 -- 172.21.15.98:0/4179955541 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fe15c0c6a00 con 0x7fe15c0ac000 2023-10-06T05:07:28.659 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.656+0000 7fe15a7fc700 1 -- 172.21.15.98:0/4179955541 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+88 (secure 0 0 0) 0x7fe14c01e070 con 0x7fe15c0ac000 2023-10-06T05:07:28.659 INFO:teuthology.orchestra.run.smithi098.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-10-06T05:07:28.661 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.659+0000 7fe1643cf700 1 -- 172.21.15.98:0/4179955541 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fe148066f30 msgr2=0x7fe1480693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:28.661 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.659+0000 7fe1643cf700 1 --2- 172.21.15.98:0/4179955541 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fe148066f30 0x7fe1480693e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fe15c064f70 tx=0x7fe150017000 comp rx=0 tx=0).stop 2023-10-06T05:07:28.661 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.659+0000 7fe1643cf700 1 -- 172.21.15.98:0/4179955541 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe15c0ac000 msgr2=0x7fe15c14a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:28.661 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.659+0000 7fe1643cf700 1 --2- 172.21.15.98:0/4179955541 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe15c0ac000 0x7fe15c14a700 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fe14c00c8b0 tx=0x7fe14c008040 comp rx=0 tx=0).stop 2023-10-06T05:07:28.661 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.659+0000 7fe1643cf700 1 -- 172.21.15.98:0/4179955541 shutdown_connections 2023-10-06T05:07:28.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.659+0000 7fe1643cf700 1 --2- 172.21.15.98:0/4179955541 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe15c0ccc60 0x7fe15c154910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:28.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.659+0000 7fe1643cf700 1 --2- 172.21.15.98:0/4179955541 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fe148066f30 0x7fe1480693e0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:28.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.659+0000 7fe1643cf700 1 --2- 172.21.15.98:0/4179955541 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe15c0ac000 0x7fe15c14a700 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:28.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.659+0000 7fe1643cf700 1 --2- 172.21.15.98:0/4179955541 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe15c0cc7f0 0x7fe15c0bf9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:28.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.659+0000 7fe1643cf700 1 -- 172.21.15.98:0/4179955541 >> 172.21.15.98:0/4179955541 conn(0x7fe15c068080 msgr2=0x7fe15c0c9e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:28.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.659+0000 7fe1643cf700 1 -- 172.21.15.98:0/4179955541 shutdown_connections 2023-10-06T05:07:28.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:28.659+0000 7fe1643cf700 1 -- 172.21.15.98:0/4179955541 wait complete. 2023-10-06T05:07:28.673 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-10-06T05:07:28.673 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-06T05:07:28.673 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi174.front.sepia.ceph.com 2023-10-06T05:07:28.673 INFO:tasks.cephfs.mount:self.client.name = client.3 2023-10-06T05:07:28.673 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2023-10-06T05:07:28.673 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-10-06T05:07:28.673 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-06T05:07:28.673 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-06T05:07:28.673 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2023-10-06T05:07:28.674 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:07:28.674 DEBUG:teuthology.orchestra.run.smithi174:> ip addr 2023-10-06T05:07:28.689 INFO:teuthology.orchestra.run.smithi174.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-06T05:07:28.689 INFO:teuthology.orchestra.run.smithi174.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-06T05:07:28.689 INFO:teuthology.orchestra.run.smithi174.stdout: inet 127.0.0.1/8 scope host lo 2023-10-06T05:07:28.690 INFO:teuthology.orchestra.run.smithi174.stdout: valid_lft forever preferred_lft forever 2023-10-06T05:07:28.690 INFO:teuthology.orchestra.run.smithi174.stdout: inet6 ::1/128 scope host 2023-10-06T05:07:28.690 INFO:teuthology.orchestra.run.smithi174.stdout: valid_lft forever preferred_lft forever 2023-10-06T05:07:28.690 INFO:teuthology.orchestra.run.smithi174.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-06T05:07:28.690 INFO:teuthology.orchestra.run.smithi174.stdout: link/ether 0c:c4:7a:d9:36:e0 brd ff:ff:ff:ff:ff:ff 2023-10-06T05:07:28.690 INFO:teuthology.orchestra.run.smithi174.stdout: altname enp7s0f0 2023-10-06T05:07:28.690 INFO:teuthology.orchestra.run.smithi174.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-06T05:07:28.690 INFO:teuthology.orchestra.run.smithi174.stdout: link/ether 0c:c4:7a:d9:36:e1 brd ff:ff:ff:ff:ff:ff 2023-10-06T05:07:28.690 INFO:teuthology.orchestra.run.smithi174.stdout: altname enp7s0f1 2023-10-06T05:07:28.690 INFO:teuthology.orchestra.run.smithi174.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-06T05:07:28.690 INFO:teuthology.orchestra.run.smithi174.stdout: link/ether 0c:c4:7a:88:77:1a brd ff:ff:ff:ff:ff:ff 2023-10-06T05:07:28.691 INFO:teuthology.orchestra.run.smithi174.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-06T05:07:28.691 INFO:teuthology.orchestra.run.smithi174.stdout: link/ether 0c:c4:7a:88:77:1b brd ff:ff:ff:ff:ff:ff 2023-10-06T05:07:28.691 INFO:teuthology.orchestra.run.smithi174.stdout: inet 172.21.15.174/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-10-06T05:07:28.691 INFO:teuthology.orchestra.run.smithi174.stdout: valid_lft 42476sec preferred_lft 42476sec 2023-10-06T05:07:28.691 INFO:teuthology.orchestra.run.smithi174.stdout: inet6 fe80::ec4:7aff:fe88:771b/64 scope link 2023-10-06T05:07:28.691 INFO:teuthology.orchestra.run.smithi174.stdout: valid_lft forever preferred_lft forever 2023-10-06T05:07:28.691 INFO:teuthology.orchestra.run.smithi174.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-06T05:07:28.691 INFO:teuthology.orchestra.run.smithi174.stdout: link/ether 06:47:fa:11:86:87 brd ff:ff:ff:ff:ff:ff 2023-10-06T05:07:28.691 INFO:teuthology.orchestra.run.smithi174.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-06T05:07:28.691 INFO:teuthology.orchestra.run.smithi174.stdout: valid_lft forever preferred_lft forever 2023-10-06T05:07:28.691 INFO:teuthology.orchestra.run.smithi174.stdout: inet6 fe80::5c6b:37ff:fe09:4124/64 scope link tentative 2023-10-06T05:07:28.691 INFO:teuthology.orchestra.run.smithi174.stdout: valid_lft forever preferred_lft forever 2023-10-06T05:07:28.692 INFO:teuthology.orchestra.run.smithi174.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-06T05:07:28.692 INFO:teuthology.orchestra.run.smithi174.stdout: link/ether 06:47:fa:11:86:87 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2023-10-06T05:07:28.692 INFO:teuthology.orchestra.run.smithi174.stdout: inet6 fe80::447:faff:fe11:8687/64 scope link tentative 2023-10-06T05:07:28.692 INFO:teuthology.orchestra.run.smithi174.stdout: valid_lft forever preferred_lft forever 2023-10-06T05:07:28.692 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:07:28.692 DEBUG:teuthology.orchestra.run.smithi174:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-06T05:07:28.754 INFO:teuthology.orchestra.run.smithi174.stdout:1 2023-10-06T05:07:28.755 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:07:28.756 DEBUG:teuthology.orchestra.run.smithi174:> route 2023-10-06T05:07:28.813 INFO:teuthology.orchestra.run.smithi174.stdout:Kernel IP routing table 2023-10-06T05:07:28.813 INFO:teuthology.orchestra.run.smithi174.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-06T05:07:28.813 INFO:teuthology.orchestra.run.smithi174.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-10-06T05:07:28.813 INFO:teuthology.orchestra.run.smithi174.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-10-06T05:07:28.813 INFO:teuthology.orchestra.run.smithi174.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-06T05:07:28.814 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:07:28.814 DEBUG:teuthology.orchestra.run.smithi174:> (cd / && exec bash -c ' 2023-10-06T05:07:28.814 DEBUG:teuthology.orchestra.run.smithi174:> set -e 2023-10-06T05:07:28.814 DEBUG:teuthology.orchestra.run.smithi174:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-10-06T05:07:28.815 DEBUG:teuthology.orchestra.run.smithi174:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-10-06T05:07:28.815 DEBUG:teuthology.orchestra.run.smithi174:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-10-06T05:07:28.815 DEBUG:teuthology.orchestra.run.smithi174:> ') 2023-10-06T05:07:28.914 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:07:28.914 DEBUG:teuthology.orchestra.run.smithi174:> ip netns list 2023-10-06T05:07:28.929 INFO:teuthology.orchestra.run.smithi174.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-10-06T05:07:28.929 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:07:28.930 DEBUG:teuthology.orchestra.run.smithi174:> ip netns list-id 2023-10-06T05:07:28.983 INFO:teuthology.orchestra.run.smithi174.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2023-10-06T05:07:28.984 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:07:28.984 DEBUG:teuthology.orchestra.run.smithi174:> (cd / && exec bash -c ' 2023-10-06T05:07:28.984 DEBUG:teuthology.orchestra.run.smithi174:> set -e 2023-10-06T05:07:28.984 DEBUG:teuthology.orchestra.run.smithi174:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2023-10-06T05:07:28.984 DEBUG:teuthology.orchestra.run.smithi174:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2023-10-06T05:07:28.984 DEBUG:teuthology.orchestra.run.smithi174:> ') 2023-10-06T05:07:29.072 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:07:29.072 DEBUG:teuthology.orchestra.run.smithi174:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2023-10-06T05:07:29.107 INFO:teuthology.orchestra.run.smithi174.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-06T05:07:29.107 INFO:teuthology.orchestra.run.smithi174.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-06T05:07:29.107 INFO:teuthology.orchestra.run.smithi174.stdout: inet 127.0.0.1/8 scope host lo 2023-10-06T05:07:29.107 INFO:teuthology.orchestra.run.smithi174.stdout: valid_lft forever preferred_lft forever 2023-10-06T05:07:29.107 INFO:teuthology.orchestra.run.smithi174.stdout: inet6 ::1/128 scope host 2023-10-06T05:07:29.107 INFO:teuthology.orchestra.run.smithi174.stdout: valid_lft forever preferred_lft forever 2023-10-06T05:07:29.107 INFO:teuthology.orchestra.run.smithi174.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-06T05:07:29.108 INFO:teuthology.orchestra.run.smithi174.stdout: link/ether 16:bd:9c:d8:94:23 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-10-06T05:07:29.108 INFO:teuthology.orchestra.run.smithi174.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-10-06T05:07:29.108 INFO:teuthology.orchestra.run.smithi174.stdout: valid_lft forever preferred_lft forever 2023-10-06T05:07:29.108 INFO:teuthology.orchestra.run.smithi174.stdout: inet6 fe80::14bd:9cff:fed8:9423/64 scope link tentative 2023-10-06T05:07:29.108 INFO:teuthology.orchestra.run.smithi174.stdout: valid_lft forever preferred_lft forever 2023-10-06T05:07:29.115 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:07:29.116 DEBUG:teuthology.orchestra.run.smithi174:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2023-10-06T05:07:29.192 INFO:teuthology.orchestra.run.smithi174.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-06T05:07:29.192 INFO:teuthology.orchestra.run.smithi174.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-06T05:07:29.192 INFO:teuthology.orchestra.run.smithi174.stdout: inet 127.0.0.1/8 scope host lo 2023-10-06T05:07:29.192 INFO:teuthology.orchestra.run.smithi174.stdout: valid_lft forever preferred_lft forever 2023-10-06T05:07:29.193 INFO:teuthology.orchestra.run.smithi174.stdout: inet6 ::1/128 scope host 2023-10-06T05:07:29.193 INFO:teuthology.orchestra.run.smithi174.stdout: valid_lft forever preferred_lft forever 2023-10-06T05:07:29.193 INFO:teuthology.orchestra.run.smithi174.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-06T05:07:29.193 INFO:teuthology.orchestra.run.smithi174.stdout: link/ether 16:bd:9c:d8:94:23 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-10-06T05:07:29.194 INFO:teuthology.orchestra.run.smithi174.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-10-06T05:07:29.194 INFO:teuthology.orchestra.run.smithi174.stdout: valid_lft forever preferred_lft forever 2023-10-06T05:07:29.198 INFO:teuthology.orchestra.run.smithi174.stdout: inet6 fe80::14bd:9cff:fed8:9423/64 scope link tentative 2023-10-06T05:07:29.198 INFO:teuthology.orchestra.run.smithi174.stdout: valid_lft forever preferred_lft forever 2023-10-06T05:07:29.204 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2023-10-06T05:07:29.205 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:07:29.205 DEBUG:teuthology.orchestra.run.smithi174:> (cd / && exec bash -c ' 2023-10-06T05:07:29.205 DEBUG:teuthology.orchestra.run.smithi174:> set -e 2023-10-06T05:07:29.205 DEBUG:teuthology.orchestra.run.smithi174:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2023-10-06T05:07:29.205 DEBUG:teuthology.orchestra.run.smithi174:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2023-10-06T05:07:29.205 DEBUG:teuthology.orchestra.run.smithi174:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2023-10-06T05:07:29.205 DEBUG:teuthology.orchestra.run.smithi174:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2023-10-06T05:07:29.205 DEBUG:teuthology.orchestra.run.smithi174:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2023-10-06T05:07:29.206 DEBUG:teuthology.orchestra.run.smithi174:> ') 2023-10-06T05:07:29.432 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:07:29.432 DEBUG:teuthology.orchestra.run.smithi174:> (cd / && exec bash -c ' 2023-10-06T05:07:29.432 DEBUG:teuthology.orchestra.run.smithi174:> set -e 2023-10-06T05:07:29.432 DEBUG:teuthology.orchestra.run.smithi174:> sudo ip link set brx.1 up 2023-10-06T05:07:29.432 DEBUG:teuthology.orchestra.run.smithi174:> sudo ip link set dev brx.1 master ceph-brx 2023-10-06T05:07:29.432 DEBUG:teuthology.orchestra.run.smithi174:> ') 2023-10-06T05:07:29.512 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-06T05:07:29.512 DEBUG:teuthology.orchestra.run.smithi174:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2023-10-06T05:07:29.527 INFO:teuthology.orchestra.run.smithi174.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2023-10-06T05:07:29.527 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-06T05:07:29.527 DEBUG:teuthology.orchestra.run.smithi174:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2023-10-06T05:07:29.581 INFO:tasks.cephfs.kernel_mount:mounting using device: 3@.cephfs=/ 2023-10-06T05:07:29.581 INFO:teuthology.orchestra.run:Running command with timeout 1800 2023-10-06T05:07:29.582 DEBUG:teuthology.orchestra.run.smithi174:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 3@.cephfs=/ /home/ubuntu/cephtest/mnt.3 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-06T05:07:29.661 INFO:teuthology.orchestra.run.smithi174.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-06T05:07:29.662 INFO:teuthology.orchestra.run.smithi174.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-06T05:07:29.680 INFO:teuthology.orchestra.run.smithi174.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-06T05:07:29.680 INFO:teuthology.orchestra.run.smithi174.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-06T05:07:29.681 INFO:teuthology.orchestra.run.smithi174.stdout:mount.ceph: resolved to: "172.21.15.98:3300,172.21.15.174:3300,172.21.15.174:3301" 2023-10-06T05:07:29.681 INFO:teuthology.orchestra.run.smithi174.stdout:mount.ceph: trying mount with new device syntax: 3@f50c5df5-cad5-47a4-9bb1-8c791d4311ec.cephfs=/ 2023-10-06T05:07:29.681 INFO:teuthology.orchestra.run.smithi174.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,ms_mode=prefer-crc,key=3,mon_addr=172.21.15.98:3300/172.21.15.174:3300/172.21.15.174:3301" will pass to kernel 2023-10-06T05:07:29.681 INFO:teuthology.orchestra.run.smithi174.stdout:mount.ceph: v2 mount support check returned 2 2023-10-06T05:07:29.681 INFO:teuthology.orchestra.run.smithi174.stdout:mount.ceph: kernel does not support v2 syntax 2023-10-06T05:07:29.681 INFO:teuthology.orchestra.run.smithi174.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-06T05:07:29.702 INFO:teuthology.orchestra.run.smithi174.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-06T05:07:29.703 INFO:teuthology.orchestra.run.smithi174.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-06T05:07:29.703 INFO:teuthology.orchestra.run.smithi174.stdout:mount.ceph: resolved to: "172.21.15.98:3300,172.21.15.174:3300,172.21.15.174:3301" 2023-10-06T05:07:29.703 INFO:teuthology.orchestra.run.smithi174.stdout:mount.ceph: trying mount with new device syntax: 3@f50c5df5-cad5-47a4-9bb1-8c791d4311ec.cephfs=/ 2023-10-06T05:07:29.703 INFO:teuthology.orchestra.run.smithi174.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,ms_mode=prefer-crc,key=3,mon_addr=172.21.15.98:3300/172.21.15.174:3300/172.21.15.174:3301" will pass to kernel 2023-10-06T05:07:29.704 INFO:teuthology.orchestra.run.smithi174.stdout:mount.ceph: v2 mount support check returned 2 2023-10-06T05:07:29.704 INFO:teuthology.orchestra.run.smithi174.stdout:mount.ceph: kernel does not support v2 syntax 2023-10-06T05:07:29.704 INFO:teuthology.orchestra.run.smithi174.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-06T05:07:29.704 INFO:teuthology.orchestra.run.smithi174.stdout:mount.ceph: resolved to: "172.21.15.98:6789,172.21.15.174:6789,172.21.15.174:6790" 2023-10-06T05:07:29.704 INFO:teuthology.orchestra.run.smithi174.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.98:6789,172.21.15.174:6789,172.21.15.174:6790:/ 2023-10-06T05:07:29.705 INFO:teuthology.orchestra.run.smithi174.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,key=3,mds_namespace=cephfs,fsid=f50c5df5-cad5-47a4-9bb1-8c791d4311ec" will pass to kernel 2023-10-06T05:07:29.706 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-06T05:07:29.706 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:07:29.706 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2023-10-06T05:07:29.770 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2023-10-06T05:07:29.792 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T05:07:29.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:29.908+0000 7f840afaa700 1 Processor -- start 2023-10-06T05:07:29.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:29.908+0000 7f840afaa700 1 -- start start 2023-10-06T05:07:29.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:29.909+0000 7f840afaa700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8404159560 0x7f84040aa530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:29.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:29.909+0000 7f840afaa700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f84040b0760 0x7f84040b1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:29.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:29.909+0000 7f840afaa700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84040b0c60 con 0x7f84040b49a0 2023-10-06T05:07:29.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:29.909+0000 7f840afaa700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84040b0f40 con 0x7f84040b0760 2023-10-06T05:07:29.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:29.909+0000 7f840afaa700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84040b1220 con 0x7f8404159560 2023-10-06T05:07:29.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:29.909+0000 7f8409547700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f84040b0760 0x7f84040b1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:29.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:29.909+0000 7f8408d46700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8404159560 0x7f84040aa530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:29.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:29.909+0000 7f8403fff700 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f84040b49a0 0x7f84040aaa70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.98:55016/0 (socket says 172.21.15.98:55016) 2023-10-06T05:07:29.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:29.909+0000 7f8403fff700 1 -- 172.21.15.98:0/1469655442 learned_addr learned my addr 172.21.15.98:0/1469655442 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:07:29.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:29.910+0000 7f8409547700 1 -- 172.21.15.98:0/1469655442 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8404159560 msgr2=0x7f84040aa530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:29.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:29.910+0000 7f8409547700 1 --2- 172.21.15.98:0/1469655442 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8404159560 0x7f84040aa530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:29.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:29.910+0000 7f8409547700 1 -- 172.21.15.98:0/1469655442 >> v1:172.21.15.174:6789/0 conn(0x7f84040b49a0 legacy=0x7f84040aaa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:29.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:29.910+0000 7f8409547700 1 -- 172.21.15.98:0/1469655442 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84040b1500 con 0x7f84040b0760 2023-10-06T05:07:29.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:29.910+0000 7f84037fe700 1 -- 172.21.15.98:0/1469655442 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4128405508 0 0) 0x7f84040b0c60 con 0x7f84040b49a0 2023-10-06T05:07:29.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:29.910+0000 7f8408d46700 1 --2- 172.21.15.98:0/1469655442 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8404159560 0x7f84040aa530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T05:07:29.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:29.910+0000 7f8409547700 1 --2- 172.21.15.98:0/1469655442 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f84040b0760 0x7f84040b1bc0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f83fc0098e0 tx=0x7f83fc0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=11d71e273c9d41df server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:29.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:29.910+0000 7f84037fe700 1 -- 172.21.15.98:0/1469655442 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83fc00b640 con 0x7f84040b0760 2023-10-06T05:07:29.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:29.910+0000 7f84037fe700 1 -- 172.21.15.98:0/1469655442 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83fc00b7a0 con 0x7f84040b0760 2023-10-06T05:07:29.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:29.910+0000 7f84037fe700 1 -- 172.21.15.98:0/1469655442 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83fc00bb40 con 0x7f84040b0760 2023-10-06T05:07:29.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:29.911+0000 7f840afaa700 1 -- 172.21.15.98:0/1469655442 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f84040b0760 msgr2=0x7f84040b1bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:29.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:29.911+0000 7f840afaa700 1 --2- 172.21.15.98:0/1469655442 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f84040b0760 0x7f84040b1bc0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f83fc0098e0 tx=0x7f83fc0091a0 comp rx=0 tx=0).stop 2023-10-06T05:07:29.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:29.911+0000 7f840afaa700 1 -- 172.21.15.98:0/1469655442 shutdown_connections 2023-10-06T05:07:29.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:29.911+0000 7f840afaa700 1 --2- 172.21.15.98:0/1469655442 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f84040b0760 0x7f84040b1bc0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:29.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:29.911+0000 7f840afaa700 1 --2- 172.21.15.98:0/1469655442 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8404159560 0x7f84040aa530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:29.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:29.911+0000 7f840afaa700 1 -- 172.21.15.98:0/1469655442 >> 172.21.15.98:0/1469655442 conn(0x7f84040997f0 msgr2=0x7f840406f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:29.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:29.911+0000 7f840afaa700 1 -- 172.21.15.98:0/1469655442 shutdown_connections 2023-10-06T05:07:29.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:29.911+0000 7f840afaa700 1 -- 172.21.15.98:0/1469655442 wait complete. 2023-10-06T05:07:29.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:29.912+0000 7f840afaa700 1 Processor -- start 2023-10-06T05:07:29.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:29.912+0000 7f840afaa700 1 -- start start 2023-10-06T05:07:29.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:29.913+0000 7f840afaa700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f84040b0760 0x7f8404157090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:29.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:29.913+0000 7f840afaa700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84040b49a0 0x7f840415d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:29.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:29.913+0000 7f840afaa700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8404159560 0x7f84040ce320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:29.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:29.913+0000 7f840afaa700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f840416a310 con 0x7f84040b49a0 2023-10-06T05:07:29.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:29.913+0000 7f840afaa700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8404169700 con 0x7f84040b0760 2023-10-06T05:07:29.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:29.913+0000 7f840afaa700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84041699e0 con 0x7f8404159560 2023-10-06T05:07:29.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:29.913+0000 7f8403fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84040b49a0 0x7f840415d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:29.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:29.913+0000 7f8403fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84040b49a0 0x7f840415d9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45020/0 (socket says 172.21.15.98:45020) 2023-10-06T05:07:29.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:29.913+0000 7f8403fff700 1 -- 172.21.15.98:0/1076486311 learned_addr learned my addr 172.21.15.98:0/1076486311 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:29.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:29.913+0000 7f8408d46700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f84040b0760 0x7f8404157090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:29.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:29.913+0000 7f8409547700 1 --2- 172.21.15.98:0/1076486311 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8404159560 0x7f84040ce320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:29.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:29.913+0000 7f8403fff700 1 -- 172.21.15.98:0/1076486311 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8404159560 msgr2=0x7f84040ce320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:29.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:29.913+0000 7f8403fff700 1 --2- 172.21.15.98:0/1076486311 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8404159560 0x7f84040ce320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:29.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:29.913+0000 7f8403fff700 1 -- 172.21.15.98:0/1076486311 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f84040b0760 msgr2=0x7f8404157090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:29.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:29.913+0000 7f8403fff700 1 --2- 172.21.15.98:0/1076486311 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f84040b0760 0x7f8404157090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:29.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:29.913+0000 7f8403fff700 1 -- 172.21.15.98:0/1076486311 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83f4009a20 con 0x7f84040b49a0 2023-10-06T05:07:29.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:29.913+0000 7f8409547700 1 --2- 172.21.15.98:0/1076486311 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8404159560 0x7f84040ce320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T05:07:29.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:29.913+0000 7f8408d46700 1 --2- 172.21.15.98:0/1076486311 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f84040b0760 0x7f8404157090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T05:07:29.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:29.913+0000 7f8403fff700 1 --2- 172.21.15.98:0/1076486311 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84040b49a0 0x7f840415d9a0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f83f800c810 tx=0x7f83f8007db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:29.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:29.913+0000 7f8400ff9700 1 -- 172.21.15.98:0/1076486311 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83f8005c00 con 0x7f84040b49a0 2023-10-06T05:07:29.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:29.913+0000 7f8400ff9700 1 -- 172.21.15.98:0/1076486311 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83f8005d60 con 0x7f84040b49a0 2023-10-06T05:07:29.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:29.913+0000 7f840afaa700 1 -- 172.21.15.98:0/1076486311 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83fc009420 con 0x7f84040b49a0 2023-10-06T05:07:29.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:29.913+0000 7f8400ff9700 1 -- 172.21.15.98:0/1076486311 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83f8007300 con 0x7f84040b49a0 2023-10-06T05:07:29.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:29.913+0000 7f840afaa700 1 -- 172.21.15.98:0/1076486311 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f840416a020 con 0x7f84040b49a0 2023-10-06T05:07:29.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:29.914+0000 7f840afaa700 1 -- 172.21.15.98:0/1076486311 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f840416a480 con 0x7f84040b49a0 2023-10-06T05:07:29.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:29.915+0000 7f8400ff9700 1 -- 172.21.15.98:0/1076486311 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f83f801b3b0 con 0x7f84040b49a0 2023-10-06T05:07:29.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:29.915+0000 7f8400ff9700 1 --2- 172.21.15.98:0/1076486311 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f83f0066f30 0x7f83f00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:29.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:29.915+0000 7f8408d46700 1 --2- 172.21.15.98:0/1076486311 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f83f0066f30 0x7f83f00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:29.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:29.915+0000 7f8400ff9700 1 -- 172.21.15.98:0/1076486311 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f83f8083e40 con 0x7f84040b49a0 2023-10-06T05:07:29.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:29.916+0000 7f8408d46700 1 --2- 172.21.15.98:0/1076486311 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f83f0066f30 0x7f83f00693e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f83f400eaa0 tx=0x7f83f400b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:29.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:29.919+0000 7f8400ff9700 1 -- 172.21.15.98:0/1076486311 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f83f8050c60 con 0x7f84040b49a0 2023-10-06T05:07:30.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.102+0000 7f840afaa700 1 -- 172.21.15.98:0/1076486311 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8404157630 con 0x7f84040b49a0 2023-10-06T05:07:30.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.103+0000 7f8400ff9700 1 -- 172.21.15.98:0/1076486311 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+18707 (secure 0 0 0) 0x7f83f8017020 con 0x7f84040b49a0 2023-10-06T05:07:30.104 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:07:30.104 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":29,"fsid":"f50c5df5-cad5-47a4-9bb1-8c791d4311ec","created":"2023-10-06T05:06:32.971175+0000","modified":"2023-10-06T05:07:02.131672+0000","last_up_change":"2023-10-06T05:06:44.016489+0000","last_in_change":"2023-10-06T05:06:35.544511+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-06T05:06:44.559671+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-06T05:06:51.432161+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-10-06T05:06:55.589057+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-10-06T05:06:56.041687+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-10-06T05:06:59.754099+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d9495d95-da75-4c3b-8927-e9bae6a8b093","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":90627},{"type":"v1","addr":"172.21.15.98:6801","nonce":90627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":90627},{"type":"v1","addr":"172.21.15.98:6803","nonce":90627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":90627},{"type":"v1","addr":"172.21.15.98:6807","nonce":90627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":90627},{"type":"v1","addr":"172.21.15.98:6805","nonce":90627}]},"public_addr":"172.21.15.98:6801/90627","cluster_addr":"172.21.15.98:6803/90627","heartbeat_back_addr":"172.21.15.98:6807/90627","heartbeat_front_addr":"172.21.15.98:6805/90627","state":["exists","up"]},{"osd":1,"uuid":"f25ba5be-d65e-4707-a134-154092deb754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":90628},{"type":"v1","addr":"172.21.15.98:6809","nonce":90628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":90628},{"type":"v1","addr":"172.21.15.98:6811","nonce":90628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":90628},{"type":"v1","addr":"172.21.15.98:6815","nonce":90628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":90628},{"type":"v1","addr":"172.21.15.98:6813","nonce":90628}]},"public_addr":"172.21.15.98:6809/90628","cluster_addr":"172.21.15.98:6811/90628","heartbeat_back_addr":"172.21.15.98:6815/90628","heartbeat_front_addr":"172.21.15.98:6813/90628","state":["exists","up"]},{"osd":2,"uuid":"5ab9e028-945c-43e2-8c95-3b0b4cf38e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":90662},{"type":"v1","addr":"172.21.15.98:6818","nonce":90662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":90662},{"type":"v1","addr":"172.21.15.98:6822","nonce":90662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":90662},{"type":"v1","addr":"172.21.15.98:6830","nonce":90662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":90662},{"type":"v1","addr":"172.21.15.98:6826","nonce":90662}]},"public_addr":"172.21.15.98:6818/90662","cluster_addr":"172.21.15.98:6822/90662","heartbeat_back_addr":"172.21.15.98:6830/90662","heartbeat_front_addr":"172.21.15.98:6826/90662","state":["exists","up"]},{"osd":3,"uuid":"3bb803ff-1738-46bf-be62-2267bf378822","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6817","nonce":90663},{"type":"v1","addr":"172.21.15.98:6819","nonce":90663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6821","nonce":90663},{"type":"v1","addr":"172.21.15.98:6823","nonce":90663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6829","nonce":90663},{"type":"v1","addr":"172.21.15.98:6831","nonce":90663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6825","nonce":90663},{"type":"v1","addr":"172.21.15.98:6827","nonce":90663}]},"public_addr":"172.21.15.98:6819/90663","cluster_addr":"172.21.15.98:6823/90663","heartbeat_back_addr":"172.21.15.98:6831/90663","heartbeat_front_addr":"172.21.15.98:6827/90663","state":["exists","up"]},{"osd":4,"uuid":"7d57a4c9-6462-43f6-b64a-7f9e9b4353d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":90847},{"type":"v1","addr":"172.21.15.174:6801","nonce":90847}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":90847},{"type":"v1","addr":"172.21.15.174:6803","nonce":90847}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":90847},{"type":"v1","addr":"172.21.15.174:6807","nonce":90847}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":90847},{"type":"v1","addr":"172.21.15.174:6805","nonce":90847}]},"public_addr":"172.21.15.174:6801/90847","cluster_addr":"172.21.15.174:6803/90847","heartbeat_back_addr":"172.21.15.174:6807/90847","heartbeat_front_addr":"172.21.15.174:6805/90847","state":["exists","up"]},{"osd":5,"uuid":"83287ea4-e44d-4af9-bc77-36d47c60d4a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":90848},{"type":"v1","addr":"172.21.15.174:6809","nonce":90848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":90848},{"type":"v1","addr":"172.21.15.174:6811","nonce":90848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":90848},{"type":"v1","addr":"172.21.15.174:6815","nonce":90848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":90848},{"type":"v1","addr":"172.21.15.174:6813","nonce":90848}]},"public_addr":"172.21.15.174:6809/90848","cluster_addr":"172.21.15.174:6811/90848","heartbeat_back_addr":"172.21.15.174:6815/90848","heartbeat_front_addr":"172.21.15.174:6813/90848","state":["exists","up"]},{"osd":6,"uuid":"496a6272-892e-43c4-a145-206623c22f99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":90877},{"type":"v1","addr":"172.21.15.174:6817","nonce":90877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":90877},{"type":"v1","addr":"172.21.15.174:6819","nonce":90877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":90877},{"type":"v1","addr":"172.21.15.174:6823","nonce":90877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":90877},{"type":"v1","addr":"172.21.15.174:6821","nonce":90877}]},"public_addr":"172.21.15.174:6817/90877","cluster_addr":"172.21.15.174:6819/90877","heartbeat_back_addr":"172.21.15.174:6823/90877","heartbeat_front_addr":"172.21.15.174:6821/90877","state":["exists","up"]},{"osd":7,"uuid":"49967e34-8b07-4bab-a13e-0c16d6a0f3a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":90878},{"type":"v1","addr":"172.21.15.174:6825","nonce":90878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":90878},{"type":"v1","addr":"172.21.15.174:6827","nonce":90878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":90878},{"type":"v1","addr":"172.21.15.174:6831","nonce":90878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":90878},{"type":"v1","addr":"172.21.15.174:6829","nonce":90878}]},"public_addr":"172.21.15.174:6825/90878","cluster_addr":"172.21.15.174:6827/90878","heartbeat_back_addr":"172.21.15.174:6831/90878","heartbeat_front_addr":"172.21.15.174:6829/90878","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.446479+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.514266+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.794108+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.810894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.731919+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.717756+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.737435+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.508682+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T05:07:30.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.105+0000 7f840afaa700 1 -- 172.21.15.98:0/1076486311 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f83f0066f30 msgr2=0x7f83f00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:30.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.105+0000 7f840afaa700 1 --2- 172.21.15.98:0/1076486311 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f83f0066f30 0x7f83f00693e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f83f400eaa0 tx=0x7f83f400b040 comp rx=0 tx=0).stop 2023-10-06T05:07:30.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.105+0000 7f840afaa700 1 -- 172.21.15.98:0/1076486311 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84040b49a0 msgr2=0x7f840415d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:30.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.105+0000 7f840afaa700 1 --2- 172.21.15.98:0/1076486311 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84040b49a0 0x7f840415d9a0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f83f800c810 tx=0x7f83f8007db0 comp rx=0 tx=0).stop 2023-10-06T05:07:30.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.105+0000 7f840afaa700 1 -- 172.21.15.98:0/1076486311 shutdown_connections 2023-10-06T05:07:30.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.105+0000 7f840afaa700 1 --2- 172.21.15.98:0/1076486311 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8404159560 0x7f84040ce320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:30.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.105+0000 7f840afaa700 1 --2- 172.21.15.98:0/1076486311 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f84040b0760 0x7f8404157090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:30.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.105+0000 7f840afaa700 1 --2- 172.21.15.98:0/1076486311 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f83f0066f30 0x7f83f00693e0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:30.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.105+0000 7f840afaa700 1 --2- 172.21.15.98:0/1076486311 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84040b49a0 0x7f840415d9a0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:30.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.105+0000 7f840afaa700 1 -- 172.21.15.98:0/1076486311 >> 172.21.15.98:0/1076486311 conn(0x7f84040997f0 msgr2=0x7f8404158a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:30.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.105+0000 7f840afaa700 1 -- 172.21.15.98:0/1076486311 shutdown_connections 2023-10-06T05:07:30.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.105+0000 7f840afaa700 1 -- 172.21.15.98:0/1076486311 wait complete. 2023-10-06T05:07:30.118 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-06T05:06:44.559671+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-10-06T05:06:51.432161+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-06T05:06:55.589057+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-10-06T05:06:56.041687+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-06T05:06:59.754099+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '29', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-06T05:07:30.118 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-06T05:07:30.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.234+0000 7fbe3809b700 1 Processor -- start 2023-10-06T05:07:30.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.234+0000 7fbe3809b700 1 -- start start 2023-10-06T05:07:30.238 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.235+0000 7fbe3809b700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbe300cbc70 0x7fbe300ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:30.238 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.235+0000 7fbe3809b700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe300cf5d0 con 0x7fbe301594d0 2023-10-06T05:07:30.238 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.235+0000 7fbe3809b700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe300cf8b0 con 0x7fbe301598a0 2023-10-06T05:07:30.238 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.235+0000 7fbe3809b700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe300cfb90 con 0x7fbe300cbc70 2023-10-06T05:07:30.238 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.235+0000 7fbe35e37700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fbe301594d0 0x7fbe300cb560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:57444/0 (socket says 172.21.15.98:57444) 2023-10-06T05:07:30.238 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.235+0000 7fbe35e37700 1 -- 172.21.15.98:0/3748110613 learned_addr learned my addr 172.21.15.98:0/3748110613 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:07:30.238 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.235+0000 7fbe36638700 1 --2- 172.21.15.98:0/3748110613 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbe300cbc70 0x7fbe300ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:30.238 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.236+0000 7fbe34e35700 1 -- 172.21.15.98:0/3748110613 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2777255879 0 0) 0x7fbe300cf5d0 con 0x7fbe301594d0 2023-10-06T05:07:30.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.236+0000 7fbe34e35700 1 -- 172.21.15.98:0/3748110613 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe1c003570 con 0x7fbe301594d0 2023-10-06T05:07:30.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.236+0000 7fbe34e35700 1 -- 172.21.15.98:0/3748110613 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1512747469 0 0) 0x7fbe300cf8b0 con 0x7fbe301598a0 2023-10-06T05:07:30.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.236+0000 7fbe34e35700 1 -- 172.21.15.98:0/3748110613 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe300cf5d0 con 0x7fbe301598a0 2023-10-06T05:07:30.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.236+0000 7fbe36638700 1 -- 172.21.15.98:0/3748110613 >> v1:172.21.15.174:6789/0 conn(0x7fbe301598a0 legacy=0x7fbe300ce7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:30.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.236+0000 7fbe34e35700 1 -- 172.21.15.98:0/3748110613 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 511115794 0 0) 0x7fbe1c003570 con 0x7fbe301594d0 2023-10-06T05:07:30.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.236+0000 7fbe36638700 1 -- 172.21.15.98:0/3748110613 >> v1:172.21.15.98:6789/0 conn(0x7fbe301594d0 legacy=0x7fbe300cb560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:30.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.236+0000 7fbe36638700 1 -- 172.21.15.98:0/3748110613 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe300cfe70 con 0x7fbe300cbc70 2023-10-06T05:07:30.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.236+0000 7fbe36638700 1 --2- 172.21.15.98:0/3748110613 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbe300cbc70 0x7fbe300ceef0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fbe28009910 tx=0x7fbe280091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c259cd8a22322b9 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:30.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.236+0000 7fbe34e35700 1 -- 172.21.15.98:0/3748110613 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe2c0017f0 con 0x7fbe300cbc70 2023-10-06T05:07:30.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.236+0000 7fbe34e35700 1 -- 172.21.15.98:0/3748110613 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe28011420 con 0x7fbe300cbc70 2023-10-06T05:07:30.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.236+0000 7fbe34e35700 1 -- 172.21.15.98:0/3748110613 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe280117c0 con 0x7fbe300cbc70 2023-10-06T05:07:30.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.237+0000 7fbe3809b700 1 -- 172.21.15.98:0/3748110613 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbe300cbc70 msgr2=0x7fbe300ceef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:30.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.237+0000 7fbe3809b700 1 --2- 172.21.15.98:0/3748110613 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbe300cbc70 0x7fbe300ceef0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fbe28009910 tx=0x7fbe280091a0 comp rx=0 tx=0).stop 2023-10-06T05:07:30.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.237+0000 7fbe3809b700 1 -- 172.21.15.98:0/3748110613 shutdown_connections 2023-10-06T05:07:30.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.237+0000 7fbe3809b700 1 --2- 172.21.15.98:0/3748110613 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbe300cbc70 0x7fbe300ceef0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:30.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.237+0000 7fbe3809b700 1 -- 172.21.15.98:0/3748110613 >> 172.21.15.98:0/3748110613 conn(0x7fbe30089e50 msgr2=0x7fbe30099770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:30.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.237+0000 7fbe3809b700 1 -- 172.21.15.98:0/3748110613 shutdown_connections 2023-10-06T05:07:30.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.237+0000 7fbe3809b700 1 -- 172.21.15.98:0/3748110613 wait complete. 2023-10-06T05:07:30.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.238+0000 7fbe3809b700 1 Processor -- start 2023-10-06T05:07:30.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.238+0000 7fbe3809b700 1 -- start start 2023-10-06T05:07:30.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.238+0000 7fbe3809b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe300cbc70 0x7fbe3015b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:30.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.238+0000 7fbe3809b700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbe301594d0 0x7fbe3014f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:30.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.238+0000 7fbe3809b700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbe301598a0 0x7fbe300b5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:30.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.238+0000 7fbe3809b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe300b33e0 con 0x7fbe300cbc70 2023-10-06T05:07:30.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.238+0000 7fbe3809b700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe3014e240 con 0x7fbe301598a0 2023-10-06T05:07:30.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.238+0000 7fbe3809b700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe3014e4c0 con 0x7fbe301594d0 2023-10-06T05:07:30.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.239+0000 7fbe35e37700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe300cbc70 0x7fbe3015b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:30.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.239+0000 7fbe35e37700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe300cbc70 0x7fbe3015b230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45030/0 (socket says 172.21.15.98:45030) 2023-10-06T05:07:30.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.239+0000 7fbe35e37700 1 -- 172.21.15.98:0/2296199565 learned_addr learned my addr 172.21.15.98:0/2296199565 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:30.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.239+0000 7fbe35636700 1 --2- 172.21.15.98:0/2296199565 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbe301594d0 0x7fbe3014f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:30.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.239+0000 7fbe36638700 1 --2- 172.21.15.98:0/2296199565 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbe301598a0 0x7fbe300b5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:30.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.239+0000 7fbe35e37700 1 -- 172.21.15.98:0/2296199565 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbe301594d0 msgr2=0x7fbe3014f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:30.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.239+0000 7fbe35e37700 1 --2- 172.21.15.98:0/2296199565 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbe301594d0 0x7fbe3014f200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:30.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.239+0000 7fbe35e37700 1 -- 172.21.15.98:0/2296199565 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbe301598a0 msgr2=0x7fbe300b5110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:30.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.239+0000 7fbe35e37700 1 --2- 172.21.15.98:0/2296199565 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbe301598a0 0x7fbe300b5110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:30.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.239+0000 7fbe35e37700 1 -- 172.21.15.98:0/2296199565 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe28009420 con 0x7fbe300cbc70 2023-10-06T05:07:30.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.239+0000 7fbe36638700 1 --2- 172.21.15.98:0/2296199565 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbe301598a0 0x7fbe300b5110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:07:30.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.239+0000 7fbe35e37700 1 --2- 172.21.15.98:0/2296199565 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe300cbc70 0x7fbe3015b230 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fbe2c00ccd0 tx=0x7fbe2c007da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:30.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.239+0000 7fbe25ffb700 1 -- 172.21.15.98:0/2296199565 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe2c004e90 con 0x7fbe300cbc70 2023-10-06T05:07:30.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.239+0000 7fbe25ffb700 1 -- 172.21.15.98:0/2296199565 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe2c00a660 con 0x7fbe300cbc70 2023-10-06T05:07:30.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.239+0000 7fbe3809b700 1 -- 172.21.15.98:0/2296199565 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe3014e7a0 con 0x7fbe300cbc70 2023-10-06T05:07:30.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.239+0000 7fbe25ffb700 1 -- 172.21.15.98:0/2296199565 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe2c00aa00 con 0x7fbe300cbc70 2023-10-06T05:07:30.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.239+0000 7fbe3809b700 1 -- 172.21.15.98:0/2296199565 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe3014ea80 con 0x7fbe300cbc70 2023-10-06T05:07:30.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.240+0000 7fbe3809b700 1 -- 172.21.15.98:0/2296199565 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe30064a90 con 0x7fbe300cbc70 2023-10-06T05:07:30.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.240+0000 7fbe25ffb700 1 -- 172.21.15.98:0/2296199565 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fbe2c007150 con 0x7fbe300cbc70 2023-10-06T05:07:30.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.241+0000 7fbe25ffb700 1 --2- 172.21.15.98:0/2296199565 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fbe14066f30 0x7fbe140693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:30.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.241+0000 7fbe25ffb700 1 -- 172.21.15.98:0/2296199565 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fbe2c084430 con 0x7fbe300cbc70 2023-10-06T05:07:30.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.241+0000 7fbe35636700 1 --2- 172.21.15.98:0/2296199565 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fbe14066f30 0x7fbe140693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:30.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.241+0000 7fbe35636700 1 --2- 172.21.15.98:0/2296199565 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fbe14066f30 0x7fbe140693e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fbe300cdca0 tx=0x7fbe20007d50 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:30.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.245+0000 7fbe25ffb700 1 -- 172.21.15.98:0/2296199565 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbe2c0512d0 con 0x7fbe300cbc70 2023-10-06T05:07:30.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.429+0000 7fbe3809b700 1 -- 172.21.15.98:0/2296199565 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbe300c66a0 con 0x7fbe300cbc70 2023-10-06T05:07:30.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.430+0000 7fbe25ffb700 1 -- 172.21.15.98:0/2296199565 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v29) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbe2c0548f0 con 0x7fbe300cbc70 2023-10-06T05:07:30.431 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2023-10-06T05:07:30.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.432+0000 7fbe3809b700 1 -- 172.21.15.98:0/2296199565 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fbe14066f30 msgr2=0x7fbe140693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:30.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.432+0000 7fbe3809b700 1 --2- 172.21.15.98:0/2296199565 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fbe14066f30 0x7fbe140693e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fbe300cdca0 tx=0x7fbe20007d50 comp rx=0 tx=0).stop 2023-10-06T05:07:30.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.432+0000 7fbe3809b700 1 -- 172.21.15.98:0/2296199565 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe300cbc70 msgr2=0x7fbe3015b230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:30.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.432+0000 7fbe3809b700 1 --2- 172.21.15.98:0/2296199565 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe300cbc70 0x7fbe3015b230 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fbe2c00ccd0 tx=0x7fbe2c007da0 comp rx=0 tx=0).stop 2023-10-06T05:07:30.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.432+0000 7fbe3809b700 1 -- 172.21.15.98:0/2296199565 shutdown_connections 2023-10-06T05:07:30.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.432+0000 7fbe3809b700 1 --2- 172.21.15.98:0/2296199565 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbe301598a0 0x7fbe300b5110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:30.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.432+0000 7fbe3809b700 1 --2- 172.21.15.98:0/2296199565 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fbe14066f30 0x7fbe140693e0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:30.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.432+0000 7fbe3809b700 1 --2- 172.21.15.98:0/2296199565 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe300cbc70 0x7fbe3015b230 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:30.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.432+0000 7fbe3809b700 1 --2- 172.21.15.98:0/2296199565 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbe301594d0 0x7fbe3014f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:30.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.432+0000 7fbe3809b700 1 -- 172.21.15.98:0/2296199565 >> 172.21.15.98:0/2296199565 conn(0x7fbe30089e50 msgr2=0x7fbe3014b110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:30.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.432+0000 7fbe3809b700 1 -- 172.21.15.98:0/2296199565 shutdown_connections 2023-10-06T05:07:30.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.432+0000 7fbe3809b700 1 -- 172.21.15.98:0/2296199565 wait complete. 2023-10-06T05:07:30.445 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-10-06T05:07:30.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.561+0000 7fa44cdc1700 1 Processor -- start 2023-10-06T05:07:30.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.561+0000 7fa44cdc1700 1 -- start start 2023-10-06T05:07:30.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.562+0000 7fa44cdc1700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa44806d110 0x7fa4480b1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:30.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.562+0000 7fa44cdc1700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4480b1cd0 con 0x7fa4480686c0 2023-10-06T05:07:30.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.562+0000 7fa44cdc1700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa448069890 con 0x7fa44806d4e0 2023-10-06T05:07:30.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.562+0000 7fa44cdc1700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa448069d20 con 0x7fa44806d110 2023-10-06T05:07:30.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.562+0000 7fa446ffd700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fa4480686c0 0x7fa44815cf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:57460/0 (socket says 172.21.15.98:57460) 2023-10-06T05:07:30.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.562+0000 7fa446ffd700 1 -- 172.21.15.98:0/744780629 learned_addr learned my addr 172.21.15.98:0/744780629 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:07:30.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.562+0000 7fa4477fe700 1 --2- 172.21.15.98:0/744780629 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa44806d110 0x7fa4480b1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:30.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.562+0000 7fa4467fc700 1 -- 172.21.15.98:0/744780629 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1943836286 0 0) 0x7fa43c001110 con 0x7fa4480686c0 2023-10-06T05:07:30.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.562+0000 7fa4467fc700 1 -- 172.21.15.98:0/744780629 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa430003570 con 0x7fa4480686c0 2023-10-06T05:07:30.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.562+0000 7fa4467fc700 1 -- 172.21.15.98:0/744780629 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 316494283 0 0) 0x7fa430003570 con 0x7fa4480686c0 2023-10-06T05:07:30.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.563+0000 7fa4467fc700 1 -- 172.21.15.98:0/744780629 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa44806d110 msgr2=0x7fa4480b1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:30.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.563+0000 7fa4467fc700 1 --2- 172.21.15.98:0/744780629 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa44806d110 0x7fa4480b1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:30.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.563+0000 7fa4467fc700 1 -- 172.21.15.98:0/744780629 >> v1:172.21.15.174:6789/0 conn(0x7fa44806d4e0 legacy=0x7fa44815d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:30.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.563+0000 7fa4467fc700 1 -- 172.21.15.98:0/744780629 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa44815dd30 con 0x7fa4480686c0 2023-10-06T05:07:30.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.563+0000 7fa4467fc700 1 -- 172.21.15.98:0/744780629 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1543991876 0 0) 0x7fa43c001600 con 0x7fa4480686c0 2023-10-06T05:07:30.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.563+0000 7fa4467fc700 1 -- 172.21.15.98:0/744780629 >> v1:172.21.15.98:6789/0 conn(0x7fa4480686c0 legacy=0x7fa44815cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:30.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.563+0000 7fa4467fc700 1 --2- 172.21.15.98:0/744780629 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa44806d110 0x7fa430003a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:30.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.563+0000 7fa4467fc700 1 --2- 172.21.15.98:0/744780629 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa430003fd0 0x7fa430004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:30.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.563+0000 7fa4467fc700 1 --2- 172.21.15.98:0/744780629 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa430004d50 0x7fa4300051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:30.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.563+0000 7fa4467fc700 1 -- 172.21.15.98:0/744780629 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa448069d20 con 0x7fa430003fd0 2023-10-06T05:07:30.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.563+0000 7fa4467fc700 1 -- 172.21.15.98:0/744780629 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa43000bb40 con 0x7fa430004d50 2023-10-06T05:07:30.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.563+0000 7fa4467fc700 1 -- 172.21.15.98:0/744780629 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa43000bd00 con 0x7fa44806d110 2023-10-06T05:07:30.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.563+0000 7fa446ffd700 1 --2- 172.21.15.98:0/744780629 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa430003fd0 0x7fa430004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:30.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.563+0000 7fa4477fe700 1 --2- 172.21.15.98:0/744780629 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa44806d110 0x7fa430003a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:30.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.563+0000 7fa447fff700 1 --2- 172.21.15.98:0/744780629 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa430004d50 0x7fa4300051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:30.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.563+0000 7fa446ffd700 1 -- 172.21.15.98:0/744780629 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa44806d110 msgr2=0x7fa430003a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:30.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.563+0000 7fa446ffd700 1 --2- 172.21.15.98:0/744780629 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa44806d110 0x7fa430003a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:30.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.563+0000 7fa446ffd700 1 -- 172.21.15.98:0/744780629 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa430004d50 msgr2=0x7fa4300051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:30.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.563+0000 7fa446ffd700 1 --2- 172.21.15.98:0/744780629 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa430004d50 0x7fa4300051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:30.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.563+0000 7fa446ffd700 1 -- 172.21.15.98:0/744780629 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa430003570 con 0x7fa430003fd0 2023-10-06T05:07:30.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.563+0000 7fa446ffd700 1 --2- 172.21.15.98:0/744780629 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa430003fd0 0x7fa430004810 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fa43c00c5f0 tx=0x7fa43c007a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df2cb3bde2e9a455 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:30.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.563+0000 7fa4467fc700 1 -- 172.21.15.98:0/744780629 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa43c00cdf0 con 0x7fa430003fd0 2023-10-06T05:07:30.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.563+0000 7fa4467fc700 1 -- 172.21.15.98:0/744780629 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa43c005920 con 0x7fa430003fd0 2023-10-06T05:07:30.573 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.563+0000 7fa4467fc700 1 -- 172.21.15.98:0/744780629 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa43c005ce0 con 0x7fa430003fd0 2023-10-06T05:07:30.573 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.564+0000 7fa44cdc1700 1 -- 172.21.15.98:0/744780629 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa430003fd0 msgr2=0x7fa430004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:30.573 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.564+0000 7fa44cdc1700 1 --2- 172.21.15.98:0/744780629 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa430003fd0 0x7fa430004810 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fa43c00c5f0 tx=0x7fa43c007a50 comp rx=0 tx=0).stop 2023-10-06T05:07:30.573 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.564+0000 7fa447fff700 1 -- 172.21.15.98:0/744780629 reap_dead start 2023-10-06T05:07:30.573 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.564+0000 7fa44cdc1700 1 -- 172.21.15.98:0/744780629 shutdown_connections 2023-10-06T05:07:30.573 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.564+0000 7fa44cdc1700 1 -- 172.21.15.98:0/744780629 >> 172.21.15.98:0/744780629 conn(0x7fa4480675b0 msgr2=0x7fa4480679b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:30.573 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.564+0000 7fa44cdc1700 1 -- 172.21.15.98:0/744780629 shutdown_connections 2023-10-06T05:07:30.573 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.564+0000 7fa44cdc1700 1 -- 172.21.15.98:0/744780629 wait complete. 2023-10-06T05:07:30.573 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.565+0000 7fa44cdc1700 1 Processor -- start 2023-10-06T05:07:30.573 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.565+0000 7fa44cdc1700 1 -- start start 2023-10-06T05:07:30.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.566+0000 7fa44cdc1700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa430003fd0 0x7fa44815bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:30.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.566+0000 7fa44cdc1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa44815a770 0x7fa4480cb860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:30.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.566+0000 7fa44cdc1700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa4480c9210 0x7fa4480cbda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:30.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.566+0000 7fa44cdc1700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa44815b8d0 con 0x7fa44815a770 2023-10-06T05:07:30.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.566+0000 7fa44cdc1700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa44815acc0 con 0x7fa430003fd0 2023-10-06T05:07:30.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.566+0000 7fa44cdc1700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa44815afa0 con 0x7fa4480c9210 2023-10-06T05:07:30.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.566+0000 7fa446ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa44815a770 0x7fa4480cb860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:30.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.566+0000 7fa446ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa44815a770 0x7fa4480cb860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45044/0 (socket says 172.21.15.98:45044) 2023-10-06T05:07:30.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.566+0000 7fa4477fe700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa430003fd0 0x7fa44815bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:30.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.566+0000 7fa446ffd700 1 -- 172.21.15.98:0/3479064160 learned_addr learned my addr 172.21.15.98:0/3479064160 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:30.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.566+0000 7fa447fff700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa4480c9210 0x7fa4480cbda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:30.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.566+0000 7fa446ffd700 1 -- 172.21.15.98:0/3479064160 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa4480c9210 msgr2=0x7fa4480cbda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:30.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.566+0000 7fa446ffd700 1 --2- 172.21.15.98:0/3479064160 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa4480c9210 0x7fa4480cbda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:30.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.566+0000 7fa446ffd700 1 -- 172.21.15.98:0/3479064160 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa430003fd0 msgr2=0x7fa44815bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:30.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.566+0000 7fa446ffd700 1 --2- 172.21.15.98:0/3479064160 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa430003fd0 0x7fa44815bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:30.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.566+0000 7fa446ffd700 1 -- 172.21.15.98:0/3479064160 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa44815b280 con 0x7fa44815a770 2023-10-06T05:07:30.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.566+0000 7fa446ffd700 1 --2- 172.21.15.98:0/3479064160 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa44815a770 0x7fa4480cb860 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fa43c00cbe0 tx=0x7fa43c015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:30.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.566+0000 7fa42f7fe700 1 -- 172.21.15.98:0/3479064160 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa43c005a90 con 0x7fa44815a770 2023-10-06T05:07:30.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.566+0000 7fa42f7fe700 1 -- 172.21.15.98:0/3479064160 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa43c005bf0 con 0x7fa44815a770 2023-10-06T05:07:30.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.566+0000 7fa44cdc1700 1 -- 172.21.15.98:0/3479064160 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa44815b8d0 con 0x7fa44815a770 2023-10-06T05:07:30.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.567+0000 7fa42f7fe700 1 -- 172.21.15.98:0/3479064160 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa43c0072e0 con 0x7fa44815a770 2023-10-06T05:07:30.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.567+0000 7fa44cdc1700 1 -- 172.21.15.98:0/3479064160 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa448152360 con 0x7fa44815a770 2023-10-06T05:07:30.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.567+0000 7fa44cdc1700 1 -- 172.21.15.98:0/3479064160 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa44815ba60 con 0x7fa44815a770 2023-10-06T05:07:30.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.568+0000 7fa42f7fe700 1 -- 172.21.15.98:0/3479064160 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fa43c019070 con 0x7fa44815a770 2023-10-06T05:07:30.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.568+0000 7fa42f7fe700 1 --2- 172.21.15.98:0/3479064160 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fa434066f30 0x7fa4340693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:30.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.568+0000 7fa4477fe700 1 --2- 172.21.15.98:0/3479064160 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fa434066f30 0x7fa4340693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:30.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.568+0000 7fa42f7fe700 1 -- 172.21.15.98:0/3479064160 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fa43c0859f0 con 0x7fa44815a770 2023-10-06T05:07:30.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.569+0000 7fa4477fe700 1 --2- 172.21.15.98:0/3479064160 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fa434066f30 0x7fa4340693e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fa438006f00 tx=0x7fa4380091a0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:30.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.573+0000 7fa42f7fe700 1 -- 172.21.15.98:0/3479064160 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa43c052810 con 0x7fa44815a770 2023-10-06T05:07:30.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.755+0000 7fa44cdc1700 1 -- 172.21.15.98:0/3479064160 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fa4480b1cd0 con 0x7fa44815a770 2023-10-06T05:07:30.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.755+0000 7fa42f7fe700 1 -- 172.21.15.98:0/3479064160 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v29) v1 ==== 92+0+10 (secure 0 0 0) 0x7fa4480b1cd0 con 0x7fa44815a770 2023-10-06T05:07:30.756 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 8 2023-10-06T05:07:30.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.757+0000 7fa44cdc1700 1 -- 172.21.15.98:0/3479064160 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fa434066f30 msgr2=0x7fa4340693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:30.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.757+0000 7fa44cdc1700 1 --2- 172.21.15.98:0/3479064160 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fa434066f30 0x7fa4340693e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fa438006f00 tx=0x7fa4380091a0 comp rx=0 tx=0).stop 2023-10-06T05:07:30.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.757+0000 7fa44cdc1700 1 -- 172.21.15.98:0/3479064160 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa44815a770 msgr2=0x7fa4480cb860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:30.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.757+0000 7fa44cdc1700 1 --2- 172.21.15.98:0/3479064160 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa44815a770 0x7fa4480cb860 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fa43c00cbe0 tx=0x7fa43c015000 comp rx=0 tx=0).stop 2023-10-06T05:07:30.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.757+0000 7fa44cdc1700 1 -- 172.21.15.98:0/3479064160 shutdown_connections 2023-10-06T05:07:30.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.757+0000 7fa44cdc1700 1 --2- 172.21.15.98:0/3479064160 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa4480c9210 0x7fa4480cbda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:30.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.757+0000 7fa44cdc1700 1 --2- 172.21.15.98:0/3479064160 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa430003fd0 0x7fa44815bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:30.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.757+0000 7fa44cdc1700 1 --2- 172.21.15.98:0/3479064160 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fa434066f30 0x7fa4340693e0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:30.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.757+0000 7fa44cdc1700 1 --2- 172.21.15.98:0/3479064160 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa44815a770 0x7fa4480cb860 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:30.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.757+0000 7fa44cdc1700 1 -- 172.21.15.98:0/3479064160 >> 172.21.15.98:0/3479064160 conn(0x7fa4480675b0 msgr2=0x7fa4480679b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:30.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.758+0000 7fa44cdc1700 1 -- 172.21.15.98:0/3479064160 shutdown_connections 2023-10-06T05:07:30.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.758+0000 7fa44cdc1700 1 -- 172.21.15.98:0/3479064160 wait complete. 2023-10-06T05:07:30.770 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-06T05:07:30.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.886+0000 7f202b39e700 1 Processor -- start 2023-10-06T05:07:30.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.886+0000 7f202b39e700 1 -- start start 2023-10-06T05:07:30.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.887+0000 7f202b39e700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f20240b0520 0x7f20240cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:30.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.887+0000 7f202b39e700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f20240c4140 0x7f20240c8a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:30.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.887+0000 7f202b39e700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20240cdd30 con 0x7f20240ccaa0 2023-10-06T05:07:30.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.887+0000 7f202b39e700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20240ccf40 con 0x7f20240c4140 2023-10-06T05:07:30.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.887+0000 7f202b39e700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20240cd120 con 0x7f20240b0520 2023-10-06T05:07:30.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.887+0000 7f202ab9d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f20240c4140 0x7f20240c8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:30.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.888+0000 7f202ab9d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f20240c4140 0x7f20240c8a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45046/0 (socket says 172.21.15.98:45046) 2023-10-06T05:07:30.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.888+0000 7f202ab9d700 1 -- 172.21.15.98:0/1077954189 learned_addr learned my addr 172.21.15.98:0/1077954189 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:30.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.888+0000 7f202a39c700 1 --2- 172.21.15.98:0/1077954189 >> v2:172.21.15.174:3300/0 conn(0x7f20240b0520 0x7f20240cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:30.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.888+0000 7f202ab9d700 1 -- 172.21.15.98:0/1077954189 >> v2:172.21.15.174:3300/0 conn(0x7f20240b0520 msgr2=0x7f20240cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:30.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.888+0000 7f202ab9d700 1 --2- 172.21.15.98:0/1077954189 >> v2:172.21.15.174:3300/0 conn(0x7f20240b0520 0x7f20240cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:30.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.888+0000 7f202ab9d700 1 -- 172.21.15.98:0/1077954189 >> v1:172.21.15.174:6789/0 conn(0x7f20240ccaa0 legacy=0x7f20240c3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:30.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.888+0000 7f202ab9d700 1 -- 172.21.15.98:0/1077954189 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20240cd400 con 0x7f20240c4140 2023-10-06T05:07:30.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.888+0000 7f202a39c700 1 --2- 172.21.15.98:0/1077954189 >> v2:172.21.15.174:3300/0 conn(0x7f20240b0520 0x7f20240cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:07:30.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.888+0000 7f202ab9d700 1 --2- 172.21.15.98:0/1077954189 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20240c4140 0x7f20240c8a10 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f202406f910 tx=0x7f2020009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f279d87a7113775 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:30.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.888+0000 7f202939a700 1 -- 172.21.15.98:0/1077954189 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f202000b860 con 0x7f20240c4140 2023-10-06T05:07:30.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.888+0000 7f202939a700 1 -- 172.21.15.98:0/1077954189 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f202000b9c0 con 0x7f20240c4140 2023-10-06T05:07:30.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.888+0000 7f202939a700 1 -- 172.21.15.98:0/1077954189 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f202000bd60 con 0x7f20240c4140 2023-10-06T05:07:30.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.889+0000 7f202b39e700 1 -- 172.21.15.98:0/1077954189 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20240c4140 msgr2=0x7f20240c8a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:30.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.889+0000 7f202b39e700 1 --2- 172.21.15.98:0/1077954189 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20240c4140 0x7f20240c8a10 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f202406f910 tx=0x7f2020009250 comp rx=0 tx=0).stop 2023-10-06T05:07:30.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.889+0000 7f202b39e700 1 -- 172.21.15.98:0/1077954189 shutdown_connections 2023-10-06T05:07:30.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.889+0000 7f202b39e700 1 --2- 172.21.15.98:0/1077954189 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20240c4140 0x7f20240c8a10 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:30.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.889+0000 7f202b39e700 1 --2- 172.21.15.98:0/1077954189 >> v2:172.21.15.174:3300/0 conn(0x7f20240b0520 0x7f20240cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:30.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.889+0000 7f202b39e700 1 -- 172.21.15.98:0/1077954189 >> 172.21.15.98:0/1077954189 conn(0x7f2024096a30 msgr2=0x7f2024096e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:30.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.889+0000 7f202b39e700 1 -- 172.21.15.98:0/1077954189 shutdown_connections 2023-10-06T05:07:30.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.889+0000 7f202b39e700 1 -- 172.21.15.98:0/1077954189 wait complete. 2023-10-06T05:07:30.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.890+0000 7f202b39e700 1 Processor -- start 2023-10-06T05:07:30.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.890+0000 7f202b39e700 1 -- start start 2023-10-06T05:07:30.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.891+0000 7f202b39e700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f20240b0520 0x7f2024151fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:30.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.891+0000 7f202b39e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20240c4140 0x7f20240b2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:30.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.891+0000 7f202b39e700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f20240ccaa0 0x7f20240c99e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:30.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.891+0000 7f202b39e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20240b3540 con 0x7f20240c4140 2023-10-06T05:07:30.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.891+0000 7f202b39e700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f202415a630 con 0x7f20240b0520 2023-10-06T05:07:30.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.891+0000 7f202b39e700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2024159d00 con 0x7f20240ccaa0 2023-10-06T05:07:30.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.892+0000 7f2029b9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20240c4140 0x7f20240b2bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:30.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.892+0000 7f202a39c700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f20240b0520 0x7f2024151fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:30.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.892+0000 7f2029b9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20240c4140 0x7f20240b2bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45062/0 (socket says 172.21.15.98:45062) 2023-10-06T05:07:30.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.892+0000 7f2029b9b700 1 -- 172.21.15.98:0/1529773372 learned_addr learned my addr 172.21.15.98:0/1529773372 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:30.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.892+0000 7f202a39c700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f20240b0520 0x7f2024151fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.98:49234/0 (socket says 172.21.15.98:49234) 2023-10-06T05:07:30.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.892+0000 7f202ab9d700 1 --2- 172.21.15.98:0/1529773372 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f20240ccaa0 0x7f20240c99e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:30.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.892+0000 7f2029b9b700 1 -- 172.21.15.98:0/1529773372 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f20240ccaa0 msgr2=0x7f20240c99e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:30.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.892+0000 7f2029b9b700 1 --2- 172.21.15.98:0/1529773372 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f20240ccaa0 0x7f20240c99e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:30.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.892+0000 7f2029b9b700 1 -- 172.21.15.98:0/1529773372 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f20240b0520 msgr2=0x7f2024151fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:30.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.892+0000 7f2029b9b700 1 --2- 172.21.15.98:0/1529773372 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f20240b0520 0x7f2024151fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:30.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.892+0000 7f2029b9b700 1 -- 172.21.15.98:0/1529773372 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20200094f0 con 0x7f20240c4140 2023-10-06T05:07:30.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.892+0000 7f2029b9b700 1 --2- 172.21.15.98:0/1529773372 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20240c4140 0x7f20240b2bd0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f201400c7c0 tx=0x7f2014007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:30.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.892+0000 7f201a7fc700 1 -- 172.21.15.98:0/1529773372 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2014011590 con 0x7f20240c4140 2023-10-06T05:07:30.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.892+0000 7f201a7fc700 1 -- 172.21.15.98:0/1529773372 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20140116f0 con 0x7f20240c4140 2023-10-06T05:07:30.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.892+0000 7f201a7fc700 1 -- 172.21.15.98:0/1529773372 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2014011ab0 con 0x7f20240c4140 2023-10-06T05:07:30.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.892+0000 7f202b39e700 1 -- 172.21.15.98:0/1529773372 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2024159fe0 con 0x7f20240c4140 2023-10-06T05:07:30.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.892+0000 7f202b39e700 1 -- 172.21.15.98:0/1529773372 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20240b3110 con 0x7f20240c4140 2023-10-06T05:07:30.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.893+0000 7f202b39e700 1 -- 172.21.15.98:0/1529773372 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f202405f230 con 0x7f20240c4140 2023-10-06T05:07:30.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.894+0000 7f201a7fc700 1 -- 172.21.15.98:0/1529773372 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f201401b3b0 con 0x7f20240c4140 2023-10-06T05:07:30.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.894+0000 7f201a7fc700 1 --2- 172.21.15.98:0/1529773372 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f2008066f30 0x7f20080693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:30.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.894+0000 7f201a7fc700 1 -- 172.21.15.98:0/1529773372 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f2014083d40 con 0x7f20240c4140 2023-10-06T05:07:30.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.894+0000 7f202a39c700 1 --2- 172.21.15.98:0/1529773372 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f2008066f30 0x7f20080693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:30.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.894+0000 7f202a39c700 1 --2- 172.21.15.98:0/1529773372 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f2008066f30 0x7f20080693e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f20240ca600 tx=0x7f201c009930 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:30.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:30.898+0000 7f201a7fc700 1 -- 172.21.15.98:0/1529773372 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2014050c10 con 0x7f20240c4140 2023-10-06T05:07:31.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.081+0000 7f202b39e700 1 -- 172.21.15.98:0/1529773372 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f202415a770 con 0x7f20240c4140 2023-10-06T05:07:31.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.081+0000 7f201a7fc700 1 -- 172.21.15.98:0/1529773372 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v29) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2014017020 con 0x7f20240c4140 2023-10-06T05:07:31.083 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2023-10-06T05:07:31.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.083+0000 7f202b39e700 1 -- 172.21.15.98:0/1529773372 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f2008066f30 msgr2=0x7f20080693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:31.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.083+0000 7f202b39e700 1 --2- 172.21.15.98:0/1529773372 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f2008066f30 0x7f20080693e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f20240ca600 tx=0x7f201c009930 comp rx=0 tx=0).stop 2023-10-06T05:07:31.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.083+0000 7f202b39e700 1 -- 172.21.15.98:0/1529773372 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20240c4140 msgr2=0x7f20240b2bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:31.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.083+0000 7f202b39e700 1 --2- 172.21.15.98:0/1529773372 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20240c4140 0x7f20240b2bd0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f201400c7c0 tx=0x7f2014007f70 comp rx=0 tx=0).stop 2023-10-06T05:07:31.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.083+0000 7f202b39e700 1 -- 172.21.15.98:0/1529773372 shutdown_connections 2023-10-06T05:07:31.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.083+0000 7f202b39e700 1 --2- 172.21.15.98:0/1529773372 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f20240ccaa0 0x7f20240c99e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:31.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.083+0000 7f202b39e700 1 --2- 172.21.15.98:0/1529773372 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f20240b0520 0x7f2024151fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:31.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.083+0000 7f202b39e700 1 --2- 172.21.15.98:0/1529773372 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f2008066f30 0x7f20080693e0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:31.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.083+0000 7f202b39e700 1 --2- 172.21.15.98:0/1529773372 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20240c4140 0x7f20240b2bd0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:31.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.083+0000 7f202b39e700 1 -- 172.21.15.98:0/1529773372 >> 172.21.15.98:0/1529773372 conn(0x7f2024096a30 msgr2=0x7f2024068c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:31.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.083+0000 7f202b39e700 1 -- 172.21.15.98:0/1529773372 shutdown_connections 2023-10-06T05:07:31.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.083+0000 7f202b39e700 1 -- 172.21.15.98:0/1529773372 wait complete. 2023-10-06T05:07:31.096 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-06T05:07:31.213 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.212+0000 7fcbc87a7700 1 Processor -- start 2023-10-06T05:07:31.213 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.212+0000 7fcbc87a7700 1 -- start start 2023-10-06T05:07:31.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.213+0000 7fcbc87a7700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fcbc01598a0 0x7fcbc00ce7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:31.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.213+0000 7fcbc87a7700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcbc00cbc70 0x7fcbc00ced20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:31.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.213+0000 7fcbc87a7700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbc00cf5d0 con 0x7fcbc01594d0 2023-10-06T05:07:31.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.213+0000 7fcbc87a7700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbc00cf8b0 con 0x7fcbc01598a0 2023-10-06T05:07:31.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.213+0000 7fcbc87a7700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc00cfb90 con 0x7fcbc00cbc70 2023-10-06T05:07:31.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.213+0000 7fcbc5d42700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fcbc01598a0 0x7fcbc00ce7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:31.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.213+0000 7fcbc5d42700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fcbc01598a0 0x7fcbc00ce7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45072/0 (socket says 172.21.15.98:45072) 2023-10-06T05:07:31.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.213+0000 7fcbc5d42700 1 -- 172.21.15.98:0/3328795680 learned_addr learned my addr 172.21.15.98:0/3328795680 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:31.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.214+0000 7fcbc6d44700 1 --2- 172.21.15.98:0/3328795680 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcbc00cbc70 0x7fcbc00ced20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:31.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.214+0000 7fcbc5d42700 1 -- 172.21.15.98:0/3328795680 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcbc00cbc70 msgr2=0x7fcbc00ced20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:31.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.214+0000 7fcbc5d42700 1 --2- 172.21.15.98:0/3328795680 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcbc00cbc70 0x7fcbc00ced20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:31.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.214+0000 7fcbc5d42700 1 -- 172.21.15.98:0/3328795680 >> v1:172.21.15.174:6789/0 conn(0x7fcbc01594d0 legacy=0x7fcbc00cb560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:31.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.214+0000 7fcbc5d42700 1 -- 172.21.15.98:0/3328795680 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbc00cfe70 con 0x7fcbc01598a0 2023-10-06T05:07:31.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.214+0000 7fcbc5d42700 1 --2- 172.21.15.98:0/3328795680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcbc01598a0 0x7fcbc00ce7e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fcbb40099b0 tx=0x7fcbb4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4b08c3192c9a7ef server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:31.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.214+0000 7fcbc5541700 1 -- 172.21.15.98:0/3328795680 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbb400b860 con 0x7fcbc01598a0 2023-10-06T05:07:31.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.214+0000 7fcbc5541700 1 -- 172.21.15.98:0/3328795680 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbb400b9c0 con 0x7fcbc01598a0 2023-10-06T05:07:31.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.214+0000 7fcbc5541700 1 -- 172.21.15.98:0/3328795680 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbb400bd60 con 0x7fcbc01598a0 2023-10-06T05:07:31.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.215+0000 7fcbc87a7700 1 -- 172.21.15.98:0/3328795680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcbc01598a0 msgr2=0x7fcbc00ce7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:31.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.215+0000 7fcbc87a7700 1 --2- 172.21.15.98:0/3328795680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcbc01598a0 0x7fcbc00ce7e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fcbb40099b0 tx=0x7fcbb4009250 comp rx=0 tx=0).stop 2023-10-06T05:07:31.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.215+0000 7fcbc87a7700 1 -- 172.21.15.98:0/3328795680 shutdown_connections 2023-10-06T05:07:31.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.215+0000 7fcbc87a7700 1 --2- 172.21.15.98:0/3328795680 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcbc00cbc70 0x7fcbc00ced20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:31.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.215+0000 7fcbc87a7700 1 --2- 172.21.15.98:0/3328795680 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcbc01598a0 0x7fcbc00ce7e0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:31.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.215+0000 7fcbc87a7700 1 -- 172.21.15.98:0/3328795680 >> 172.21.15.98:0/3328795680 conn(0x7fcbc0089e50 msgr2=0x7fcbc0099770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:31.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.215+0000 7fcbc87a7700 1 -- 172.21.15.98:0/3328795680 shutdown_connections 2023-10-06T05:07:31.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.215+0000 7fcbc87a7700 1 -- 172.21.15.98:0/3328795680 wait complete. 2023-10-06T05:07:31.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.216+0000 7fcbc87a7700 1 Processor -- start 2023-10-06T05:07:31.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.216+0000 7fcbc87a7700 1 -- start start 2023-10-06T05:07:31.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.216+0000 7fcbc87a7700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcbc00cbc70 0x7fcbc015b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:31.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.216+0000 7fcbc87a7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcbc01594d0 0x7fcbc0169a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:31.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.217+0000 7fcbc87a7700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcbc01598a0 0x7fcbc0153540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:31.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.217+0000 7fcbc87a7700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc015b670 con 0x7fcbc01594d0 2023-10-06T05:07:31.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.217+0000 7fcbc87a7700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc014ffb0 con 0x7fcbc00cbc70 2023-10-06T05:07:31.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.217+0000 7fcbc87a7700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc015aca0 con 0x7fcbc01598a0 2023-10-06T05:07:31.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.217+0000 7fcbc5d42700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcbc01594d0 0x7fcbc0169a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:31.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.217+0000 7fcbc5d42700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcbc01594d0 0x7fcbc0169a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45088/0 (socket says 172.21.15.98:45088) 2023-10-06T05:07:31.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.217+0000 7fcbc5d42700 1 -- 172.21.15.98:0/1566289141 learned_addr learned my addr 172.21.15.98:0/1566289141 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:31.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.217+0000 7fcbc6543700 1 --2- 172.21.15.98:0/1566289141 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcbc00cbc70 0x7fcbc015b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:31.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.217+0000 7fcbc6d44700 1 --2- 172.21.15.98:0/1566289141 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcbc01598a0 0x7fcbc0153540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:31.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.217+0000 7fcbc5d42700 1 -- 172.21.15.98:0/1566289141 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcbc01598a0 msgr2=0x7fcbc0153540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:31.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.217+0000 7fcbc5d42700 1 --2- 172.21.15.98:0/1566289141 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcbc01598a0 0x7fcbc0153540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:31.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.217+0000 7fcbc5d42700 1 -- 172.21.15.98:0/1566289141 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcbc00cbc70 msgr2=0x7fcbc015b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:31.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.217+0000 7fcbc5d42700 1 --2- 172.21.15.98:0/1566289141 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcbc00cbc70 0x7fcbc015b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:31.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.217+0000 7fcbc5d42700 1 -- 172.21.15.98:0/1566289141 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbb40094f0 con 0x7fcbc01594d0 2023-10-06T05:07:31.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.217+0000 7fcbc5d42700 1 --2- 172.21.15.98:0/1566289141 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcbc01594d0 0x7fcbc0169a50 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fcbc0067c20 tx=0x7fcbb4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:31.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.217+0000 7fcbbe7fc700 1 -- 172.21.15.98:0/1566289141 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbb4011e00 con 0x7fcbc01594d0 2023-10-06T05:07:31.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.217+0000 7fcbbe7fc700 1 -- 172.21.15.98:0/1566289141 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbb4024440 con 0x7fcbc01594d0 2023-10-06T05:07:31.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.217+0000 7fcbc87a7700 1 -- 172.21.15.98:0/1566289141 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbc015a970 con 0x7fcbc01594d0 2023-10-06T05:07:31.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.217+0000 7fcbbe7fc700 1 -- 172.21.15.98:0/1566289141 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbb4024800 con 0x7fcbc01594d0 2023-10-06T05:07:31.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.217+0000 7fcbc87a7700 1 -- 172.21.15.98:0/1566289141 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbc015c500 con 0x7fcbc01594d0 2023-10-06T05:07:31.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.218+0000 7fcbc87a7700 1 -- 172.21.15.98:0/1566289141 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbc0064a90 con 0x7fcbc01594d0 2023-10-06T05:07:31.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.219+0000 7fcbbe7fc700 1 -- 172.21.15.98:0/1566289141 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fcbb401a420 con 0x7fcbc01594d0 2023-10-06T05:07:31.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.219+0000 7fcbbe7fc700 1 --2- 172.21.15.98:0/1566289141 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fcbac066f30 0x7fcbac0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:31.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.219+0000 7fcbbe7fc700 1 -- 172.21.15.98:0/1566289141 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fcbb4086760 con 0x7fcbc01594d0 2023-10-06T05:07:31.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.219+0000 7fcbc6543700 1 --2- 172.21.15.98:0/1566289141 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fcbac066f30 0x7fcbac0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:31.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.219+0000 7fcbc6543700 1 --2- 172.21.15.98:0/1566289141 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fcbac066f30 0x7fcbac0693e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fcbb0007fd0 tx=0x7fcbb00016f0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:31.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.223+0000 7fcbbe7fc700 1 -- 172.21.15.98:0/1566289141 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcbb4053580 con 0x7fcbc01594d0 2023-10-06T05:07:31.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.406+0000 7fcbc87a7700 1 -- 172.21.15.98:0/1566289141 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fcbc00cf260 con 0x7fcbc01594d0 2023-10-06T05:07:31.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.407+0000 7fcbbe7fc700 1 -- 172.21.15.98:0/1566289141 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v29) v1 ==== 100+0+11 (secure 0 0 0) 0x7fcbb4056ba0 con 0x7fcbc01594d0 2023-10-06T05:07:31.408 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2023-10-06T05:07:31.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.409+0000 7fcbc87a7700 1 -- 172.21.15.98:0/1566289141 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fcbac066f30 msgr2=0x7fcbac0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:31.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.409+0000 7fcbc87a7700 1 --2- 172.21.15.98:0/1566289141 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fcbac066f30 0x7fcbac0693e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fcbb0007fd0 tx=0x7fcbb00016f0 comp rx=0 tx=0).stop 2023-10-06T05:07:31.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.409+0000 7fcbc87a7700 1 -- 172.21.15.98:0/1566289141 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcbc01594d0 msgr2=0x7fcbc0169a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:31.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.409+0000 7fcbc87a7700 1 --2- 172.21.15.98:0/1566289141 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcbc01594d0 0x7fcbc0169a50 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fcbc0067c20 tx=0x7fcbb4017000 comp rx=0 tx=0).stop 2023-10-06T05:07:31.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.409+0000 7fcbc87a7700 1 -- 172.21.15.98:0/1566289141 shutdown_connections 2023-10-06T05:07:31.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.409+0000 7fcbc87a7700 1 --2- 172.21.15.98:0/1566289141 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcbc01598a0 0x7fcbc0153540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:31.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.409+0000 7fcbc87a7700 1 --2- 172.21.15.98:0/1566289141 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcbc00cbc70 0x7fcbc015b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:31.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.409+0000 7fcbc87a7700 1 --2- 172.21.15.98:0/1566289141 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fcbac066f30 0x7fcbac0693e0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:31.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.409+0000 7fcbc87a7700 1 --2- 172.21.15.98:0/1566289141 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcbc01594d0 0x7fcbc0169a50 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:31.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.409+0000 7fcbc87a7700 1 -- 172.21.15.98:0/1566289141 >> 172.21.15.98:0/1566289141 conn(0x7fcbc0089e50 msgr2=0x7fcbc0099770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:31.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.409+0000 7fcbc87a7700 1 -- 172.21.15.98:0/1566289141 shutdown_connections 2023-10-06T05:07:31.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.409+0000 7fcbc87a7700 1 -- 172.21.15.98:0/1566289141 wait complete. 2023-10-06T05:07:31.421 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-06T05:07:31.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.537+0000 7ff342c6f700 1 Processor -- start 2023-10-06T05:07:31.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.537+0000 7ff342c6f700 1 -- start start 2023-10-06T05:07:31.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.538+0000 7ff342c6f700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff33c159470 0x7ff33c0ce980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:31.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.538+0000 7ff342c6f700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff33c159840 0x7ff33c0ceec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:31.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.538+0000 7ff342c6f700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff33c0cbb80 con 0x7ff33c0c6170 2023-10-06T05:07:31.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.538+0000 7ff342c6f700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff33c0c6d00 con 0x7ff33c159840 2023-10-06T05:07:31.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.538+0000 7ff342c6f700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff33c0c6f00 con 0x7ff33c159470 2023-10-06T05:07:31.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.538+0000 7ff333fff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff33c159840 0x7ff33c0ceec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:31.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.538+0000 7ff333fff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff33c159840 0x7ff33c0ceec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45092/0 (socket says 172.21.15.98:45092) 2023-10-06T05:07:31.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.538+0000 7ff333fff700 1 -- 172.21.15.98:0/4273123395 learned_addr learned my addr 172.21.15.98:0/4273123395 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:31.541 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.538+0000 7ff340a0b700 1 --2- 172.21.15.98:0/4273123395 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff33c159470 0x7ff33c0ce980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:31.542 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.539+0000 7ff333fff700 1 -- 172.21.15.98:0/4273123395 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff33c159470 msgr2=0x7ff33c0ce980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:31.542 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.539+0000 7ff333fff700 1 --2- 172.21.15.98:0/4273123395 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff33c159470 0x7ff33c0ce980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:31.542 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.539+0000 7ff333fff700 1 -- 172.21.15.98:0/4273123395 >> v1:172.21.15.174:6789/0 conn(0x7ff33c0c6170 legacy=0x7ff33c0c65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:31.542 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.539+0000 7ff333fff700 1 -- 172.21.15.98:0/4273123395 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff33c0cb530 con 0x7ff33c159840 2023-10-06T05:07:31.542 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.539+0000 7ff333fff700 1 --2- 172.21.15.98:0/4273123395 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff33c159840 0x7ff33c0ceec0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7ff32c0099b0 tx=0x7ff32c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be08cd8b6fbc9cf1 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:31.542 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.539+0000 7ff3337fe700 1 -- 172.21.15.98:0/4273123395 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff32c009ea0 con 0x7ff33c159840 2023-10-06T05:07:31.542 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.539+0000 7ff3337fe700 1 -- 172.21.15.98:0/4273123395 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff32c00b860 con 0x7ff33c159840 2023-10-06T05:07:31.542 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.539+0000 7ff3337fe700 1 -- 172.21.15.98:0/4273123395 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff32c00bc00 con 0x7ff33c159840 2023-10-06T05:07:31.543 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.539+0000 7ff342c6f700 1 -- 172.21.15.98:0/4273123395 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff33c159840 msgr2=0x7ff33c0ceec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:31.543 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.539+0000 7ff342c6f700 1 --2- 172.21.15.98:0/4273123395 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff33c159840 0x7ff33c0ceec0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7ff32c0099b0 tx=0x7ff32c009250 comp rx=0 tx=0).stop 2023-10-06T05:07:31.543 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.539+0000 7ff342c6f700 1 -- 172.21.15.98:0/4273123395 shutdown_connections 2023-10-06T05:07:31.543 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.540+0000 7ff342c6f700 1 --2- 172.21.15.98:0/4273123395 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff33c159470 0x7ff33c0ce980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:31.543 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.540+0000 7ff342c6f700 1 --2- 172.21.15.98:0/4273123395 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff33c159840 0x7ff33c0ceec0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:31.543 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.540+0000 7ff342c6f700 1 -- 172.21.15.98:0/4273123395 >> 172.21.15.98:0/4273123395 conn(0x7ff33c0996e0 msgr2=0x7ff33c06f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:31.543 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.540+0000 7ff342c6f700 1 -- 172.21.15.98:0/4273123395 shutdown_connections 2023-10-06T05:07:31.544 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.540+0000 7ff342c6f700 1 -- 172.21.15.98:0/4273123395 wait complete. 2023-10-06T05:07:31.546 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.541+0000 7ff342c6f700 1 Processor -- start 2023-10-06T05:07:31.546 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.541+0000 7ff342c6f700 1 -- start start 2023-10-06T05:07:31.546 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.541+0000 7ff342c6f700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff33c0c6170 0x7ff33c155160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:31.546 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.541+0000 7ff342c6f700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff33c159470 0x7ff33c1436e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:31.546 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.541+0000 7ff342c6f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff33c159840 0x7ff33c159c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:31.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.541+0000 7ff342c6f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff33c15a710 con 0x7ff33c159840 2023-10-06T05:07:31.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.541+0000 7ff342c6f700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff33c15a9f0 con 0x7ff33c0c6170 2023-10-06T05:07:31.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.541+0000 7ff342c6f700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff33c15acd0 con 0x7ff33c159470 2023-10-06T05:07:31.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.542+0000 7ff34120c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff33c159840 0x7ff33c159c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:31.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.542+0000 7ff340a0b700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff33c0c6170 0x7ff33c155160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:31.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.542+0000 7ff333fff700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff33c159470 0x7ff33c1436e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:31.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.542+0000 7ff34120c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff33c159840 0x7ff33c159c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45108/0 (socket says 172.21.15.98:45108) 2023-10-06T05:07:31.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.542+0000 7ff340a0b700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff33c0c6170 0x7ff33c155160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.98:49260/0 (socket says 172.21.15.98:49260) 2023-10-06T05:07:31.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.542+0000 7ff34120c700 1 -- 172.21.15.98:0/840295907 learned_addr learned my addr 172.21.15.98:0/840295907 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:31.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.542+0000 7ff34120c700 1 -- 172.21.15.98:0/840295907 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff33c159470 msgr2=0x7ff33c1436e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:31.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.542+0000 7ff34120c700 1 --2- 172.21.15.98:0/840295907 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff33c159470 0x7ff33c1436e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:31.547 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.542+0000 7ff34120c700 1 -- 172.21.15.98:0/840295907 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff33c0c6170 msgr2=0x7ff33c155160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:31.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.542+0000 7ff34120c700 1 --2- 172.21.15.98:0/840295907 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff33c0c6170 0x7ff33c155160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:31.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.542+0000 7ff34120c700 1 -- 172.21.15.98:0/840295907 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff32c0094f0 con 0x7ff33c159840 2023-10-06T05:07:31.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.542+0000 7ff333fff700 1 --2- 172.21.15.98:0/840295907 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff33c159470 0x7ff33c1436e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T05:07:31.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.542+0000 7ff340a0b700 1 --2- 172.21.15.98:0/840295907 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff33c0c6170 0x7ff33c155160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T05:07:31.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.542+0000 7ff34120c700 1 --2- 172.21.15.98:0/840295907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff33c159840 0x7ff33c159c50 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7ff33400c4c0 tx=0x7ff334007f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:31.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.542+0000 7ff330ff9700 1 -- 172.21.15.98:0/840295907 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff33400ce70 con 0x7ff33c159840 2023-10-06T05:07:31.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.542+0000 7ff330ff9700 1 -- 172.21.15.98:0/840295907 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff334011590 con 0x7ff33c159840 2023-10-06T05:07:31.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.542+0000 7ff330ff9700 1 -- 172.21.15.98:0/840295907 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff334011970 con 0x7ff33c159840 2023-10-06T05:07:31.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.542+0000 7ff342c6f700 1 -- 172.21.15.98:0/840295907 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff33c15b960 con 0x7ff33c159840 2023-10-06T05:07:31.548 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.542+0000 7ff342c6f700 1 -- 172.21.15.98:0/840295907 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff33c158680 con 0x7ff33c159840 2023-10-06T05:07:31.549 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.543+0000 7ff342c6f700 1 -- 172.21.15.98:0/840295907 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff33c064a90 con 0x7ff33c159840 2023-10-06T05:07:31.549 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.543+0000 7ff330ff9700 1 -- 172.21.15.98:0/840295907 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7ff334011ce0 con 0x7ff33c159840 2023-10-06T05:07:31.549 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.544+0000 7ff330ff9700 1 --2- 172.21.15.98:0/840295907 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7ff328066f30 0x7ff3280693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:31.549 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.544+0000 7ff330ff9700 1 -- 172.21.15.98:0/840295907 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7ff334083c60 con 0x7ff33c159840 2023-10-06T05:07:31.549 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.544+0000 7ff340a0b700 1 --2- 172.21.15.98:0/840295907 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7ff328066f30 0x7ff3280693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:31.549 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.544+0000 7ff340a0b700 1 --2- 172.21.15.98:0/840295907 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7ff328066f30 0x7ff3280693e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ff33800e9a0 tx=0x7ff33800b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:31.549 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.548+0000 7ff330ff9700 1 -- 172.21.15.98:0/840295907 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff334050a80 con 0x7ff33c159840 2023-10-06T05:07:31.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.729+0000 7ff342c6f700 1 -- 172.21.15.98:0/840295907 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7ff33c0cb810 con 0x7ff33c159840 2023-10-06T05:07:31.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.729+0000 7ff330ff9700 1 -- 172.21.15.98:0/840295907 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v29) v1 ==== 103+0+11 (secure 0 0 0) 0x7ff334017070 con 0x7ff33c159840 2023-10-06T05:07:31.730 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2023-10-06T05:07:31.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.731+0000 7ff342c6f700 1 -- 172.21.15.98:0/840295907 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7ff328066f30 msgr2=0x7ff3280693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:31.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.731+0000 7ff342c6f700 1 --2- 172.21.15.98:0/840295907 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7ff328066f30 0x7ff3280693e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ff33800e9a0 tx=0x7ff33800b040 comp rx=0 tx=0).stop 2023-10-06T05:07:31.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.731+0000 7ff342c6f700 1 -- 172.21.15.98:0/840295907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff33c159840 msgr2=0x7ff33c159c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:31.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.731+0000 7ff342c6f700 1 --2- 172.21.15.98:0/840295907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff33c159840 0x7ff33c159c50 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7ff33400c4c0 tx=0x7ff334007f50 comp rx=0 tx=0).stop 2023-10-06T05:07:31.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.731+0000 7ff342c6f700 1 -- 172.21.15.98:0/840295907 shutdown_connections 2023-10-06T05:07:31.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.731+0000 7ff342c6f700 1 --2- 172.21.15.98:0/840295907 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7ff328066f30 0x7ff3280693e0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:31.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.731+0000 7ff342c6f700 1 --2- 172.21.15.98:0/840295907 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff33c159840 0x7ff33c159c50 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:31.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.731+0000 7ff342c6f700 1 --2- 172.21.15.98:0/840295907 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff33c0c6170 0x7ff33c155160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:31.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.731+0000 7ff342c6f700 1 --2- 172.21.15.98:0/840295907 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff33c159470 0x7ff33c1436e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:31.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.731+0000 7ff342c6f700 1 -- 172.21.15.98:0/840295907 >> 172.21.15.98:0/840295907 conn(0x7ff33c0996e0 msgr2=0x7ff33c0cbf10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:31.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.731+0000 7ff342c6f700 1 -- 172.21.15.98:0/840295907 shutdown_connections 2023-10-06T05:07:31.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.731+0000 7ff342c6f700 1 -- 172.21.15.98:0/840295907 wait complete. 2023-10-06T05:07:31.744 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T05:07:31.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.857+0000 7fec98c80700 1 Processor -- start 2023-10-06T05:07:31.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.857+0000 7fec98c80700 1 -- start start 2023-10-06T05:07:31.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.858+0000 7fec98c80700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fec94159440 0x7fec940b3090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:31.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.858+0000 7fec98c80700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fec940b35d0 0x7fec940cbc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:31.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.858+0000 7fec98c80700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec940cb7a0 con 0x7fec940cc170 2023-10-06T05:07:31.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.858+0000 7fec98c80700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec940b0860 con 0x7fec940b35d0 2023-10-06T05:07:31.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.858+0000 7fec98c80700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec940cb150 con 0x7fec94159440 2023-10-06T05:07:31.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.858+0000 7fec92ffd700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fec940b35d0 0x7fec940cbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:31.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.858+0000 7fec93fff700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fec940cc170 0x7fec940b2100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:57472/0 (socket says 172.21.15.98:57472) 2023-10-06T05:07:31.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.858+0000 7fec93fff700 1 -- 172.21.15.98:0/426978413 learned_addr learned my addr 172.21.15.98:0/426978413 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:07:31.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.858+0000 7fec937fe700 1 --2- 172.21.15.98:0/426978413 >> v2:172.21.15.174:3300/0 conn(0x7fec94159440 0x7fec940b3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:31.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.858+0000 7fec92ffd700 1 -- 172.21.15.98:0/426978413 >> v2:172.21.15.174:3300/0 conn(0x7fec94159440 msgr2=0x7fec940b3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:31.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.858+0000 7fec92ffd700 1 --2- 172.21.15.98:0/426978413 >> v2:172.21.15.174:3300/0 conn(0x7fec94159440 0x7fec940b3090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:31.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.858+0000 7fec92ffd700 1 -- 172.21.15.98:0/426978413 >> v1:172.21.15.98:6789/0 conn(0x7fec940cc170 legacy=0x7fec940b2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:31.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.858+0000 7fec92ffd700 1 -- 172.21.15.98:0/426978413 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec940cb430 con 0x7fec940b35d0 2023-10-06T05:07:31.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.858+0000 7fec927fc700 1 -- 172.21.15.98:0/426978413 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3734434832 0 0) 0x7fec940cb7a0 con 0x7fec940cc170 2023-10-06T05:07:31.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.858+0000 7fec92ffd700 1 --2- 172.21.15.98:0/426978413 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec940b35d0 0x7fec940cbc30 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fec9406f9f0 tx=0x7fec880091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=849b7d3466062098 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:31.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.859+0000 7fec927fc700 1 -- 172.21.15.98:0/426978413 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec8800b6b0 con 0x7fec940b35d0 2023-10-06T05:07:31.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.859+0000 7fec927fc700 1 -- 172.21.15.98:0/426978413 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec8800b810 con 0x7fec940b35d0 2023-10-06T05:07:31.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.859+0000 7fec927fc700 1 -- 172.21.15.98:0/426978413 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec8800bbb0 con 0x7fec940b35d0 2023-10-06T05:07:31.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.859+0000 7fec98c80700 1 -- 172.21.15.98:0/426978413 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec940b35d0 msgr2=0x7fec940cbc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:31.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.859+0000 7fec98c80700 1 --2- 172.21.15.98:0/426978413 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec940b35d0 0x7fec940cbc30 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fec9406f9f0 tx=0x7fec880091a0 comp rx=0 tx=0).stop 2023-10-06T05:07:31.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.859+0000 7fec98c80700 1 -- 172.21.15.98:0/426978413 shutdown_connections 2023-10-06T05:07:31.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.859+0000 7fec98c80700 1 --2- 172.21.15.98:0/426978413 >> v2:172.21.15.174:3300/0 conn(0x7fec94159440 0x7fec940b3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:31.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.859+0000 7fec98c80700 1 --2- 172.21.15.98:0/426978413 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec940b35d0 0x7fec940cbc30 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:31.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.859+0000 7fec98c80700 1 -- 172.21.15.98:0/426978413 >> 172.21.15.98:0/426978413 conn(0x7fec9406ece0 msgr2=0x7fec9406f0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:31.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.859+0000 7fec98c80700 1 -- 172.21.15.98:0/426978413 shutdown_connections 2023-10-06T05:07:31.864 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.859+0000 7fec98c80700 1 -- 172.21.15.98:0/426978413 wait complete. 2023-10-06T05:07:31.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.860+0000 7fec98c80700 1 Processor -- start 2023-10-06T05:07:31.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.860+0000 7fec98c80700 1 -- start start 2023-10-06T05:07:31.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.861+0000 7fec98c80700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec940b35d0 0x7fec940c3f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:31.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.861+0000 7fec98c80700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fec940cc170 0x7fec940c4480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:31.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.861+0000 7fec98c80700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fec94159440 0x7fec9416cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:31.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.861+0000 7fec98c80700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec940c5c00 con 0x7fec940b35d0 2023-10-06T05:07:31.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.861+0000 7fec98c80700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec940c5e50 con 0x7fec940cc170 2023-10-06T05:07:31.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.861+0000 7fec98c80700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec940c6130 con 0x7fec94159440 2023-10-06T05:07:31.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.861+0000 7fec937fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec940b35d0 0x7fec940c3f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:31.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.861+0000 7fec937fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec940b35d0 0x7fec940c3f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45124/0 (socket says 172.21.15.98:45124) 2023-10-06T05:07:31.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.861+0000 7fec937fe700 1 -- 172.21.15.98:0/3322505868 learned_addr learned my addr 172.21.15.98:0/3322505868 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:31.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.862+0000 7fec92ffd700 1 --2- 172.21.15.98:0/3322505868 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fec940cc170 0x7fec940c4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:31.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.862+0000 7fec93fff700 1 --2- 172.21.15.98:0/3322505868 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fec94159440 0x7fec9416cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:31.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.862+0000 7fec937fe700 1 -- 172.21.15.98:0/3322505868 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fec94159440 msgr2=0x7fec9416cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:31.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.862+0000 7fec937fe700 1 --2- 172.21.15.98:0/3322505868 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fec94159440 0x7fec9416cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:31.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.862+0000 7fec937fe700 1 -- 172.21.15.98:0/3322505868 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fec940cc170 msgr2=0x7fec940c4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:31.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.862+0000 7fec937fe700 1 --2- 172.21.15.98:0/3322505868 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fec940cc170 0x7fec940c4480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:31.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.862+0000 7fec937fe700 1 -- 172.21.15.98:0/3322505868 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec88009420 con 0x7fec940b35d0 2023-10-06T05:07:31.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.862+0000 7fec92ffd700 1 --2- 172.21.15.98:0/3322505868 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fec940cc170 0x7fec940c4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:07:31.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.862+0000 7fec93fff700 1 --2- 172.21.15.98:0/3322505868 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fec94159440 0x7fec9416cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:07:31.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.862+0000 7fec937fe700 1 --2- 172.21.15.98:0/3322505868 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec940b35d0 0x7fec940c3f40 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fec8400d860 tx=0x7fec84008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:31.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.862+0000 7fec7b7fe700 1 -- 172.21.15.98:0/3322505868 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec84009610 con 0x7fec940b35d0 2023-10-06T05:07:31.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.862+0000 7fec7b7fe700 1 -- 172.21.15.98:0/3322505868 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec84009770 con 0x7fec940b35d0 2023-10-06T05:07:31.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.862+0000 7fec98c80700 1 -- 172.21.15.98:0/3322505868 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec940c6410 con 0x7fec940b35d0 2023-10-06T05:07:31.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.862+0000 7fec7b7fe700 1 -- 172.21.15.98:0/3322505868 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec84009b30 con 0x7fec940b35d0 2023-10-06T05:07:31.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.862+0000 7fec98c80700 1 -- 172.21.15.98:0/3322505868 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec940c49c0 con 0x7fec940b35d0 2023-10-06T05:07:31.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.863+0000 7fec98c80700 1 -- 172.21.15.98:0/3322505868 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec940c68b0 con 0x7fec940b35d0 2023-10-06T05:07:31.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.863+0000 7fec7b7fe700 1 -- 172.21.15.98:0/3322505868 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fec8401e460 con 0x7fec940b35d0 2023-10-06T05:07:31.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.864+0000 7fec7b7fe700 1 --2- 172.21.15.98:0/3322505868 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fec80066f30 0x7fec800693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:31.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.864+0000 7fec7b7fe700 1 -- 172.21.15.98:0/3322505868 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fec84084ee0 con 0x7fec940b35d0 2023-10-06T05:07:31.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.864+0000 7fec92ffd700 1 --2- 172.21.15.98:0/3322505868 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fec80066f30 0x7fec800693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:31.870 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.864+0000 7fec92ffd700 1 --2- 172.21.15.98:0/3322505868 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fec80066f30 0x7fec800693e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fec94157c80 tx=0x7fec88017000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:31.870 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:31.868+0000 7fec7b7fe700 1 -- 172.21.15.98:0/3322505868 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fec84051d00 con 0x7fec940b35d0 2023-10-06T05:07:32.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.054+0000 7fec98c80700 1 -- 172.21.15.98:0/3322505868 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fec9405f230 con 0x7fec940b35d0 2023-10-06T05:07:32.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.054+0000 7fec7b7fe700 1 -- 172.21.15.98:0/3322505868 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+18707 (secure 0 0 0) 0x7fec84055320 con 0x7fec940b35d0 2023-10-06T05:07:32.057 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:07:32.058 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":29,"fsid":"f50c5df5-cad5-47a4-9bb1-8c791d4311ec","created":"2023-10-06T05:06:32.971175+0000","modified":"2023-10-06T05:07:02.131672+0000","last_up_change":"2023-10-06T05:06:44.016489+0000","last_in_change":"2023-10-06T05:06:35.544511+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-06T05:06:44.559671+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-06T05:06:51.432161+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-10-06T05:06:55.589057+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-10-06T05:06:56.041687+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-10-06T05:06:59.754099+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d9495d95-da75-4c3b-8927-e9bae6a8b093","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":90627},{"type":"v1","addr":"172.21.15.98:6801","nonce":90627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":90627},{"type":"v1","addr":"172.21.15.98:6803","nonce":90627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":90627},{"type":"v1","addr":"172.21.15.98:6807","nonce":90627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":90627},{"type":"v1","addr":"172.21.15.98:6805","nonce":90627}]},"public_addr":"172.21.15.98:6801/90627","cluster_addr":"172.21.15.98:6803/90627","heartbeat_back_addr":"172.21.15.98:6807/90627","heartbeat_front_addr":"172.21.15.98:6805/90627","state":["exists","up"]},{"osd":1,"uuid":"f25ba5be-d65e-4707-a134-154092deb754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":90628},{"type":"v1","addr":"172.21.15.98:6809","nonce":90628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":90628},{"type":"v1","addr":"172.21.15.98:6811","nonce":90628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":90628},{"type":"v1","addr":"172.21.15.98:6815","nonce":90628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":90628},{"type":"v1","addr":"172.21.15.98:6813","nonce":90628}]},"public_addr":"172.21.15.98:6809/90628","cluster_addr":"172.21.15.98:6811/90628","heartbeat_back_addr":"172.21.15.98:6815/90628","heartbeat_front_addr":"172.21.15.98:6813/90628","state":["exists","up"]},{"osd":2,"uuid":"5ab9e028-945c-43e2-8c95-3b0b4cf38e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":90662},{"type":"v1","addr":"172.21.15.98:6818","nonce":90662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":90662},{"type":"v1","addr":"172.21.15.98:6822","nonce":90662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":90662},{"type":"v1","addr":"172.21.15.98:6830","nonce":90662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":90662},{"type":"v1","addr":"172.21.15.98:6826","nonce":90662}]},"public_addr":"172.21.15.98:6818/90662","cluster_addr":"172.21.15.98:6822/90662","heartbeat_back_addr":"172.21.15.98:6830/90662","heartbeat_front_addr":"172.21.15.98:6826/90662","state":["exists","up"]},{"osd":3,"uuid":"3bb803ff-1738-46bf-be62-2267bf378822","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6817","nonce":90663},{"type":"v1","addr":"172.21.15.98:6819","nonce":90663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6821","nonce":90663},{"type":"v1","addr":"172.21.15.98:6823","nonce":90663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6829","nonce":90663},{"type":"v1","addr":"172.21.15.98:6831","nonce":90663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6825","nonce":90663},{"type":"v1","addr":"172.21.15.98:6827","nonce":90663}]},"public_addr":"172.21.15.98:6819/90663","cluster_addr":"172.21.15.98:6823/90663","heartbeat_back_addr":"172.21.15.98:6831/90663","heartbeat_front_addr":"172.21.15.98:6827/90663","state":["exists","up"]},{"osd":4,"uuid":"7d57a4c9-6462-43f6-b64a-7f9e9b4353d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":90847},{"type":"v1","addr":"172.21.15.174:6801","nonce":90847}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":90847},{"type":"v1","addr":"172.21.15.174:6803","nonce":90847}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":90847},{"type":"v1","addr":"172.21.15.174:6807","nonce":90847}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":90847},{"type":"v1","addr":"172.21.15.174:6805","nonce":90847}]},"public_addr":"172.21.15.174:6801/90847","cluster_addr":"172.21.15.174:6803/90847","heartbeat_back_addr":"172.21.15.174:6807/90847","heartbeat_front_addr":"172.21.15.174:6805/90847","state":["exists","up"]},{"osd":5,"uuid":"83287ea4-e44d-4af9-bc77-36d47c60d4a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":90848},{"type":"v1","addr":"172.21.15.174:6809","nonce":90848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":90848},{"type":"v1","addr":"172.21.15.174:6811","nonce":90848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":90848},{"type":"v1","addr":"172.21.15.174:6815","nonce":90848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":90848},{"type":"v1","addr":"172.21.15.174:6813","nonce":90848}]},"public_addr":"172.21.15.174:6809/90848","cluster_addr":"172.21.15.174:6811/90848","heartbeat_back_addr":"172.21.15.174:6815/90848","heartbeat_front_addr":"172.21.15.174:6813/90848","state":["exists","up"]},{"osd":6,"uuid":"496a6272-892e-43c4-a145-206623c22f99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":90877},{"type":"v1","addr":"172.21.15.174:6817","nonce":90877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":90877},{"type":"v1","addr":"172.21.15.174:6819","nonce":90877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":90877},{"type":"v1","addr":"172.21.15.174:6823","nonce":90877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":90877},{"type":"v1","addr":"172.21.15.174:6821","nonce":90877}]},"public_addr":"172.21.15.174:6817/90877","cluster_addr":"172.21.15.174:6819/90877","heartbeat_back_addr":"172.21.15.174:6823/90877","heartbeat_front_addr":"172.21.15.174:6821/90877","state":["exists","up"]},{"osd":7,"uuid":"49967e34-8b07-4bab-a13e-0c16d6a0f3a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":90878},{"type":"v1","addr":"172.21.15.174:6825","nonce":90878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":90878},{"type":"v1","addr":"172.21.15.174:6827","nonce":90878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":90878},{"type":"v1","addr":"172.21.15.174:6831","nonce":90878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":90878},{"type":"v1","addr":"172.21.15.174:6829","nonce":90878}]},"public_addr":"172.21.15.174:6825/90878","cluster_addr":"172.21.15.174:6827/90878","heartbeat_back_addr":"172.21.15.174:6831/90878","heartbeat_front_addr":"172.21.15.174:6829/90878","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.446479+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.514266+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.794108+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.810894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.731919+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.717756+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.737435+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.508682+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T05:07:32.058 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.056+0000 7fec98c80700 1 -- 172.21.15.98:0/3322505868 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fec80066f30 msgr2=0x7fec800693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:32.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.056+0000 7fec98c80700 1 --2- 172.21.15.98:0/3322505868 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fec80066f30 0x7fec800693e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fec94157c80 tx=0x7fec88017000 comp rx=0 tx=0).stop 2023-10-06T05:07:32.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.056+0000 7fec98c80700 1 -- 172.21.15.98:0/3322505868 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec940b35d0 msgr2=0x7fec940c3f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:32.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.056+0000 7fec98c80700 1 --2- 172.21.15.98:0/3322505868 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec940b35d0 0x7fec940c3f40 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fec8400d860 tx=0x7fec84008040 comp rx=0 tx=0).stop 2023-10-06T05:07:32.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.056+0000 7fec98c80700 1 -- 172.21.15.98:0/3322505868 shutdown_connections 2023-10-06T05:07:32.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.056+0000 7fec98c80700 1 --2- 172.21.15.98:0/3322505868 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fec94159440 0x7fec9416cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:32.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.056+0000 7fec98c80700 1 --2- 172.21.15.98:0/3322505868 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fec80066f30 0x7fec800693e0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:32.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.056+0000 7fec98c80700 1 --2- 172.21.15.98:0/3322505868 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec940b35d0 0x7fec940c3f40 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:32.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.056+0000 7fec98c80700 1 --2- 172.21.15.98:0/3322505868 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fec940cc170 0x7fec940c4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:32.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.056+0000 7fec98c80700 1 -- 172.21.15.98:0/3322505868 >> 172.21.15.98:0/3322505868 conn(0x7fec9406ece0 msgr2=0x7fec9415be70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:32.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.056+0000 7fec98c80700 1 -- 172.21.15.98:0/3322505868 shutdown_connections 2023-10-06T05:07:32.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.057+0000 7fec98c80700 1 -- 172.21.15.98:0/3322505868 wait complete. 2023-10-06T05:07:32.070 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-06T05:06:44.559671+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-10-06T05:06:51.432161+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-06T05:06:55.589057+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-10-06T05:06:56.041687+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-06T05:06:59.754099+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '29', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-06T05:07:32.071 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-06T05:07:32.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.187+0000 7fe578d6f700 1 Processor -- start 2023-10-06T05:07:32.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.187+0000 7fe578d6f700 1 -- start start 2023-10-06T05:07:32.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.188+0000 7fe578d6f700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe5740b2620 0x7fe5740cccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:32.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.188+0000 7fe578d6f700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe5740b29f0 0x7fe5740cd1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:32.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.188+0000 7fe578d6f700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fe5740ceaa0 0x7fe5740cef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:32.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.188+0000 7fe578d6f700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5740cc970 con 0x7fe5740b2620 2023-10-06T05:07:32.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.188+0000 7fe578d6f700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5740cf480 con 0x7fe5740ceaa0 2023-10-06T05:07:32.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.188+0000 7fe578d6f700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5740cbd60 con 0x7fe5740b29f0 2023-10-06T05:07:32.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.188+0000 7fe5737fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe5740b2620 0x7fe5740cccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:32.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.188+0000 7fe5737fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe5740b2620 0x7fe5740cccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45132/0 (socket says 172.21.15.98:45132) 2023-10-06T05:07:32.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.188+0000 7fe5737fe700 1 -- 172.21.15.98:0/25271333 learned_addr learned my addr 172.21.15.98:0/25271333 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:32.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.188+0000 7fe573fff700 1 --2- 172.21.15.98:0/25271333 >> v2:172.21.15.174:3300/0 conn(0x7fe5740ceaa0 0x7fe5740cef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:32.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.188+0000 7fe572ffd700 1 --2- 172.21.15.98:0/25271333 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe5740b29f0 0x7fe5740cd1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:32.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.188+0000 7fe5737fe700 1 -- 172.21.15.98:0/25271333 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe5740b29f0 msgr2=0x7fe5740cd1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:32.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.188+0000 7fe5737fe700 1 --2- 172.21.15.98:0/25271333 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe5740b29f0 0x7fe5740cd1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:32.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.188+0000 7fe5737fe700 1 -- 172.21.15.98:0/25271333 >> v2:172.21.15.174:3300/0 conn(0x7fe5740ceaa0 msgr2=0x7fe5740cef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:32.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.188+0000 7fe5737fe700 1 --2- 172.21.15.98:0/25271333 >> v2:172.21.15.174:3300/0 conn(0x7fe5740ceaa0 0x7fe5740cef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:32.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.188+0000 7fe5737fe700 1 -- 172.21.15.98:0/25271333 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5740cc040 con 0x7fe5740b2620 2023-10-06T05:07:32.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.188+0000 7fe572ffd700 1 --2- 172.21.15.98:0/25271333 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe5740b29f0 0x7fe5740cd1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:07:32.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.189+0000 7fe5737fe700 1 --2- 172.21.15.98:0/25271333 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5740b2620 0x7fe5740cccb0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fe574089650 tx=0x7fe564009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=293078f035e672ec server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:32.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.189+0000 7fe5727fc700 1 -- 172.21.15.98:0/25271333 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe564005c10 con 0x7fe5740b2620 2023-10-06T05:07:32.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.189+0000 7fe5727fc700 1 -- 172.21.15.98:0/25271333 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe564005d70 con 0x7fe5740b2620 2023-10-06T05:07:32.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.189+0000 7fe5727fc700 1 -- 172.21.15.98:0/25271333 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe56400b880 con 0x7fe5740b2620 2023-10-06T05:07:32.192 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.189+0000 7fe578d6f700 1 -- 172.21.15.98:0/25271333 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5740b2620 msgr2=0x7fe5740cccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:32.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.189+0000 7fe578d6f700 1 --2- 172.21.15.98:0/25271333 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5740b2620 0x7fe5740cccb0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fe574089650 tx=0x7fe564009220 comp rx=0 tx=0).stop 2023-10-06T05:07:32.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.189+0000 7fe578d6f700 1 -- 172.21.15.98:0/25271333 shutdown_connections 2023-10-06T05:07:32.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.189+0000 7fe578d6f700 1 --2- 172.21.15.98:0/25271333 >> v2:172.21.15.174:3300/0 conn(0x7fe5740ceaa0 0x7fe5740cef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:32.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.189+0000 7fe578d6f700 1 --2- 172.21.15.98:0/25271333 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5740b2620 0x7fe5740cccb0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:32.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.189+0000 7fe578d6f700 1 --2- 172.21.15.98:0/25271333 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe5740b29f0 0x7fe5740cd1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:32.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.189+0000 7fe578d6f700 1 -- 172.21.15.98:0/25271333 >> 172.21.15.98:0/25271333 conn(0x7fe574099520 msgr2=0x7fe57406f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:32.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.189+0000 7fe578d6f700 1 -- 172.21.15.98:0/25271333 shutdown_connections 2023-10-06T05:07:32.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.190+0000 7fe578d6f700 1 -- 172.21.15.98:0/25271333 wait complete. 2023-10-06T05:07:32.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.191+0000 7fe578d6f700 1 Processor -- start 2023-10-06T05:07:32.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.191+0000 7fe578d6f700 1 -- start start 2023-10-06T05:07:32.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.191+0000 7fe578d6f700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe5740b2620 0x7fe5740c98b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:32.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.191+0000 7fe578d6f700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe5740b29f0 0x7fe57415d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:32.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.191+0000 7fe578d6f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5740ceaa0 0x7fe574156b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:32.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.191+0000 7fe578d6f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5740c8d90 con 0x7fe5740ceaa0 2023-10-06T05:07:32.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.191+0000 7fe578d6f700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe57415db30 con 0x7fe5740b29f0 2023-10-06T05:07:32.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.191+0000 7fe578d6f700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5740c9220 con 0x7fe5740b2620 2023-10-06T05:07:32.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.191+0000 7fe573fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5740ceaa0 0x7fe574156b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:32.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.192+0000 7fe573fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5740ceaa0 0x7fe574156b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45142/0 (socket says 172.21.15.98:45142) 2023-10-06T05:07:32.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.192+0000 7fe573fff700 1 -- 172.21.15.98:0/2009478556 learned_addr learned my addr 172.21.15.98:0/2009478556 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:32.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.192+0000 7fe572ffd700 1 --2- 172.21.15.98:0/2009478556 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe5740b29f0 0x7fe57415d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:32.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.192+0000 7fe5737fe700 1 --2- 172.21.15.98:0/2009478556 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe5740b2620 0x7fe5740c98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:32.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.192+0000 7fe573fff700 1 -- 172.21.15.98:0/2009478556 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe5740b2620 msgr2=0x7fe5740c98b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:32.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.192+0000 7fe573fff700 1 --2- 172.21.15.98:0/2009478556 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe5740b2620 0x7fe5740c98b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:32.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.192+0000 7fe573fff700 1 -- 172.21.15.98:0/2009478556 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe5740b29f0 msgr2=0x7fe57415d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:32.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.192+0000 7fe573fff700 1 --2- 172.21.15.98:0/2009478556 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe5740b29f0 0x7fe57415d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:32.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.192+0000 7fe573fff700 1 -- 172.21.15.98:0/2009478556 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe564009480 con 0x7fe5740ceaa0 2023-10-06T05:07:32.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.192+0000 7fe5737fe700 1 --2- 172.21.15.98:0/2009478556 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe5740b2620 0x7fe5740c98b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:07:32.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.192+0000 7fe573fff700 1 --2- 172.21.15.98:0/2009478556 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5740ceaa0 0x7fe574156b30 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fe56c009fa0 tx=0x7fe56c009900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:32.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.192+0000 7fe55b7fe700 1 -- 172.21.15.98:0/2009478556 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe56c00ec70 con 0x7fe5740ceaa0 2023-10-06T05:07:32.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.192+0000 7fe55b7fe700 1 -- 172.21.15.98:0/2009478556 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe56c00edd0 con 0x7fe5740ceaa0 2023-10-06T05:07:32.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.192+0000 7fe578d6f700 1 -- 172.21.15.98:0/2009478556 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5740c8460 con 0x7fe5740ceaa0 2023-10-06T05:07:32.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.192+0000 7fe55b7fe700 1 -- 172.21.15.98:0/2009478556 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe56c014670 con 0x7fe5740ceaa0 2023-10-06T05:07:32.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.192+0000 7fe578d6f700 1 -- 172.21.15.98:0/2009478556 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5740c8a20 con 0x7fe5740ceaa0 2023-10-06T05:07:32.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.193+0000 7fe578d6f700 1 -- 172.21.15.98:0/2009478556 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5740c8f40 con 0x7fe5740ceaa0 2023-10-06T05:07:32.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.194+0000 7fe55b7fe700 1 -- 172.21.15.98:0/2009478556 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fe56c0149e0 con 0x7fe5740ceaa0 2023-10-06T05:07:32.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.194+0000 7fe55b7fe700 1 --2- 172.21.15.98:0/2009478556 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fe560066f30 0x7fe5600693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:32.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.194+0000 7fe55b7fe700 1 -- 172.21.15.98:0/2009478556 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fe56c085020 con 0x7fe5740ceaa0 2023-10-06T05:07:32.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.194+0000 7fe5737fe700 1 --2- 172.21.15.98:0/2009478556 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fe560066f30 0x7fe5600693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:32.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.194+0000 7fe5737fe700 1 --2- 172.21.15.98:0/2009478556 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fe560066f30 0x7fe5600693e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe564009260 tx=0x7fe564016000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:32.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.198+0000 7fe55b7fe700 1 -- 172.21.15.98:0/2009478556 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe5740c8f40 con 0x7fe5740ceaa0 2023-10-06T05:07:32.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.378+0000 7fe578d6f700 1 -- 172.21.15.98:0/2009478556 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe57405f230 con 0x7fe5740ceaa0 2023-10-06T05:07:32.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.379+0000 7fe55b7fe700 1 -- 172.21.15.98:0/2009478556 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v29) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe56c051e40 con 0x7fe5740ceaa0 2023-10-06T05:07:32.380 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2023-10-06T05:07:32.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.381+0000 7fe578d6f700 1 -- 172.21.15.98:0/2009478556 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fe560066f30 msgr2=0x7fe5600693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:32.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.381+0000 7fe578d6f700 1 --2- 172.21.15.98:0/2009478556 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fe560066f30 0x7fe5600693e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe564009260 tx=0x7fe564016000 comp rx=0 tx=0).stop 2023-10-06T05:07:32.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.381+0000 7fe578d6f700 1 -- 172.21.15.98:0/2009478556 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5740ceaa0 msgr2=0x7fe574156b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:32.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.381+0000 7fe578d6f700 1 --2- 172.21.15.98:0/2009478556 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5740ceaa0 0x7fe574156b30 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fe56c009fa0 tx=0x7fe56c009900 comp rx=0 tx=0).stop 2023-10-06T05:07:32.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.381+0000 7fe578d6f700 1 -- 172.21.15.98:0/2009478556 shutdown_connections 2023-10-06T05:07:32.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.381+0000 7fe578d6f700 1 --2- 172.21.15.98:0/2009478556 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fe560066f30 0x7fe5600693e0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:32.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.381+0000 7fe578d6f700 1 --2- 172.21.15.98:0/2009478556 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe5740ceaa0 0x7fe574156b30 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:32.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.381+0000 7fe578d6f700 1 --2- 172.21.15.98:0/2009478556 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe5740b2620 0x7fe5740c98b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:32.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.381+0000 7fe578d6f700 1 --2- 172.21.15.98:0/2009478556 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe5740b29f0 0x7fe57415d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:32.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.381+0000 7fe578d6f700 1 -- 172.21.15.98:0/2009478556 >> 172.21.15.98:0/2009478556 conn(0x7fe574099520 msgr2=0x7fe574068d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:32.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.381+0000 7fe578d6f700 1 -- 172.21.15.98:0/2009478556 shutdown_connections 2023-10-06T05:07:32.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.381+0000 7fe578d6f700 1 -- 172.21.15.98:0/2009478556 wait complete. 2023-10-06T05:07:32.394 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-10-06T05:07:32.511 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.510+0000 7f0605b43700 1 Processor -- start 2023-10-06T05:07:32.511 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.510+0000 7f0605b43700 1 -- start start 2023-10-06T05:07:32.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.511+0000 7f0605b43700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f06001594d0 0x7f06000cb560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:32.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.511+0000 7f0605b43700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06000cbee0 con 0x7f06001598a0 2023-10-06T05:07:32.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.511+0000 7f0605b43700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06000cf5d0 con 0x7f06000cbaa0 2023-10-06T05:07:32.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.511+0000 7f0605b43700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06000cf8b0 con 0x7f06001594d0 2023-10-06T05:07:32.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.511+0000 7f05f6ffd700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f06001598a0 0x7f06000ce7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:57482/0 (socket says 172.21.15.98:57482) 2023-10-06T05:07:32.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.511+0000 7f05f6ffd700 1 -- 172.21.15.98:0/2516776011 learned_addr learned my addr 172.21.15.98:0/2516776011 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:07:32.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.511+0000 7f05ff7fe700 1 --2- 172.21.15.98:0/2516776011 >> v2:172.21.15.174:3300/0 conn(0x7f06001594d0 0x7f06000cb560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:32.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.511+0000 7f05feffd700 1 -- 172.21.15.98:0/2516776011 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3164649803 0 0) 0x7f06000cbee0 con 0x7f06001598a0 2023-10-06T05:07:32.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.511+0000 7f05feffd700 1 -- 172.21.15.98:0/2516776011 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05ec003570 con 0x7f06001598a0 2023-10-06T05:07:32.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.511+0000 7f05ff7fe700 1 -- 172.21.15.98:0/2516776011 >> v1:172.21.15.174:6789/0 conn(0x7f06000cbaa0 legacy=0x7f06000c5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:32.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.511+0000 7f05ff7fe700 1 -- 172.21.15.98:0/2516776011 >> v1:172.21.15.98:6789/0 conn(0x7f06001598a0 legacy=0x7f06000ce7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:32.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.511+0000 7f05ff7fe700 1 -- 172.21.15.98:0/2516776011 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06000cfb90 con 0x7f06001594d0 2023-10-06T05:07:32.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.512+0000 7f05ff7fe700 1 --2- 172.21.15.98:0/2516776011 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06001594d0 0x7f06000cb560 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f05f00098e0 tx=0x7f05f00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fb46594a220ac2eb server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:32.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.512+0000 7f05feffd700 1 -- 172.21.15.98:0/2516776011 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f000b640 con 0x7f06001594d0 2023-10-06T05:07:32.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.512+0000 7f05feffd700 1 -- 172.21.15.98:0/2516776011 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05f000b7a0 con 0x7f06001594d0 2023-10-06T05:07:32.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.512+0000 7f05feffd700 1 -- 172.21.15.98:0/2516776011 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f000bb20 con 0x7f06001594d0 2023-10-06T05:07:32.516 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.512+0000 7f0605b43700 1 -- 172.21.15.98:0/2516776011 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06001594d0 msgr2=0x7f06000cb560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:32.517 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.512+0000 7f0605b43700 1 --2- 172.21.15.98:0/2516776011 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06001594d0 0x7f06000cb560 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f05f00098e0 tx=0x7f05f00091a0 comp rx=0 tx=0).stop 2023-10-06T05:07:32.517 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.512+0000 7f0605b43700 1 -- 172.21.15.98:0/2516776011 shutdown_connections 2023-10-06T05:07:32.517 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.512+0000 7f0605b43700 1 --2- 172.21.15.98:0/2516776011 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06001594d0 0x7f06000cb560 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:32.517 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.512+0000 7f0605b43700 1 -- 172.21.15.98:0/2516776011 >> 172.21.15.98:0/2516776011 conn(0x7f0600089e50 msgr2=0x7f0600099770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:32.517 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.512+0000 7f0605b43700 1 -- 172.21.15.98:0/2516776011 shutdown_connections 2023-10-06T05:07:32.517 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.512+0000 7f0605b43700 1 -- 172.21.15.98:0/2516776011 wait complete. 2023-10-06T05:07:32.517 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.513+0000 7f0605b43700 1 Processor -- start 2023-10-06T05:07:32.517 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.514+0000 7f0605b43700 1 -- start start 2023-10-06T05:07:32.517 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.514+0000 7f0605b43700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06000cbaa0 0x7f060015d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:32.517 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.514+0000 7f0605b43700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06001594d0 0x7f060009c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:32.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.514+0000 7f0605b43700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f06001598a0 0x7f060014ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:32.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.514+0000 7f0605b43700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f060009dc20 con 0x7f06000cbaa0 2023-10-06T05:07:32.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.514+0000 7f0605b43700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f060009d010 con 0x7f06001594d0 2023-10-06T05:07:32.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.514+0000 7f0605b43700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f060009d2f0 con 0x7f06001598a0 2023-10-06T05:07:32.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.514+0000 7f05ff7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06000cbaa0 0x7f060015d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:32.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.514+0000 7f05ff7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06000cbaa0 0x7f060015d880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45154/0 (socket says 172.21.15.98:45154) 2023-10-06T05:07:32.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.514+0000 7f05ff7fe700 1 -- 172.21.15.98:0/371890150 learned_addr learned my addr 172.21.15.98:0/371890150 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:32.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.515+0000 7f05ff7fe700 1 -- 172.21.15.98:0/371890150 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f06001598a0 msgr2=0x7f060014ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:32.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.515+0000 7f05ff7fe700 1 --2- 172.21.15.98:0/371890150 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f06001598a0 0x7f060014ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:32.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.515+0000 7f05ff7fe700 1 -- 172.21.15.98:0/371890150 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06001594d0 msgr2=0x7f060009c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:32.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.515+0000 7f05f6ffd700 1 --2- 172.21.15.98:0/371890150 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06001594d0 0x7f060009c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:32.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.515+0000 7f05ff7fe700 1 --2- 172.21.15.98:0/371890150 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06001594d0 0x7f060009c5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:32.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.515+0000 7f05ff7fe700 1 -- 172.21.15.98:0/371890150 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05f0009420 con 0x7f06000cbaa0 2023-10-06T05:07:32.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.515+0000 7f05ff7fe700 1 --2- 172.21.15.98:0/371890150 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06000cbaa0 0x7f060015d880 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f06000cc070 tx=0x7f05f0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:32.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.515+0000 7f05f7fff700 1 -- 172.21.15.98:0/371890150 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f000bb20 con 0x7f06000cbaa0 2023-10-06T05:07:32.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.515+0000 7f05f7fff700 1 -- 172.21.15.98:0/371890150 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05f0011420 con 0x7f06000cbaa0 2023-10-06T05:07:32.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.515+0000 7f0605b43700 1 -- 172.21.15.98:0/371890150 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f060009d5d0 con 0x7f06000cbaa0 2023-10-06T05:07:32.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.515+0000 7f05f7fff700 1 -- 172.21.15.98:0/371890150 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f0011800 con 0x7f06000cbaa0 2023-10-06T05:07:32.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.515+0000 7f0605b43700 1 -- 172.21.15.98:0/371890150 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f060015ddc0 con 0x7f06000cbaa0 2023-10-06T05:07:32.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.516+0000 7f0605b43700 1 -- 172.21.15.98:0/371890150 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f060015e010 con 0x7f06000cbaa0 2023-10-06T05:07:32.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.516+0000 7f05f7fff700 1 -- 172.21.15.98:0/371890150 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f05f0011c30 con 0x7f06000cbaa0 2023-10-06T05:07:32.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.517+0000 7f05f7fff700 1 --2- 172.21.15.98:0/371890150 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f05e0066f30 0x7f05e00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:32.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.517+0000 7f05f6ffd700 1 --2- 172.21.15.98:0/371890150 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f05e0066f30 0x7f05e00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:32.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.517+0000 7f05f7fff700 1 -- 172.21.15.98:0/371890150 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f05f00863b0 con 0x7f06000cbaa0 2023-10-06T05:07:32.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.517+0000 7f05f6ffd700 1 --2- 172.21.15.98:0/371890150 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f05e0066f30 0x7f05e00693e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f05e80058d0 tx=0x7f05e8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:32.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.521+0000 7f05f7fff700 1 -- 172.21.15.98:0/371890150 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f05f0053130 con 0x7f06000cbaa0 2023-10-06T05:07:32.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.704+0000 7f0605b43700 1 -- 172.21.15.98:0/371890150 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f0600064a90 con 0x7f06000cbaa0 2023-10-06T05:07:32.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.705+0000 7f05f7fff700 1 -- 172.21.15.98:0/371890150 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v29) v1 ==== 92+0+10 (secure 0 0 0) 0x7f05f0056750 con 0x7f06000cbaa0 2023-10-06T05:07:32.706 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 8 2023-10-06T05:07:32.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.707+0000 7f0605b43700 1 -- 172.21.15.98:0/371890150 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f05e0066f30 msgr2=0x7f05e00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:32.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.707+0000 7f0605b43700 1 --2- 172.21.15.98:0/371890150 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f05e0066f30 0x7f05e00693e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f05e80058d0 tx=0x7f05e8009040 comp rx=0 tx=0).stop 2023-10-06T05:07:32.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.707+0000 7f0605b43700 1 -- 172.21.15.98:0/371890150 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06000cbaa0 msgr2=0x7f060015d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:32.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.707+0000 7f0605b43700 1 --2- 172.21.15.98:0/371890150 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06000cbaa0 0x7f060015d880 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f06000cc070 tx=0x7f05f0017000 comp rx=0 tx=0).stop 2023-10-06T05:07:32.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.707+0000 7f0605b43700 1 -- 172.21.15.98:0/371890150 shutdown_connections 2023-10-06T05:07:32.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.707+0000 7f0605b43700 1 --2- 172.21.15.98:0/371890150 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f06001598a0 0x7f060014ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:32.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.707+0000 7f0605b43700 1 --2- 172.21.15.98:0/371890150 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f05e0066f30 0x7f05e00693e0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:32.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.707+0000 7f0605b43700 1 --2- 172.21.15.98:0/371890150 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f06000cbaa0 0x7f060015d880 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:32.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.707+0000 7f0605b43700 1 --2- 172.21.15.98:0/371890150 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06001594d0 0x7f060009c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:32.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.707+0000 7f0605b43700 1 -- 172.21.15.98:0/371890150 >> 172.21.15.98:0/371890150 conn(0x7f0600089e50 msgr2=0x7f060015bcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:32.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.707+0000 7f0605b43700 1 -- 172.21.15.98:0/371890150 shutdown_connections 2023-10-06T05:07:32.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.707+0000 7f0605b43700 1 -- 172.21.15.98:0/371890150 wait complete. 2023-10-06T05:07:32.720 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-06T05:07:32.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.837+0000 7f63e335c700 1 Processor -- start 2023-10-06T05:07:32.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.837+0000 7f63e335c700 1 -- start start 2023-10-06T05:07:32.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.837+0000 7f63e335c700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f63dc0b2620 0x7f63dc0cccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:32.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.837+0000 7f63e335c700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63dc0cc970 con 0x7f63dc0b29f0 2023-10-06T05:07:32.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.837+0000 7f63e335c700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63dc0cbd60 con 0x7f63dc0cf1b0 2023-10-06T05:07:32.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.837+0000 7f63e335c700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63dc0cc040 con 0x7f63dc0b2620 2023-10-06T05:07:32.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.838+0000 7f63e1b59700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f63dc0b29f0 0x7f63dc0ceaa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:57492/0 (socket says 172.21.15.98:57492) 2023-10-06T05:07:32.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.838+0000 7f63e1b59700 1 -- 172.21.15.98:0/982899172 learned_addr learned my addr 172.21.15.98:0/982899172 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:07:32.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.838+0000 7f63e235a700 1 --2- 172.21.15.98:0/982899172 >> v2:172.21.15.174:3300/0 conn(0x7f63dc0b2620 0x7f63dc0cccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:32.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.838+0000 7f63e1358700 1 -- 172.21.15.98:0/982899172 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3261491567 0 0) 0x7f63dc0cc970 con 0x7f63dc0b29f0 2023-10-06T05:07:32.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.838+0000 7f63e1358700 1 -- 172.21.15.98:0/982899172 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63c8003570 con 0x7f63dc0b29f0 2023-10-06T05:07:32.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.838+0000 7f63e1358700 1 -- 172.21.15.98:0/982899172 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 517364496 0 0) 0x7f63dc0cbd60 con 0x7f63dc0cf1b0 2023-10-06T05:07:32.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.838+0000 7f63e1358700 1 -- 172.21.15.98:0/982899172 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63dc0cc970 con 0x7f63dc0cf1b0 2023-10-06T05:07:32.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.838+0000 7f63e235a700 1 -- 172.21.15.98:0/982899172 >> v1:172.21.15.174:6789/0 conn(0x7f63dc0cf1b0 legacy=0x7f63dc0c5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:32.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.838+0000 7f63e235a700 1 -- 172.21.15.98:0/982899172 >> v1:172.21.15.98:6789/0 conn(0x7f63dc0b29f0 legacy=0x7f63dc0ceaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:32.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.838+0000 7f63e235a700 1 -- 172.21.15.98:0/982899172 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63dc0cc320 con 0x7f63dc0b2620 2023-10-06T05:07:32.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.838+0000 7f63e235a700 1 --2- 172.21.15.98:0/982899172 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63dc0b2620 0x7f63dc0cccb0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f63d80098d0 tx=0x7f63d8009120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=384d44bb50828dc0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:32.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.838+0000 7f63e1358700 1 -- 172.21.15.98:0/982899172 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d8005c10 con 0x7f63dc0b2620 2023-10-06T05:07:32.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.838+0000 7f63e1358700 1 -- 172.21.15.98:0/982899172 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63d8005d70 con 0x7f63dc0b2620 2023-10-06T05:07:32.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.839+0000 7f63e1358700 1 -- 172.21.15.98:0/982899172 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d8006260 con 0x7f63dc0b2620 2023-10-06T05:07:32.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.839+0000 7f63e335c700 1 -- 172.21.15.98:0/982899172 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63dc0b2620 msgr2=0x7f63dc0cccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:32.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.839+0000 7f63e335c700 1 --2- 172.21.15.98:0/982899172 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63dc0b2620 0x7f63dc0cccb0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f63d80098d0 tx=0x7f63d8009120 comp rx=0 tx=0).stop 2023-10-06T05:07:32.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.839+0000 7f63e335c700 1 -- 172.21.15.98:0/982899172 shutdown_connections 2023-10-06T05:07:32.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.839+0000 7f63e335c700 1 --2- 172.21.15.98:0/982899172 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63dc0b2620 0x7f63dc0cccb0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:32.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.839+0000 7f63e335c700 1 -- 172.21.15.98:0/982899172 >> 172.21.15.98:0/982899172 conn(0x7f63dc099520 msgr2=0x7f63dc06f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:32.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.839+0000 7f63e335c700 1 -- 172.21.15.98:0/982899172 shutdown_connections 2023-10-06T05:07:32.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.839+0000 7f63e335c700 1 -- 172.21.15.98:0/982899172 wait complete. 2023-10-06T05:07:32.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.840+0000 7f63e335c700 1 Processor -- start 2023-10-06T05:07:32.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.840+0000 7f63e335c700 1 -- start start 2023-10-06T05:07:32.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.841+0000 7f63e335c700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f63dc0b2620 0x7f63dc0b3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:32.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.841+0000 7f63e335c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63dc0b29f0 0x7f63dc0b5880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:32.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.841+0000 7f63e335c700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63dc0cf1b0 0x7f63dc152aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:32.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.841+0000 7f63e335c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63dc0b56c0 con 0x7f63dc0b29f0 2023-10-06T05:07:32.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.841+0000 7f63e335c700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63dc0ca560 con 0x7f63dc0cf1b0 2023-10-06T05:07:32.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.841+0000 7f63e335c700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63dc0c9cc0 con 0x7f63dc0b2620 2023-10-06T05:07:32.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.841+0000 7f63e1b59700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63dc0b29f0 0x7f63dc0b5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:32.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.841+0000 7f63e1b59700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63dc0b29f0 0x7f63dc0b5880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45162/0 (socket says 172.21.15.98:45162) 2023-10-06T05:07:32.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.841+0000 7f63e1b59700 1 -- 172.21.15.98:0/2170374653 learned_addr learned my addr 172.21.15.98:0/2170374653 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:32.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.841+0000 7f63e2b5b700 1 --2- 172.21.15.98:0/2170374653 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63dc0cf1b0 0x7f63dc152aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:32.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.841+0000 7f63e235a700 1 --2- 172.21.15.98:0/2170374653 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f63dc0b2620 0x7f63dc0b3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:32.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.841+0000 7f63e1b59700 1 -- 172.21.15.98:0/2170374653 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f63dc0b2620 msgr2=0x7f63dc0b3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:32.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.841+0000 7f63e1b59700 1 --2- 172.21.15.98:0/2170374653 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f63dc0b2620 0x7f63dc0b3680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:32.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.841+0000 7f63e1b59700 1 -- 172.21.15.98:0/2170374653 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63dc0cf1b0 msgr2=0x7f63dc152aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:32.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.841+0000 7f63e1b59700 1 --2- 172.21.15.98:0/2170374653 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63dc0cf1b0 0x7f63dc152aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:32.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.841+0000 7f63e1b59700 1 -- 172.21.15.98:0/2170374653 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63d80093c0 con 0x7f63dc0b29f0 2023-10-06T05:07:32.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.841+0000 7f63e235a700 1 --2- 172.21.15.98:0/2170374653 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f63dc0b2620 0x7f63dc0b3680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:07:32.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.841+0000 7f63e1b59700 1 --2- 172.21.15.98:0/2170374653 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63dc0b29f0 0x7f63dc0b5880 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f63cc00c790 tx=0x7f63cc007e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:32.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.842+0000 7f63d27fc700 1 -- 172.21.15.98:0/2170374653 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63cc005b30 con 0x7f63dc0b29f0 2023-10-06T05:07:32.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.842+0000 7f63d27fc700 1 -- 172.21.15.98:0/2170374653 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63cc005c90 con 0x7f63dc0b29f0 2023-10-06T05:07:32.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.842+0000 7f63e335c700 1 -- 172.21.15.98:0/2170374653 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63dc0c9fa0 con 0x7f63dc0b29f0 2023-10-06T05:07:32.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.842+0000 7f63d27fc700 1 -- 172.21.15.98:0/2170374653 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63cc007320 con 0x7f63dc0b29f0 2023-10-06T05:07:32.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.842+0000 7f63e335c700 1 -- 172.21.15.98:0/2170374653 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63dc153710 con 0x7f63dc0b29f0 2023-10-06T05:07:32.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.843+0000 7f63e335c700 1 -- 172.21.15.98:0/2170374653 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63dc153930 con 0x7f63dc0b29f0 2023-10-06T05:07:32.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.843+0000 7f63d27fc700 1 -- 172.21.15.98:0/2170374653 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f63cc01b400 con 0x7f63dc0b29f0 2023-10-06T05:07:32.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.843+0000 7f63d27fc700 1 --2- 172.21.15.98:0/2170374653 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f63c0066f30 0x7f63c00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:32.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.843+0000 7f63d27fc700 1 -- 172.21.15.98:0/2170374653 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f63cc084450 con 0x7f63dc0b29f0 2023-10-06T05:07:32.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.843+0000 7f63e235a700 1 --2- 172.21.15.98:0/2170374653 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f63c0066f30 0x7f63c00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:32.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.844+0000 7f63e235a700 1 --2- 172.21.15.98:0/2170374653 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f63c0066f30 0x7f63c00693e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f63d8009340 tx=0x7f63d8016000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:32.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:32.848+0000 7f63d27fc700 1 -- 172.21.15.98:0/2170374653 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f63cc0512f0 con 0x7f63dc0b29f0 2023-10-06T05:07:33.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.031+0000 7f63e335c700 1 -- 172.21.15.98:0/2170374653 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f63dc05f230 con 0x7f63dc0b29f0 2023-10-06T05:07:33.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.031+0000 7f63d27fc700 1 -- 172.21.15.98:0/2170374653 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v29) v1 ==== 104+0+11 (secure 0 0 0) 0x7f63cc054910 con 0x7f63dc0b29f0 2023-10-06T05:07:33.032 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2023-10-06T05:07:33.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.033+0000 7f63e335c700 1 -- 172.21.15.98:0/2170374653 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f63c0066f30 msgr2=0x7f63c00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:33.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.033+0000 7f63e335c700 1 --2- 172.21.15.98:0/2170374653 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f63c0066f30 0x7f63c00693e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f63d8009340 tx=0x7f63d8016000 comp rx=0 tx=0).stop 2023-10-06T05:07:33.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.033+0000 7f63e335c700 1 -- 172.21.15.98:0/2170374653 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63dc0b29f0 msgr2=0x7f63dc0b5880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:33.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.033+0000 7f63e335c700 1 --2- 172.21.15.98:0/2170374653 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63dc0b29f0 0x7f63dc0b5880 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f63cc00c790 tx=0x7f63cc007e40 comp rx=0 tx=0).stop 2023-10-06T05:07:33.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.033+0000 7f63e335c700 1 -- 172.21.15.98:0/2170374653 shutdown_connections 2023-10-06T05:07:33.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.033+0000 7f63e335c700 1 --2- 172.21.15.98:0/2170374653 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63dc0cf1b0 0x7f63dc152aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:33.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.033+0000 7f63e335c700 1 --2- 172.21.15.98:0/2170374653 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f63dc0b2620 0x7f63dc0b3680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:33.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.033+0000 7f63e335c700 1 --2- 172.21.15.98:0/2170374653 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f63c0066f30 0x7f63c00693e0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:33.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.033+0000 7f63e335c700 1 --2- 172.21.15.98:0/2170374653 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63dc0b29f0 0x7f63dc0b5880 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:33.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.033+0000 7f63e335c700 1 -- 172.21.15.98:0/2170374653 >> 172.21.15.98:0/2170374653 conn(0x7f63dc099520 msgr2=0x7f63dc0c6440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:33.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.033+0000 7f63e335c700 1 -- 172.21.15.98:0/2170374653 shutdown_connections 2023-10-06T05:07:33.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.033+0000 7f63e335c700 1 -- 172.21.15.98:0/2170374653 wait complete. 2023-10-06T05:07:33.046 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-06T05:07:33.162 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.161+0000 7f9ab0f6f700 1 Processor -- start 2023-10-06T05:07:33.162 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.161+0000 7f9ab0f6f700 1 -- start start 2023-10-06T05:07:33.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.162+0000 7f9ab0f6f700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9aac0caa40 0x7f9aac160c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:33.165 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.162+0000 7f9ab0f6f700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9aac0684d0 con 0x7f9aac0707d0 2023-10-06T05:07:33.165 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.162+0000 7f9ab0f6f700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9aac0aece0 con 0x7f9aac0c7a20 2023-10-06T05:07:33.165 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.162+0000 7f9ab0f6f700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9aac067f60 con 0x7f9aac0caa40 2023-10-06T05:07:33.165 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.162+0000 7f9aabfff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9aac0caa40 0x7f9aac160c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:33.165 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.162+0000 7f9aabfff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9aac0caa40 0x7f9aac160c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45172/0 (socket says 172.21.15.98:45172) 2023-10-06T05:07:33.165 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.162+0000 7f9aa2ffd700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f9aac0707d0 0x7f9aac0ca330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:57504/0 (socket says 172.21.15.98:57504) 2023-10-06T05:07:33.165 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.162+0000 7f9aabfff700 1 -- 172.21.15.98:0/1675072012 learned_addr learned my addr 172.21.15.98:0/1675072012 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:33.165 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.162+0000 7f9aabfff700 1 -- 172.21.15.98:0/1675072012 >> v1:172.21.15.174:6789/0 conn(0x7f9aac0c7a20 legacy=0x7f9aac068960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:33.165 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.163+0000 7f9aabfff700 1 -- 172.21.15.98:0/1675072012 >> v1:172.21.15.98:6789/0 conn(0x7f9aac0707d0 legacy=0x7f9aac0ca330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:33.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.163+0000 7f9aabfff700 1 -- 172.21.15.98:0/1675072012 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aac0cae40 con 0x7f9aac0caa40 2023-10-06T05:07:33.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.163+0000 7f9aaaffd700 1 -- 172.21.15.98:0/1675072012 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4038162043 0 0) 0x7f9aac0684d0 con 0x7f9aac0707d0 2023-10-06T05:07:33.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.163+0000 7f9aabfff700 1 --2- 172.21.15.98:0/1675072012 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9aac0caa40 0x7f9aac160c40 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f9aa4009f20 tx=0x7f9aa4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4703d7577e8a0ec server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:33.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.163+0000 7f9aaaffd700 1 -- 172.21.15.98:0/1675072012 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aa400e070 con 0x7f9aac0caa40 2023-10-06T05:07:33.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.163+0000 7f9aaaffd700 1 -- 172.21.15.98:0/1675072012 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9aa400ba80 con 0x7f9aac0caa40 2023-10-06T05:07:33.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.163+0000 7f9aaaffd700 1 -- 172.21.15.98:0/1675072012 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aa40092e0 con 0x7f9aac0caa40 2023-10-06T05:07:33.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.163+0000 7f9ab0f6f700 1 -- 172.21.15.98:0/1675072012 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9aac0caa40 msgr2=0x7f9aac160c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:33.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.163+0000 7f9ab0f6f700 1 --2- 172.21.15.98:0/1675072012 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9aac0caa40 0x7f9aac160c40 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f9aa4009f20 tx=0x7f9aa4009840 comp rx=0 tx=0).stop 2023-10-06T05:07:33.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.163+0000 7f9ab0f6f700 1 -- 172.21.15.98:0/1675072012 shutdown_connections 2023-10-06T05:07:33.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.163+0000 7f9ab0f6f700 1 --2- 172.21.15.98:0/1675072012 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9aac0caa40 0x7f9aac160c40 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:33.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.163+0000 7f9ab0f6f700 1 -- 172.21.15.98:0/1675072012 >> 172.21.15.98:0/1675072012 conn(0x7f9aac0cd1e0 msgr2=0x7f9aac0c91d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:33.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.163+0000 7f9ab0f6f700 1 -- 172.21.15.98:0/1675072012 shutdown_connections 2023-10-06T05:07:33.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.163+0000 7f9ab0f6f700 1 -- 172.21.15.98:0/1675072012 wait complete. 2023-10-06T05:07:33.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.164+0000 7f9ab0f6f700 1 Processor -- start 2023-10-06T05:07:33.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.165+0000 7f9ab0f6f700 1 -- start start 2023-10-06T05:07:33.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.165+0000 7f9ab0f6f700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9aac0707d0 0x7f9aac14a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:33.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.165+0000 7f9ab0f6f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9aac0c7a20 0x7f9aac0c4da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:33.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.165+0000 7f9ab0f6f700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9aac0caa40 0x7f9aac0c52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:33.171 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.165+0000 7f9ab0f6f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aac0b1cc0 con 0x7f9aac0c7a20 2023-10-06T05:07:33.171 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.165+0000 7f9ab0f6f700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aac16c6a0 con 0x7f9aac0caa40 2023-10-06T05:07:33.171 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.165+0000 7f9ab0f6f700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9aac16c980 con 0x7f9aac0707d0 2023-10-06T05:07:33.171 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.165+0000 7f9aa2ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9aac0c7a20 0x7f9aac0c4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:33.171 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.165+0000 7f9aa2ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9aac0c7a20 0x7f9aac0c4da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45186/0 (socket says 172.21.15.98:45186) 2023-10-06T05:07:33.171 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.165+0000 7f9aa2ffd700 1 -- 172.21.15.98:0/3803385412 learned_addr learned my addr 172.21.15.98:0/3803385412 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:33.171 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.166+0000 7f9aa2ffd700 1 -- 172.21.15.98:0/3803385412 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9aac0707d0 msgr2=0x7f9aac14a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:33.171 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.166+0000 7f9aa2ffd700 1 --2- 172.21.15.98:0/3803385412 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9aac0707d0 0x7f9aac14a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:33.171 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.166+0000 7f9aa2ffd700 1 -- 172.21.15.98:0/3803385412 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9aac0caa40 msgr2=0x7f9aac0c52e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:33.171 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.166+0000 7f9aa2ffd700 1 --2- 172.21.15.98:0/3803385412 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9aac0caa40 0x7f9aac0c52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:33.171 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.166+0000 7f9aa2ffd700 1 -- 172.21.15.98:0/3803385412 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aa4009ae0 con 0x7f9aac0c7a20 2023-10-06T05:07:33.171 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.166+0000 7f9aa2ffd700 1 --2- 172.21.15.98:0/3803385412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9aac0c7a20 0x7f9aac0c4da0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f9a940011e0 tx=0x7f9a94007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:33.172 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.166+0000 7f9aa3fff700 1 -- 172.21.15.98:0/3803385412 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a94007300 con 0x7f9aac0c7a20 2023-10-06T05:07:33.172 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.166+0000 7f9aa3fff700 1 -- 172.21.15.98:0/3803385412 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a94007460 con 0x7f9aac0c7a20 2023-10-06T05:07:33.172 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.166+0000 7f9ab0f6f700 1 -- 172.21.15.98:0/3803385412 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9aac16cc60 con 0x7f9aac0c7a20 2023-10-06T05:07:33.172 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.166+0000 7f9aa3fff700 1 -- 172.21.15.98:0/3803385412 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a94004830 con 0x7f9aac0c7a20 2023-10-06T05:07:33.172 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.166+0000 7f9ab0f6f700 1 -- 172.21.15.98:0/3803385412 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9aac14ab40 con 0x7f9aac0c7a20 2023-10-06T05:07:33.172 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.167+0000 7f9ab0f6f700 1 -- 172.21.15.98:0/3803385412 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9aac05f230 con 0x7f9aac0c7a20 2023-10-06T05:07:33.172 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.167+0000 7f9aa3fff700 1 -- 172.21.15.98:0/3803385412 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f9a94004be0 con 0x7f9aac0c7a20 2023-10-06T05:07:33.172 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.168+0000 7f9aa3fff700 1 --2- 172.21.15.98:0/3803385412 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f9a90066f30 0x7f9a900693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:33.172 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.168+0000 7f9aa3fff700 1 -- 172.21.15.98:0/3803385412 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f9a94083ec0 con 0x7f9aac0c7a20 2023-10-06T05:07:33.172 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.168+0000 7f9aab7fe700 1 --2- 172.21.15.98:0/3803385412 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f9a90066f30 0x7f9a900693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:33.172 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.168+0000 7f9aab7fe700 1 --2- 172.21.15.98:0/3803385412 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f9a90066f30 0x7f9a900693e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c007fd0 tx=0x7f9a9c007b20 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:33.173 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.172+0000 7f9aa3fff700 1 -- 172.21.15.98:0/3803385412 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9a94050d60 con 0x7f9aac0c7a20 2023-10-06T05:07:33.354 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.353+0000 7f9ab0f6f700 1 -- 172.21.15.98:0/3803385412 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f9aac0aef80 con 0x7f9aac0c7a20 2023-10-06T05:07:33.354 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.353+0000 7f9aa3fff700 1 -- 172.21.15.98:0/3803385412 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v29) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9a94018020 con 0x7f9aac0c7a20 2023-10-06T05:07:33.355 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2023-10-06T05:07:33.357 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.355+0000 7f9ab0f6f700 1 -- 172.21.15.98:0/3803385412 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f9a90066f30 msgr2=0x7f9a900693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:33.358 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.356+0000 7f9ab0f6f700 1 --2- 172.21.15.98:0/3803385412 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f9a90066f30 0x7f9a900693e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c007fd0 tx=0x7f9a9c007b20 comp rx=0 tx=0).stop 2023-10-06T05:07:33.358 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.356+0000 7f9ab0f6f700 1 -- 172.21.15.98:0/3803385412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9aac0c7a20 msgr2=0x7f9aac0c4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:33.358 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.356+0000 7f9ab0f6f700 1 --2- 172.21.15.98:0/3803385412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9aac0c7a20 0x7f9aac0c4da0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f9a940011e0 tx=0x7f9a94007e10 comp rx=0 tx=0).stop 2023-10-06T05:07:33.358 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.356+0000 7f9ab0f6f700 1 -- 172.21.15.98:0/3803385412 shutdown_connections 2023-10-06T05:07:33.358 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.356+0000 7f9ab0f6f700 1 --2- 172.21.15.98:0/3803385412 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9aac0caa40 0x7f9aac0c52e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:33.358 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.356+0000 7f9ab0f6f700 1 --2- 172.21.15.98:0/3803385412 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9aac0707d0 0x7f9aac14a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:33.358 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.356+0000 7f9ab0f6f700 1 --2- 172.21.15.98:0/3803385412 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f9a90066f30 0x7f9a900693e0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:33.358 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.356+0000 7f9ab0f6f700 1 --2- 172.21.15.98:0/3803385412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9aac0c7a20 0x7f9aac0c4da0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:33.358 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.356+0000 7f9ab0f6f700 1 -- 172.21.15.98:0/3803385412 >> 172.21.15.98:0/3803385412 conn(0x7f9aac0cd1e0 msgr2=0x7f9aac1540a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:33.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.356+0000 7f9ab0f6f700 1 -- 172.21.15.98:0/3803385412 shutdown_connections 2023-10-06T05:07:33.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.356+0000 7f9ab0f6f700 1 -- 172.21.15.98:0/3803385412 wait complete. 2023-10-06T05:07:33.369 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-06T05:07:33.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.485+0000 7fa3e2657700 1 Processor -- start 2023-10-06T05:07:33.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.485+0000 7fa3e2657700 1 -- start start 2023-10-06T05:07:33.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.486+0000 7fa3e2657700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fa3dc140cc0 0x7fa3dc0c66e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:33.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.486+0000 7fa3e2657700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa3dc06dab0 0x7fa3dc0695b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:33.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.486+0000 7fa3e2657700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3dc141090 con 0x7fa3dc0c6c20 2023-10-06T05:07:33.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.486+0000 7fa3e2657700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3dc0c7e60 con 0x7fa3dc06dab0 2023-10-06T05:07:33.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.486+0000 7fa3e2657700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3dc06f6e0 con 0x7fa3dc140cc0 2023-10-06T05:07:33.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.486+0000 7fa3e0bf4700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa3dc06dab0 0x7fa3dc0695b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:33.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.486+0000 7fa3e0bf4700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa3dc06dab0 0x7fa3dc0695b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45196/0 (socket says 172.21.15.98:45196) 2023-10-06T05:07:33.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.486+0000 7fa3e0bf4700 1 -- 172.21.15.98:0/2248525457 learned_addr learned my addr 172.21.15.98:0/2248525457 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:33.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.486+0000 7fa3dbfff700 1 --2- 172.21.15.98:0/2248525457 >> v2:172.21.15.174:3300/0 conn(0x7fa3dc140cc0 0x7fa3dc0c66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:33.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.487+0000 7fa3e0bf4700 1 -- 172.21.15.98:0/2248525457 >> v2:172.21.15.174:3300/0 conn(0x7fa3dc140cc0 msgr2=0x7fa3dc0c66e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:33.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.487+0000 7fa3e0bf4700 1 --2- 172.21.15.98:0/2248525457 >> v2:172.21.15.174:3300/0 conn(0x7fa3dc140cc0 0x7fa3dc0c66e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:33.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.487+0000 7fa3e0bf4700 1 -- 172.21.15.98:0/2248525457 >> v1:172.21.15.174:6789/0 conn(0x7fa3dc0c6c20 legacy=0x7fa3dc06d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:33.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.487+0000 7fa3e0bf4700 1 -- 172.21.15.98:0/2248525457 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3dc0c7250 con 0x7fa3dc06dab0 2023-10-06T05:07:33.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.487+0000 7fa3daffd700 1 -- 172.21.15.98:0/2248525457 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3510896361 0 0) 0x7fa3dc141090 con 0x7fa3dc0c6c20 2023-10-06T05:07:33.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.487+0000 7fa3e0bf4700 1 --2- 172.21.15.98:0/2248525457 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3dc06dab0 0x7fa3dc0695b0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fa3dc06ce50 tx=0x7fa3d4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=710504a4bcd582df server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:33.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.487+0000 7fa3daffd700 1 -- 172.21.15.98:0/2248525457 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3d400e070 con 0x7fa3dc06dab0 2023-10-06T05:07:33.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.487+0000 7fa3daffd700 1 -- 172.21.15.98:0/2248525457 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3d400ba20 con 0x7fa3dc06dab0 2023-10-06T05:07:33.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.487+0000 7fa3daffd700 1 -- 172.21.15.98:0/2248525457 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3d40092e0 con 0x7fa3dc06dab0 2023-10-06T05:07:33.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.487+0000 7fa3e2657700 1 -- 172.21.15.98:0/2248525457 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3dc06dab0 msgr2=0x7fa3dc0695b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:33.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.487+0000 7fa3e2657700 1 --2- 172.21.15.98:0/2248525457 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3dc06dab0 0x7fa3dc0695b0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fa3dc06ce50 tx=0x7fa3d4009840 comp rx=0 tx=0).stop 2023-10-06T05:07:33.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.487+0000 7fa3e2657700 1 -- 172.21.15.98:0/2248525457 shutdown_connections 2023-10-06T05:07:33.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.487+0000 7fa3e2657700 1 --2- 172.21.15.98:0/2248525457 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3dc06dab0 0x7fa3dc0695b0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:33.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.487+0000 7fa3e2657700 1 --2- 172.21.15.98:0/2248525457 >> v2:172.21.15.174:3300/0 conn(0x7fa3dc140cc0 0x7fa3dc0c66e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:33.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.487+0000 7fa3e2657700 1 -- 172.21.15.98:0/2248525457 >> 172.21.15.98:0/2248525457 conn(0x7fa3dc0cc0a0 msgr2=0x7fa3dc0cc4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:33.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.487+0000 7fa3e2657700 1 -- 172.21.15.98:0/2248525457 shutdown_connections 2023-10-06T05:07:33.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.487+0000 7fa3e2657700 1 -- 172.21.15.98:0/2248525457 wait complete. 2023-10-06T05:07:33.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.488+0000 7fa3e2657700 1 Processor -- start 2023-10-06T05:07:33.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.489+0000 7fa3e2657700 1 -- start start 2023-10-06T05:07:33.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.489+0000 7fa3e2657700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa3dc06dab0 0x7fa3dc13b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:33.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.489+0000 7fa3e2657700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3dc0c6c20 0x7fa3dc169f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:33.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.489+0000 7fa3e2657700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa3dc140cc0 0x7fa3dc0afd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:33.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.489+0000 7fa3e2657700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3dc0b1d00 con 0x7fa3dc0c6c20 2023-10-06T05:07:33.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.489+0000 7fa3e2657700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3dc0af8d0 con 0x7fa3dc06dab0 2023-10-06T05:07:33.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.489+0000 7fa3e2657700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3dc0b1740 con 0x7fa3dc140cc0 2023-10-06T05:07:33.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.490+0000 7fa3db7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3dc0c6c20 0x7fa3dc169f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:33.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.490+0000 7fa3db7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3dc0c6c20 0x7fa3dc169f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45208/0 (socket says 172.21.15.98:45208) 2023-10-06T05:07:33.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.490+0000 7fa3db7fe700 1 -- 172.21.15.98:0/1860982274 learned_addr learned my addr 172.21.15.98:0/1860982274 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:33.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.490+0000 7fa3dbfff700 1 --2- 172.21.15.98:0/1860982274 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa3dc06dab0 0x7fa3dc13b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:33.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.490+0000 7fa3e0bf4700 1 --2- 172.21.15.98:0/1860982274 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa3dc140cc0 0x7fa3dc0afd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:33.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.490+0000 7fa3db7fe700 1 -- 172.21.15.98:0/1860982274 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa3dc140cc0 msgr2=0x7fa3dc0afd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:33.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.490+0000 7fa3db7fe700 1 --2- 172.21.15.98:0/1860982274 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa3dc140cc0 0x7fa3dc0afd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:33.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.490+0000 7fa3db7fe700 1 -- 172.21.15.98:0/1860982274 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa3dc06dab0 msgr2=0x7fa3dc13b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:33.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.490+0000 7fa3db7fe700 1 --2- 172.21.15.98:0/1860982274 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa3dc06dab0 0x7fa3dc13b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:33.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.490+0000 7fa3db7fe700 1 -- 172.21.15.98:0/1860982274 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3d4009ae0 con 0x7fa3dc0c6c20 2023-10-06T05:07:33.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.490+0000 7fa3db7fe700 1 --2- 172.21.15.98:0/1860982274 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3dc0c6c20 0x7fa3dc169f10 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fa3d000c7e0 tx=0x7fa3d0007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:33.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.490+0000 7fa3c3fff700 1 -- 172.21.15.98:0/1860982274 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3d0011590 con 0x7fa3dc0c6c20 2023-10-06T05:07:33.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.490+0000 7fa3c3fff700 1 -- 172.21.15.98:0/1860982274 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3d0011710 con 0x7fa3dc0c6c20 2023-10-06T05:07:33.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.490+0000 7fa3c3fff700 1 -- 172.21.15.98:0/1860982274 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3d0011ad0 con 0x7fa3dc0c6c20 2023-10-06T05:07:33.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.490+0000 7fa3e2657700 1 -- 172.21.15.98:0/1860982274 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3dc0b1a20 con 0x7fa3dc0c6c20 2023-10-06T05:07:33.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.490+0000 7fa3e2657700 1 -- 172.21.15.98:0/1860982274 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3dc16a450 con 0x7fa3dc0c6c20 2023-10-06T05:07:33.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.491+0000 7fa3e2657700 1 -- 172.21.15.98:0/1860982274 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3dc064a90 con 0x7fa3dc0c6c20 2023-10-06T05:07:33.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.491+0000 7fa3c3fff700 1 -- 172.21.15.98:0/1860982274 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fa3d001b3b0 con 0x7fa3dc0c6c20 2023-10-06T05:07:33.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.492+0000 7fa3c3fff700 1 --2- 172.21.15.98:0/1860982274 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fa3c8066f30 0x7fa3c80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:33.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.492+0000 7fa3c3fff700 1 -- 172.21.15.98:0/1860982274 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fa3d0083ed0 con 0x7fa3dc0c6c20 2023-10-06T05:07:33.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.492+0000 7fa3dbfff700 1 --2- 172.21.15.98:0/1860982274 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fa3c8066f30 0x7fa3c80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:33.497 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.492+0000 7fa3dbfff700 1 --2- 172.21.15.98:0/1860982274 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fa3c8066f30 0x7fa3c80693e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc009d10 tx=0x7fa3cc0091a0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:33.497 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.496+0000 7fa3c3fff700 1 -- 172.21.15.98:0/1860982274 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa3d0050cf0 con 0x7fa3dc0c6c20 2023-10-06T05:07:33.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.682+0000 7fa3e2657700 1 -- 172.21.15.98:0/1860982274 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fa3dc0afa10 con 0x7fa3dc0c6c20 2023-10-06T05:07:33.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.683+0000 7fa3c3fff700 1 -- 172.21.15.98:0/1860982274 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v29) v1 ==== 103+0+11 (secure 0 0 0) 0x7fa3d0017070 con 0x7fa3dc0c6c20 2023-10-06T05:07:33.685 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2023-10-06T05:07:33.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.685+0000 7fa3e2657700 1 -- 172.21.15.98:0/1860982274 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fa3c8066f30 msgr2=0x7fa3c80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:33.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.685+0000 7fa3e2657700 1 --2- 172.21.15.98:0/1860982274 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fa3c8066f30 0x7fa3c80693e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc009d10 tx=0x7fa3cc0091a0 comp rx=0 tx=0).stop 2023-10-06T05:07:33.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.685+0000 7fa3e2657700 1 -- 172.21.15.98:0/1860982274 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3dc0c6c20 msgr2=0x7fa3dc169f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:33.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.685+0000 7fa3e2657700 1 --2- 172.21.15.98:0/1860982274 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3dc0c6c20 0x7fa3dc169f10 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fa3d000c7e0 tx=0x7fa3d0007df0 comp rx=0 tx=0).stop 2023-10-06T05:07:33.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.685+0000 7fa3e2657700 1 -- 172.21.15.98:0/1860982274 shutdown_connections 2023-10-06T05:07:33.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.685+0000 7fa3e2657700 1 --2- 172.21.15.98:0/1860982274 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa3dc140cc0 0x7fa3dc0afd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:33.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.685+0000 7fa3e2657700 1 --2- 172.21.15.98:0/1860982274 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa3dc06dab0 0x7fa3dc13b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:33.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.685+0000 7fa3e2657700 1 --2- 172.21.15.98:0/1860982274 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fa3c8066f30 0x7fa3c80693e0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:33.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.685+0000 7fa3e2657700 1 --2- 172.21.15.98:0/1860982274 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa3dc0c6c20 0x7fa3dc169f10 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:33.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.685+0000 7fa3e2657700 1 -- 172.21.15.98:0/1860982274 >> 172.21.15.98:0/1860982274 conn(0x7fa3dc0cc0a0 msgr2=0x7fa3dc0c5160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:33.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.685+0000 7fa3e2657700 1 -- 172.21.15.98:0/1860982274 shutdown_connections 2023-10-06T05:07:33.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.685+0000 7fa3e2657700 1 -- 172.21.15.98:0/1860982274 wait complete. 2023-10-06T05:07:33.698 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T05:07:33.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.814+0000 7f2823ee1700 1 Processor -- start 2023-10-06T05:07:33.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.814+0000 7f2823ee1700 1 -- start start 2023-10-06T05:07:33.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.815+0000 7f2823ee1700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f281c159560 0x7f281c0aa530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:33.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.815+0000 7f2823ee1700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f281c0b49a0 0x7f281c0aaa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:33.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.815+0000 7f2823ee1700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f281c0b0760 0x7f281c0b3890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:33.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.815+0000 7f2823ee1700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f281c0b0c60 con 0x7f281c0b0760 2023-10-06T05:07:33.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.815+0000 7f2823ee1700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f281c0b0f40 con 0x7f281c159560 2023-10-06T05:07:33.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.815+0000 7f2823ee1700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f281c0b1220 con 0x7f281c0b49a0 2023-10-06T05:07:33.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.815+0000 7f282247e700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f281c0b0760 0x7f281c0b3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:33.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.815+0000 7f282247e700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f281c0b0760 0x7f281c0b3890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45222/0 (socket says 172.21.15.98:45222) 2023-10-06T05:07:33.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.815+0000 7f282247e700 1 -- 172.21.15.98:0/844332205 learned_addr learned my addr 172.21.15.98:0/844332205 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:33.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.815+0000 7f282147c700 1 --2- 172.21.15.98:0/844332205 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f281c0b49a0 0x7f281c0aaa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:33.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.815+0000 7f2821c7d700 1 --2- 172.21.15.98:0/844332205 >> v2:172.21.15.174:3300/0 conn(0x7f281c159560 0x7f281c0aa530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:33.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.816+0000 7f282247e700 1 -- 172.21.15.98:0/844332205 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f281c0b49a0 msgr2=0x7f281c0aaa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:33.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.816+0000 7f282247e700 1 --2- 172.21.15.98:0/844332205 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f281c0b49a0 0x7f281c0aaa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:33.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.816+0000 7f282247e700 1 -- 172.21.15.98:0/844332205 >> v2:172.21.15.174:3300/0 conn(0x7f281c159560 msgr2=0x7f281c0aa530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:33.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.816+0000 7f282247e700 1 --2- 172.21.15.98:0/844332205 >> v2:172.21.15.174:3300/0 conn(0x7f281c159560 0x7f281c0aa530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:33.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.816+0000 7f282247e700 1 -- 172.21.15.98:0/844332205 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f281c0b1500 con 0x7f281c0b0760 2023-10-06T05:07:33.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.816+0000 7f282247e700 1 --2- 172.21.15.98:0/844332205 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f281c0b0760 0x7f281c0b3890 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f281c088c80 tx=0x7f2814009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c79050e4893578e server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:33.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.816+0000 7f2820c7b700 1 -- 172.21.15.98:0/844332205 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f281400e070 con 0x7f281c0b0760 2023-10-06T05:07:33.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.816+0000 7f2820c7b700 1 -- 172.21.15.98:0/844332205 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f281400ba50 con 0x7f281c0b0760 2023-10-06T05:07:33.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.816+0000 7f2820c7b700 1 -- 172.21.15.98:0/844332205 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28140092e0 con 0x7f281c0b0760 2023-10-06T05:07:33.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.816+0000 7f2823ee1700 1 -- 172.21.15.98:0/844332205 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f281c0b0760 msgr2=0x7f281c0b3890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:33.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.816+0000 7f2823ee1700 1 --2- 172.21.15.98:0/844332205 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f281c0b0760 0x7f281c0b3890 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f281c088c80 tx=0x7f2814009840 comp rx=0 tx=0).stop 2023-10-06T05:07:33.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.816+0000 7f2823ee1700 1 -- 172.21.15.98:0/844332205 shutdown_connections 2023-10-06T05:07:33.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.816+0000 7f2823ee1700 1 --2- 172.21.15.98:0/844332205 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f281c0b0760 0x7f281c0b3890 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:33.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.816+0000 7f2823ee1700 1 --2- 172.21.15.98:0/844332205 >> v2:172.21.15.174:3300/0 conn(0x7f281c159560 0x7f281c0aa530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:33.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.816+0000 7f2823ee1700 1 --2- 172.21.15.98:0/844332205 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f281c0b49a0 0x7f281c0aaa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:33.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.816+0000 7f2823ee1700 1 -- 172.21.15.98:0/844332205 >> 172.21.15.98:0/844332205 conn(0x7f281c0997f0 msgr2=0x7f281c06f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:33.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.816+0000 7f2823ee1700 1 -- 172.21.15.98:0/844332205 shutdown_connections 2023-10-06T05:07:33.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.817+0000 7f2823ee1700 1 -- 172.21.15.98:0/844332205 wait complete. 2023-10-06T05:07:33.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.818+0000 7f2823ee1700 1 Processor -- start 2023-10-06T05:07:33.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.818+0000 7f2823ee1700 1 -- start start 2023-10-06T05:07:33.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.818+0000 7f2823ee1700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f281c0b0760 0x7f281c09b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:33.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.818+0000 7f2823ee1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f281c0b49a0 0x7f281c16d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:33.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.818+0000 7f2823ee1700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f281c159560 0x7f281c09d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:33.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.818+0000 7f2823ee1700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f281c16d4e0 con 0x7f281c0b49a0 2023-10-06T05:07:33.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.818+0000 7f2823ee1700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f281c16deb0 con 0x7f281c0b0760 2023-10-06T05:07:33.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.818+0000 7f2823ee1700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f281c09bd30 con 0x7f281c159560 2023-10-06T05:07:33.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.818+0000 7f282147c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f281c0b49a0 0x7f281c16d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:33.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.818+0000 7f282147c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f281c0b49a0 0x7f281c16d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45226/0 (socket says 172.21.15.98:45226) 2023-10-06T05:07:33.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.818+0000 7f282147c700 1 -- 172.21.15.98:0/569160297 learned_addr learned my addr 172.21.15.98:0/569160297 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:33.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.818+0000 7f2821c7d700 1 --2- 172.21.15.98:0/569160297 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f281c0b0760 0x7f281c09b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:33.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.818+0000 7f282247e700 1 --2- 172.21.15.98:0/569160297 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f281c159560 0x7f281c09d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:33.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.819+0000 7f282147c700 1 -- 172.21.15.98:0/569160297 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f281c159560 msgr2=0x7f281c09d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:33.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.819+0000 7f282147c700 1 --2- 172.21.15.98:0/569160297 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f281c159560 0x7f281c09d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:33.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.819+0000 7f282147c700 1 -- 172.21.15.98:0/569160297 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f281c0b0760 msgr2=0x7f281c09b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:33.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.819+0000 7f282147c700 1 --2- 172.21.15.98:0/569160297 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f281c0b0760 0x7f281c09b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:33.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.819+0000 7f282147c700 1 -- 172.21.15.98:0/569160297 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2814009ae0 con 0x7f281c0b49a0 2023-10-06T05:07:33.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.819+0000 7f282247e700 1 --2- 172.21.15.98:0/569160297 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f281c159560 0x7f281c09d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:07:33.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.819+0000 7f282147c700 1 --2- 172.21.15.98:0/569160297 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f281c0b49a0 0x7f281c16d970 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f280c00d890 tx=0x7f280c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:33.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.819+0000 7f2811ffb700 1 -- 172.21.15.98:0/569160297 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f280c0125f0 con 0x7f281c0b49a0 2023-10-06T05:07:33.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.819+0000 7f2811ffb700 1 -- 172.21.15.98:0/569160297 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f280c012750 con 0x7f281c0b49a0 2023-10-06T05:07:33.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.819+0000 7f2823ee1700 1 -- 172.21.15.98:0/569160297 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f281c09c010 con 0x7f281c0b49a0 2023-10-06T05:07:33.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.819+0000 7f2811ffb700 1 -- 172.21.15.98:0/569160297 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f280c012ad0 con 0x7f281c0b49a0 2023-10-06T05:07:33.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.819+0000 7f2823ee1700 1 -- 172.21.15.98:0/569160297 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f281c16d170 con 0x7f281c0b49a0 2023-10-06T05:07:33.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.820+0000 7f2823ee1700 1 -- 172.21.15.98:0/569160297 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f281c16cfd0 con 0x7f281c0b49a0 2023-10-06T05:07:33.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.820+0000 7f2811ffb700 1 -- 172.21.15.98:0/569160297 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f280c01e460 con 0x7f281c0b49a0 2023-10-06T05:07:33.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.821+0000 7f2811ffb700 1 --2- 172.21.15.98:0/569160297 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f2800066f30 0x7f28000693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:33.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.821+0000 7f2811ffb700 1 -- 172.21.15.98:0/569160297 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f280c084fa0 con 0x7f281c0b49a0 2023-10-06T05:07:33.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.821+0000 7f2821c7d700 1 --2- 172.21.15.98:0/569160297 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f2800066f30 0x7f28000693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:33.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.821+0000 7f2821c7d700 1 --2- 172.21.15.98:0/569160297 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f2800066f30 0x7f28000693e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f281c15a960 tx=0x7f2818008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:33.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:33.825+0000 7f2811ffb700 1 -- 172.21.15.98:0/569160297 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f281c16cfd0 con 0x7f281c0b49a0 2023-10-06T05:07:34.010 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.009+0000 7f2823ee1700 1 -- 172.21.15.98:0/569160297 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f281c16cfd0 con 0x7f281c0b49a0 2023-10-06T05:07:34.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.010+0000 7f2811ffb700 1 -- 172.21.15.98:0/569160297 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+18707 (secure 0 0 0) 0x7f281c16cfd0 con 0x7f281c0b49a0 2023-10-06T05:07:34.012 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:07:34.012 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":29,"fsid":"f50c5df5-cad5-47a4-9bb1-8c791d4311ec","created":"2023-10-06T05:06:32.971175+0000","modified":"2023-10-06T05:07:02.131672+0000","last_up_change":"2023-10-06T05:06:44.016489+0000","last_in_change":"2023-10-06T05:06:35.544511+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-06T05:06:44.559671+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-06T05:06:51.432161+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-10-06T05:06:55.589057+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-10-06T05:06:56.041687+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-10-06T05:06:59.754099+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d9495d95-da75-4c3b-8927-e9bae6a8b093","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":90627},{"type":"v1","addr":"172.21.15.98:6801","nonce":90627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":90627},{"type":"v1","addr":"172.21.15.98:6803","nonce":90627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":90627},{"type":"v1","addr":"172.21.15.98:6807","nonce":90627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":90627},{"type":"v1","addr":"172.21.15.98:6805","nonce":90627}]},"public_addr":"172.21.15.98:6801/90627","cluster_addr":"172.21.15.98:6803/90627","heartbeat_back_addr":"172.21.15.98:6807/90627","heartbeat_front_addr":"172.21.15.98:6805/90627","state":["exists","up"]},{"osd":1,"uuid":"f25ba5be-d65e-4707-a134-154092deb754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":90628},{"type":"v1","addr":"172.21.15.98:6809","nonce":90628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":90628},{"type":"v1","addr":"172.21.15.98:6811","nonce":90628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":90628},{"type":"v1","addr":"172.21.15.98:6815","nonce":90628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":90628},{"type":"v1","addr":"172.21.15.98:6813","nonce":90628}]},"public_addr":"172.21.15.98:6809/90628","cluster_addr":"172.21.15.98:6811/90628","heartbeat_back_addr":"172.21.15.98:6815/90628","heartbeat_front_addr":"172.21.15.98:6813/90628","state":["exists","up"]},{"osd":2,"uuid":"5ab9e028-945c-43e2-8c95-3b0b4cf38e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":90662},{"type":"v1","addr":"172.21.15.98:6818","nonce":90662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":90662},{"type":"v1","addr":"172.21.15.98:6822","nonce":90662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":90662},{"type":"v1","addr":"172.21.15.98:6830","nonce":90662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":90662},{"type":"v1","addr":"172.21.15.98:6826","nonce":90662}]},"public_addr":"172.21.15.98:6818/90662","cluster_addr":"172.21.15.98:6822/90662","heartbeat_back_addr":"172.21.15.98:6830/90662","heartbeat_front_addr":"172.21.15.98:6826/90662","state":["exists","up"]},{"osd":3,"uuid":"3bb803ff-1738-46bf-be62-2267bf378822","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6817","nonce":90663},{"type":"v1","addr":"172.21.15.98:6819","nonce":90663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6821","nonce":90663},{"type":"v1","addr":"172.21.15.98:6823","nonce":90663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6829","nonce":90663},{"type":"v1","addr":"172.21.15.98:6831","nonce":90663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6825","nonce":90663},{"type":"v1","addr":"172.21.15.98:6827","nonce":90663}]},"public_addr":"172.21.15.98:6819/90663","cluster_addr":"172.21.15.98:6823/90663","heartbeat_back_addr":"172.21.15.98:6831/90663","heartbeat_front_addr":"172.21.15.98:6827/90663","state":["exists","up"]},{"osd":4,"uuid":"7d57a4c9-6462-43f6-b64a-7f9e9b4353d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":90847},{"type":"v1","addr":"172.21.15.174:6801","nonce":90847}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":90847},{"type":"v1","addr":"172.21.15.174:6803","nonce":90847}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":90847},{"type":"v1","addr":"172.21.15.174:6807","nonce":90847}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":90847},{"type":"v1","addr":"172.21.15.174:6805","nonce":90847}]},"public_addr":"172.21.15.174:6801/90847","cluster_addr":"172.21.15.174:6803/90847","heartbeat_back_addr":"172.21.15.174:6807/90847","heartbeat_front_addr":"172.21.15.174:6805/90847","state":["exists","up"]},{"osd":5,"uuid":"83287ea4-e44d-4af9-bc77-36d47c60d4a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":90848},{"type":"v1","addr":"172.21.15.174:6809","nonce":90848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":90848},{"type":"v1","addr":"172.21.15.174:6811","nonce":90848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":90848},{"type":"v1","addr":"172.21.15.174:6815","nonce":90848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":90848},{"type":"v1","addr":"172.21.15.174:6813","nonce":90848}]},"public_addr":"172.21.15.174:6809/90848","cluster_addr":"172.21.15.174:6811/90848","heartbeat_back_addr":"172.21.15.174:6815/90848","heartbeat_front_addr":"172.21.15.174:6813/90848","state":["exists","up"]},{"osd":6,"uuid":"496a6272-892e-43c4-a145-206623c22f99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":90877},{"type":"v1","addr":"172.21.15.174:6817","nonce":90877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":90877},{"type":"v1","addr":"172.21.15.174:6819","nonce":90877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":90877},{"type":"v1","addr":"172.21.15.174:6823","nonce":90877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":90877},{"type":"v1","addr":"172.21.15.174:6821","nonce":90877}]},"public_addr":"172.21.15.174:6817/90877","cluster_addr":"172.21.15.174:6819/90877","heartbeat_back_addr":"172.21.15.174:6823/90877","heartbeat_front_addr":"172.21.15.174:6821/90877","state":["exists","up"]},{"osd":7,"uuid":"49967e34-8b07-4bab-a13e-0c16d6a0f3a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":90878},{"type":"v1","addr":"172.21.15.174:6825","nonce":90878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":90878},{"type":"v1","addr":"172.21.15.174:6827","nonce":90878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":90878},{"type":"v1","addr":"172.21.15.174:6831","nonce":90878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":90878},{"type":"v1","addr":"172.21.15.174:6829","nonce":90878}]},"public_addr":"172.21.15.174:6825/90878","cluster_addr":"172.21.15.174:6827/90878","heartbeat_back_addr":"172.21.15.174:6831/90878","heartbeat_front_addr":"172.21.15.174:6829/90878","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.446479+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.514266+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.794108+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.810894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.731919+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.717756+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.737435+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.508682+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T05:07:34.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.012+0000 7f2823ee1700 1 -- 172.21.15.98:0/569160297 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f2800066f30 msgr2=0x7f28000693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:34.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.012+0000 7f2823ee1700 1 --2- 172.21.15.98:0/569160297 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f2800066f30 0x7f28000693e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f281c15a960 tx=0x7f2818008040 comp rx=0 tx=0).stop 2023-10-06T05:07:34.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.012+0000 7f2823ee1700 1 -- 172.21.15.98:0/569160297 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f281c0b49a0 msgr2=0x7f281c16d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:34.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.012+0000 7f2823ee1700 1 --2- 172.21.15.98:0/569160297 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f281c0b49a0 0x7f281c16d970 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f280c00d890 tx=0x7f280c008040 comp rx=0 tx=0).stop 2023-10-06T05:07:34.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.012+0000 7f2823ee1700 1 -- 172.21.15.98:0/569160297 shutdown_connections 2023-10-06T05:07:34.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.012+0000 7f2823ee1700 1 --2- 172.21.15.98:0/569160297 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f281c159560 0x7f281c09d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:34.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.012+0000 7f2823ee1700 1 --2- 172.21.15.98:0/569160297 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f281c0b0760 0x7f281c09b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:34.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.012+0000 7f2823ee1700 1 --2- 172.21.15.98:0/569160297 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f2800066f30 0x7f28000693e0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:34.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.012+0000 7f2823ee1700 1 --2- 172.21.15.98:0/569160297 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f281c0b49a0 0x7f281c16d970 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:34.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.012+0000 7f2823ee1700 1 -- 172.21.15.98:0/569160297 >> 172.21.15.98:0/569160297 conn(0x7f281c0997f0 msgr2=0x7f281c0b3230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:34.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.012+0000 7f2823ee1700 1 -- 172.21.15.98:0/569160297 shutdown_connections 2023-10-06T05:07:34.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.012+0000 7f2823ee1700 1 -- 172.21.15.98:0/569160297 wait complete. 2023-10-06T05:07:34.026 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-06T05:06:44.559671+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-10-06T05:06:51.432161+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-06T05:06:55.589057+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-10-06T05:06:56.041687+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-06T05:06:59.754099+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '29', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-06T05:07:34.027 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-06T05:07:34.144 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.143+0000 7f038eaf5700 1 Processor -- start 2023-10-06T05:07:34.145 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.143+0000 7f038eaf5700 1 -- start start 2023-10-06T05:07:34.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.144+0000 7f038eaf5700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f0388159840 0x7f03880c6170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:34.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.144+0000 7f038eaf5700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f03880cf090 0x7f03880c66b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:34.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.144+0000 7f038eaf5700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03880cbb80 con 0x7f0388159470 2023-10-06T05:07:34.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.144+0000 7f038eaf5700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03880c6f00 con 0x7f03880cf090 2023-10-06T05:07:34.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.144+0000 7f038eaf5700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03880cb530 con 0x7f0388159840 2023-10-06T05:07:34.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.144+0000 7f038d092700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f03880cf090 0x7f03880c66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:34.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.144+0000 7f038c891700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f0388159470 0x7f03880ce980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:57512/0 (socket says 172.21.15.98:57512) 2023-10-06T05:07:34.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.144+0000 7f038c891700 1 -- 172.21.15.98:0/2466665858 learned_addr learned my addr 172.21.15.98:0/2466665858 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:07:34.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.144+0000 7f0387fff700 1 --2- 172.21.15.98:0/2466665858 >> v2:172.21.15.174:3300/0 conn(0x7f0388159840 0x7f03880c6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:34.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.145+0000 7f038d092700 1 -- 172.21.15.98:0/2466665858 >> v2:172.21.15.174:3300/0 conn(0x7f0388159840 msgr2=0x7f03880c6170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:34.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.145+0000 7f038d092700 1 --2- 172.21.15.98:0/2466665858 >> v2:172.21.15.174:3300/0 conn(0x7f0388159840 0x7f03880c6170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:34.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.145+0000 7f038d092700 1 -- 172.21.15.98:0/2466665858 >> v1:172.21.15.98:6789/0 conn(0x7f0388159470 legacy=0x7f03880ce980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:34.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.145+0000 7f038d092700 1 -- 172.21.15.98:0/2466665858 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03880cb810 con 0x7f03880cf090 2023-10-06T05:07:34.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.145+0000 7f03877fe700 1 -- 172.21.15.98:0/2466665858 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2080141049 0 0) 0x7f03880cbb80 con 0x7f0388159470 2023-10-06T05:07:34.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.145+0000 7f0387fff700 1 --2- 172.21.15.98:0/2466665858 >> v2:172.21.15.174:3300/0 conn(0x7f0388159840 0x7f03880c6170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:07:34.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.145+0000 7f038d092700 1 --2- 172.21.15.98:0/2466665858 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03880cf090 0x7f03880c66b0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f0388089680 tx=0x7f03800091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f561291fa636c2e server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:34.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.145+0000 7f03877fe700 1 -- 172.21.15.98:0/2466665858 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f038000b6b0 con 0x7f03880cf090 2023-10-06T05:07:34.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.145+0000 7f03877fe700 1 -- 172.21.15.98:0/2466665858 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f038000b810 con 0x7f03880cf090 2023-10-06T05:07:34.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.145+0000 7f03877fe700 1 -- 172.21.15.98:0/2466665858 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f038000bbb0 con 0x7f03880cf090 2023-10-06T05:07:34.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.145+0000 7f038eaf5700 1 -- 172.21.15.98:0/2466665858 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03880cf090 msgr2=0x7f03880c66b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:34.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.145+0000 7f038eaf5700 1 --2- 172.21.15.98:0/2466665858 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03880cf090 0x7f03880c66b0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f0388089680 tx=0x7f03800091a0 comp rx=0 tx=0).stop 2023-10-06T05:07:34.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.145+0000 7f038eaf5700 1 -- 172.21.15.98:0/2466665858 shutdown_connections 2023-10-06T05:07:34.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.145+0000 7f038eaf5700 1 --2- 172.21.15.98:0/2466665858 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03880cf090 0x7f03880c66b0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:34.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.145+0000 7f038eaf5700 1 --2- 172.21.15.98:0/2466665858 >> v2:172.21.15.174:3300/0 conn(0x7f0388159840 0x7f03880c6170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:34.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.145+0000 7f038eaf5700 1 -- 172.21.15.98:0/2466665858 >> 172.21.15.98:0/2466665858 conn(0x7f03880996e0 msgr2=0x7f038806f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:34.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.145+0000 7f038eaf5700 1 -- 172.21.15.98:0/2466665858 shutdown_connections 2023-10-06T05:07:34.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.145+0000 7f038eaf5700 1 -- 172.21.15.98:0/2466665858 wait complete. 2023-10-06T05:07:34.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.146+0000 7f038eaf5700 1 Processor -- start 2023-10-06T05:07:34.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.147+0000 7f038eaf5700 1 -- start start 2023-10-06T05:07:34.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.147+0000 7f038eaf5700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f03880cf090 0x7f038814afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:34.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.147+0000 7f038eaf5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0388159470 0x7f03880c9550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:34.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.147+0000 7f038eaf5700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0388159840 0x7f03880c9a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:34.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.147+0000 7f038eaf5700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f038816a680 con 0x7f0388159470 2023-10-06T05:07:34.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.147+0000 7f038eaf5700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03880c83d0 con 0x7f0388159840 2023-10-06T05:07:34.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.147+0000 7f038eaf5700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03880ca0c0 con 0x7f03880cf090 2023-10-06T05:07:34.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.147+0000 7f0387fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0388159470 0x7f03880c9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:34.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.148+0000 7f0387fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0388159470 0x7f03880c9550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45234/0 (socket says 172.21.15.98:45234) 2023-10-06T05:07:34.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.148+0000 7f0387fff700 1 -- 172.21.15.98:0/2393933412 learned_addr learned my addr 172.21.15.98:0/2393933412 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:34.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.148+0000 7f038d092700 1 --2- 172.21.15.98:0/2393933412 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0388159840 0x7f03880c9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:34.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.148+0000 7f038c891700 1 --2- 172.21.15.98:0/2393933412 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f03880cf090 0x7f038814afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:34.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.148+0000 7f0387fff700 1 -- 172.21.15.98:0/2393933412 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f03880cf090 msgr2=0x7f038814afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:34.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.148+0000 7f0387fff700 1 --2- 172.21.15.98:0/2393933412 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f03880cf090 0x7f038814afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:34.157 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.148+0000 7f0387fff700 1 -- 172.21.15.98:0/2393933412 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0388159840 msgr2=0x7f03880c9a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:34.157 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.148+0000 7f0387fff700 1 --2- 172.21.15.98:0/2393933412 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0388159840 0x7f03880c9a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:34.157 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.148+0000 7f0387fff700 1 -- 172.21.15.98:0/2393933412 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0380009420 con 0x7f0388159470 2023-10-06T05:07:34.157 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.148+0000 7f0387fff700 1 --2- 172.21.15.98:0/2393933412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0388159470 0x7f03880c9550 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f037c00d860 tx=0x7f037c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:34.157 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.148+0000 7f0384ff9700 1 -- 172.21.15.98:0/2393933412 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f037c0125f0 con 0x7f0388159470 2023-10-06T05:07:34.157 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.148+0000 7f0384ff9700 1 -- 172.21.15.98:0/2393933412 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f037c012750 con 0x7f0388159470 2023-10-06T05:07:34.157 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.148+0000 7f038eaf5700 1 -- 172.21.15.98:0/2393933412 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f038816a030 con 0x7f0388159470 2023-10-06T05:07:34.157 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.148+0000 7f0384ff9700 1 -- 172.21.15.98:0/2393933412 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f037c012af0 con 0x7f0388159470 2023-10-06T05:07:34.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.148+0000 7f038eaf5700 1 -- 172.21.15.98:0/2393933412 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0388169c80 con 0x7f0388159470 2023-10-06T05:07:34.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.149+0000 7f038eaf5700 1 -- 172.21.15.98:0/2393933412 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03880c5550 con 0x7f0388159470 2023-10-06T05:07:34.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.149+0000 7f0384ff9700 1 -- 172.21.15.98:0/2393933412 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f037c014430 con 0x7f0388159470 2023-10-06T05:07:34.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.150+0000 7f0384ff9700 1 --2- 172.21.15.98:0/2393933412 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f0374066f30 0x7f03740693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:34.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.150+0000 7f0384ff9700 1 -- 172.21.15.98:0/2393933412 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f037c084d50 con 0x7f0388159470 2023-10-06T05:07:34.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.150+0000 7f038c891700 1 --2- 172.21.15.98:0/2393933412 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f0374066f30 0x7f03740693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:34.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.150+0000 7f038c891700 1 --2- 172.21.15.98:0/2393933412 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f0374066f30 0x7f03740693e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f0378007fd0 tx=0x7f0378009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:34.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.154+0000 7f0384ff9700 1 -- 172.21.15.98:0/2393933412 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f037c088030 con 0x7f0388159470 2023-10-06T05:07:34.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.337+0000 7f038eaf5700 1 -- 172.21.15.98:0/2393933412 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0388064a90 con 0x7f0388159470 2023-10-06T05:07:34.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.338+0000 7f0384ff9700 1 -- 172.21.15.98:0/2393933412 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v29) v1 ==== 93+0+10 (secure 0 0 0) 0x7f037c051b70 con 0x7f0388159470 2023-10-06T05:07:34.339 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2023-10-06T05:07:34.341 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.340+0000 7f038eaf5700 1 -- 172.21.15.98:0/2393933412 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f0374066f30 msgr2=0x7f03740693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:34.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.340+0000 7f038eaf5700 1 --2- 172.21.15.98:0/2393933412 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f0374066f30 0x7f03740693e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f0378007fd0 tx=0x7f0378009040 comp rx=0 tx=0).stop 2023-10-06T05:07:34.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.340+0000 7f038eaf5700 1 -- 172.21.15.98:0/2393933412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0388159470 msgr2=0x7f03880c9550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:34.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.340+0000 7f038eaf5700 1 --2- 172.21.15.98:0/2393933412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0388159470 0x7f03880c9550 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f037c00d860 tx=0x7f037c008040 comp rx=0 tx=0).stop 2023-10-06T05:07:34.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.340+0000 7f038eaf5700 1 -- 172.21.15.98:0/2393933412 shutdown_connections 2023-10-06T05:07:34.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.340+0000 7f038eaf5700 1 --2- 172.21.15.98:0/2393933412 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0388159840 0x7f03880c9a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:34.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.340+0000 7f038eaf5700 1 --2- 172.21.15.98:0/2393933412 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f03880cf090 0x7f038814afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:34.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.340+0000 7f038eaf5700 1 --2- 172.21.15.98:0/2393933412 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f0374066f30 0x7f03740693e0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:34.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.340+0000 7f038eaf5700 1 --2- 172.21.15.98:0/2393933412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0388159470 0x7f03880c9550 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:34.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.340+0000 7f038eaf5700 1 -- 172.21.15.98:0/2393933412 >> 172.21.15.98:0/2393933412 conn(0x7f03880996e0 msgr2=0x7f0388153a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:34.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.340+0000 7f038eaf5700 1 -- 172.21.15.98:0/2393933412 shutdown_connections 2023-10-06T05:07:34.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.340+0000 7f038eaf5700 1 -- 172.21.15.98:0/2393933412 wait complete. 2023-10-06T05:07:34.353 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-10-06T05:07:34.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.470+0000 7f8a58865700 1 Processor -- start 2023-10-06T05:07:34.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.470+0000 7f8a58865700 1 -- start start 2023-10-06T05:07:34.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.471+0000 7f8a58865700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f8a540b2620 0x7f8a540cccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:34.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.471+0000 7f8a58865700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8a540b29f0 0x7f8a540cd1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:34.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.471+0000 7f8a58865700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a540cc970 con 0x7f8a540ceaa0 2023-10-06T05:07:34.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.471+0000 7f8a58865700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a540cbd60 con 0x7f8a540b2620 2023-10-06T05:07:34.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.471+0000 7f8a58865700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a540cc040 con 0x7f8a540b29f0 2023-10-06T05:07:34.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.471+0000 7f8a5359e700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f8a540ceaa0 0x7f8a540cef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:57526/0 (socket says 172.21.15.98:57526) 2023-10-06T05:07:34.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.471+0000 7f8a5359e700 1 -- 172.21.15.98:0/3208432989 learned_addr learned my addr 172.21.15.98:0/3208432989 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:07:34.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.472+0000 7f8a52d9d700 1 --2- 172.21.15.98:0/3208432989 >> v2:172.21.15.174:3300/0 conn(0x7f8a540b2620 0x7f8a540cccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:34.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.472+0000 7f8a4a59c700 1 --2- 172.21.15.98:0/3208432989 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8a540b29f0 0x7f8a540cd1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:34.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.472+0000 7f8a5259c700 1 -- 172.21.15.98:0/3208432989 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3290959162 0 0) 0x7f8a540cc970 con 0x7f8a540ceaa0 2023-10-06T05:07:34.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.472+0000 7f8a5259c700 1 -- 172.21.15.98:0/3208432989 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a40003570 con 0x7f8a540ceaa0 2023-10-06T05:07:34.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.472+0000 7f8a5259c700 1 -- 172.21.15.98:0/3208432989 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4073693970 0 0) 0x7f8a40003570 con 0x7f8a540ceaa0 2023-10-06T05:07:34.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.472+0000 7f8a5259c700 1 -- 172.21.15.98:0/3208432989 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8a540b29f0 msgr2=0x7f8a540cd1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:34.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.472+0000 7f8a5259c700 1 --2- 172.21.15.98:0/3208432989 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8a540b29f0 0x7f8a540cd1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:34.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.472+0000 7f8a5259c700 1 -- 172.21.15.98:0/3208432989 >> v2:172.21.15.174:3300/0 conn(0x7f8a540b2620 msgr2=0x7f8a540cccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:34.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.472+0000 7f8a5259c700 1 --2- 172.21.15.98:0/3208432989 >> v2:172.21.15.174:3300/0 conn(0x7f8a540b2620 0x7f8a540cccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:34.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.472+0000 7f8a5259c700 1 -- 172.21.15.98:0/3208432989 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a540cc320 con 0x7f8a540ceaa0 2023-10-06T05:07:34.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.472+0000 7f8a5259c700 1 -- 172.21.15.98:0/3208432989 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1543991876 0 0) 0x7f8a4c0017f0 con 0x7f8a540ceaa0 2023-10-06T05:07:34.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.472+0000 7f8a5259c700 1 -- 172.21.15.98:0/3208432989 >> v1:172.21.15.98:6789/0 conn(0x7f8a540ceaa0 legacy=0x7f8a540cef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:34.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.472+0000 7f8a5259c700 1 --2- 172.21.15.98:0/3208432989 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a40003a60 0x7f8a40003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:34.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.472+0000 7f8a5259c700 1 --2- 172.21.15.98:0/3208432989 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a40004810 0x7f8a40004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:34.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.472+0000 7f8a5259c700 1 --2- 172.21.15.98:0/3208432989 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8a540b29f0 0x7f8a400051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:34.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.472+0000 7f8a5259c700 1 -- 172.21.15.98:0/3208432989 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a540cbd60 con 0x7f8a40004810 2023-10-06T05:07:34.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.472+0000 7f8a5259c700 1 -- 172.21.15.98:0/3208432989 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a540cc040 con 0x7f8a40003a60 2023-10-06T05:07:34.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.472+0000 7f8a5259c700 1 -- 172.21.15.98:0/3208432989 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a4000ba60 con 0x7f8a540b29f0 2023-10-06T05:07:34.477 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.472+0000 7f8a4a59c700 1 --2- 172.21.15.98:0/3208432989 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a40004810 0x7f8a40004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:34.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.472+0000 7f8a52d9d700 1 --2- 172.21.15.98:0/3208432989 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a40003a60 0x7f8a40003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:34.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.472+0000 7f8a5359e700 1 --2- 172.21.15.98:0/3208432989 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8a540b29f0 0x7f8a400051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:34.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.472+0000 7f8a4a59c700 1 -- 172.21.15.98:0/3208432989 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8a540b29f0 msgr2=0x7f8a400051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:34.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.472+0000 7f8a4a59c700 1 --2- 172.21.15.98:0/3208432989 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8a540b29f0 0x7f8a400051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:34.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.472+0000 7f8a4a59c700 1 -- 172.21.15.98:0/3208432989 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a40003a60 msgr2=0x7f8a40003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:34.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.472+0000 7f8a4a59c700 1 --2- 172.21.15.98:0/3208432989 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a40003a60 0x7f8a40003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:34.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.472+0000 7f8a4a59c700 1 -- 172.21.15.98:0/3208432989 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a40003570 con 0x7f8a40004810 2023-10-06T05:07:34.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.473+0000 7f8a4a59c700 1 --2- 172.21.15.98:0/3208432989 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a40004810 0x7f8a40004c80 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c009c30 tx=0x7f8a3c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1718680c2a4cd275 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:34.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.473+0000 7f8a5259c700 1 -- 172.21.15.98:0/3208432989 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a3c00e710 con 0x7f8a40004810 2023-10-06T05:07:34.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.473+0000 7f8a5259c700 1 -- 172.21.15.98:0/3208432989 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a3c00e870 con 0x7f8a40004810 2023-10-06T05:07:34.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.473+0000 7f8a5259c700 1 -- 172.21.15.98:0/3208432989 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a3c00ec30 con 0x7f8a40004810 2023-10-06T05:07:34.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.473+0000 7f8a58865700 1 -- 172.21.15.98:0/3208432989 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a40004810 msgr2=0x7f8a40004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:34.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.473+0000 7f8a58865700 1 --2- 172.21.15.98:0/3208432989 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a40004810 0x7f8a40004c80 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c009c30 tx=0x7f8a3c0091a0 comp rx=0 tx=0).stop 2023-10-06T05:07:34.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.473+0000 7f8a5359e700 1 -- 172.21.15.98:0/3208432989 reap_dead start 2023-10-06T05:07:34.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.473+0000 7f8a58865700 1 -- 172.21.15.98:0/3208432989 shutdown_connections 2023-10-06T05:07:34.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.473+0000 7f8a58865700 1 -- 172.21.15.98:0/3208432989 >> 172.21.15.98:0/3208432989 conn(0x7f8a54099520 msgr2=0x7f8a5406f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:34.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.473+0000 7f8a58865700 1 -- 172.21.15.98:0/3208432989 shutdown_connections 2023-10-06T05:07:34.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.473+0000 7f8a58865700 1 -- 172.21.15.98:0/3208432989 wait complete. 2023-10-06T05:07:34.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.474+0000 7f8a58865700 1 Processor -- start 2023-10-06T05:07:34.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.475+0000 7f8a58865700 1 -- start start 2023-10-06T05:07:34.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.475+0000 7f8a58865700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8a40004810 0x7f8a540c8de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:34.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.475+0000 7f8a58865700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a54154060 0x7f8a54140080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:34.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.475+0000 7f8a58865700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a54140850 0x7f8a54140cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:34.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.475+0000 7f8a58865700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a541405c0 con 0x7f8a54154060 2023-10-06T05:07:34.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.475+0000 7f8a58865700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a540c8950 con 0x7f8a54140850 2023-10-06T05:07:34.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.475+0000 7f8a58865700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a540b3470 con 0x7f8a40004810 2023-10-06T05:07:34.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.475+0000 7f8a4a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a54154060 0x7f8a54140080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:34.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.475+0000 7f8a4a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a54154060 0x7f8a54140080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45264/0 (socket says 172.21.15.98:45264) 2023-10-06T05:07:34.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.475+0000 7f8a4a59c700 1 -- 172.21.15.98:0/274239103 learned_addr learned my addr 172.21.15.98:0/274239103 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:34.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.476+0000 7f8a52d9d700 1 --2- 172.21.15.98:0/274239103 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8a40004810 0x7f8a540c8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:34.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.476+0000 7f8a5359e700 1 --2- 172.21.15.98:0/274239103 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a54140850 0x7f8a54140cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:34.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.476+0000 7f8a4a59c700 1 -- 172.21.15.98:0/274239103 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8a40004810 msgr2=0x7f8a540c8de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:34.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.476+0000 7f8a4a59c700 1 --2- 172.21.15.98:0/274239103 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8a40004810 0x7f8a540c8de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:34.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.476+0000 7f8a4a59c700 1 -- 172.21.15.98:0/274239103 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a54140850 msgr2=0x7f8a54140cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:34.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.476+0000 7f8a4a59c700 1 --2- 172.21.15.98:0/274239103 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a54140850 0x7f8a54140cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:34.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.476+0000 7f8a4a59c700 1 -- 172.21.15.98:0/274239103 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a540b3750 con 0x7f8a54154060 2023-10-06T05:07:34.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.476+0000 7f8a4a59c700 1 --2- 172.21.15.98:0/274239103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a54154060 0x7f8a54140080 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c00e4b0 tx=0x7f8a3c01a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:34.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.476+0000 7f8a4b7fe700 1 -- 172.21.15.98:0/274239103 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a3c023070 con 0x7f8a54154060 2023-10-06T05:07:34.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.476+0000 7f8a4b7fe700 1 -- 172.21.15.98:0/274239103 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a3c00ec10 con 0x7f8a54154060 2023-10-06T05:07:34.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.476+0000 7f8a58865700 1 -- 172.21.15.98:0/274239103 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a540c8300 con 0x7f8a54154060 2023-10-06T05:07:34.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.476+0000 7f8a4b7fe700 1 -- 172.21.15.98:0/274239103 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a3c00e710 con 0x7f8a54154060 2023-10-06T05:07:34.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.476+0000 7f8a58865700 1 -- 172.21.15.98:0/274239103 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a540b3090 con 0x7f8a54154060 2023-10-06T05:07:34.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.477+0000 7f8a58865700 1 -- 172.21.15.98:0/274239103 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a540c8ac0 con 0x7f8a54154060 2023-10-06T05:07:34.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.477+0000 7f8a4b7fe700 1 -- 172.21.15.98:0/274239103 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f8a3c013680 con 0x7f8a54154060 2023-10-06T05:07:34.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.478+0000 7f8a4b7fe700 1 --2- 172.21.15.98:0/274239103 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f8a34066f30 0x7f8a340693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:34.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.478+0000 7f8a4b7fe700 1 -- 172.21.15.98:0/274239103 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f8a3c0860d0 con 0x7f8a54154060 2023-10-06T05:07:34.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.478+0000 7f8a52d9d700 1 --2- 172.21.15.98:0/274239103 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f8a34066f30 0x7f8a340693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:34.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.478+0000 7f8a52d9d700 1 --2- 172.21.15.98:0/274239103 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f8a34066f30 0x7f8a340693e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f8a44006aa0 tx=0x7f8a44008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:34.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.482+0000 7f8a4b7fe700 1 -- 172.21.15.98:0/274239103 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8a3c052ef0 con 0x7f8a54154060 2023-10-06T05:07:34.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.665+0000 7f8a58865700 1 -- 172.21.15.98:0/274239103 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f8a541405c0 con 0x7f8a54154060 2023-10-06T05:07:34.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.666+0000 7f8a4b7fe700 1 -- 172.21.15.98:0/274239103 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v29) v1 ==== 92+0+10 (secure 0 0 0) 0x7f8a3c056510 con 0x7f8a54154060 2023-10-06T05:07:34.667 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 8 2023-10-06T05:07:34.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.668+0000 7f8a58865700 1 -- 172.21.15.98:0/274239103 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f8a34066f30 msgr2=0x7f8a340693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:34.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.668+0000 7f8a58865700 1 --2- 172.21.15.98:0/274239103 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f8a34066f30 0x7f8a340693e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f8a44006aa0 tx=0x7f8a44008040 comp rx=0 tx=0).stop 2023-10-06T05:07:34.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.668+0000 7f8a58865700 1 -- 172.21.15.98:0/274239103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a54154060 msgr2=0x7f8a54140080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:34.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.668+0000 7f8a58865700 1 --2- 172.21.15.98:0/274239103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a54154060 0x7f8a54140080 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c00e4b0 tx=0x7f8a3c01a000 comp rx=0 tx=0).stop 2023-10-06T05:07:34.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.668+0000 7f8a58865700 1 -- 172.21.15.98:0/274239103 shutdown_connections 2023-10-06T05:07:34.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.668+0000 7f8a58865700 1 --2- 172.21.15.98:0/274239103 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a54140850 0x7f8a54140cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:34.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.668+0000 7f8a58865700 1 --2- 172.21.15.98:0/274239103 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8a40004810 0x7f8a540c8de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:34.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.668+0000 7f8a58865700 1 --2- 172.21.15.98:0/274239103 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f8a34066f30 0x7f8a340693e0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:34.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.668+0000 7f8a58865700 1 --2- 172.21.15.98:0/274239103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a54154060 0x7f8a54140080 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:34.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.668+0000 7f8a58865700 1 -- 172.21.15.98:0/274239103 >> 172.21.15.98:0/274239103 conn(0x7f8a54099520 msgr2=0x7f8a54156c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:34.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.668+0000 7f8a58865700 1 -- 172.21.15.98:0/274239103 shutdown_connections 2023-10-06T05:07:34.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.668+0000 7f8a58865700 1 -- 172.21.15.98:0/274239103 wait complete. 2023-10-06T05:07:34.681 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-06T05:07:34.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.798+0000 7fcedddc1700 1 Processor -- start 2023-10-06T05:07:34.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.798+0000 7fcedddc1700 1 -- start start 2023-10-06T05:07:34.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.798+0000 7fcedddc1700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fced81598a0 0x7fced80ce7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:34.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.798+0000 7fcedddc1700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fced80cbc70 0x7fced80ced20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:34.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.798+0000 7fcedddc1700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fced80cf5d0 con 0x7fced81594d0 2023-10-06T05:07:34.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.798+0000 7fcedddc1700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fced80cf8b0 con 0x7fced81598a0 2023-10-06T05:07:34.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.798+0000 7fcedddc1700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fced80cfb90 con 0x7fced80cbc70 2023-10-06T05:07:34.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.799+0000 7fced6ffd700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fced81598a0 0x7fced80ce7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:34.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.799+0000 7fced6ffd700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fced81598a0 0x7fced80ce7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45278/0 (socket says 172.21.15.98:45278) 2023-10-06T05:07:34.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.799+0000 7fced6ffd700 1 -- 172.21.15.98:0/3138934255 learned_addr learned my addr 172.21.15.98:0/3138934255 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:34.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.799+0000 7fced7fff700 1 --2- 172.21.15.98:0/3138934255 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fced80cbc70 0x7fced80ced20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:34.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.799+0000 7fced6ffd700 1 -- 172.21.15.98:0/3138934255 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fced80cbc70 msgr2=0x7fced80ced20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:34.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.799+0000 7fced6ffd700 1 --2- 172.21.15.98:0/3138934255 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fced80cbc70 0x7fced80ced20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:34.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.799+0000 7fced6ffd700 1 -- 172.21.15.98:0/3138934255 >> v1:172.21.15.174:6789/0 conn(0x7fced81594d0 legacy=0x7fced80cb560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:34.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.799+0000 7fced6ffd700 1 -- 172.21.15.98:0/3138934255 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fced80cfe70 con 0x7fced81598a0 2023-10-06T05:07:34.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.799+0000 7fced6ffd700 1 --2- 172.21.15.98:0/3138934255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fced81598a0 0x7fced80ce7e0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fcec00099b0 tx=0x7fcec0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17415407a6e2bbd2 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:34.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.799+0000 7fced67fc700 1 -- 172.21.15.98:0/3138934255 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcec000b820 con 0x7fced81598a0 2023-10-06T05:07:34.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.799+0000 7fced67fc700 1 -- 172.21.15.98:0/3138934255 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcec000b980 con 0x7fced81598a0 2023-10-06T05:07:34.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.799+0000 7fced67fc700 1 -- 172.21.15.98:0/3138934255 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcec000bd20 con 0x7fced81598a0 2023-10-06T05:07:34.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.800+0000 7fcedddc1700 1 -- 172.21.15.98:0/3138934255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fced81598a0 msgr2=0x7fced80ce7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:34.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.800+0000 7fcedddc1700 1 --2- 172.21.15.98:0/3138934255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fced81598a0 0x7fced80ce7e0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fcec00099b0 tx=0x7fcec0009250 comp rx=0 tx=0).stop 2023-10-06T05:07:34.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.800+0000 7fcedddc1700 1 -- 172.21.15.98:0/3138934255 shutdown_connections 2023-10-06T05:07:34.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.800+0000 7fcedddc1700 1 --2- 172.21.15.98:0/3138934255 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fced80cbc70 0x7fced80ced20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:34.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.800+0000 7fcedddc1700 1 --2- 172.21.15.98:0/3138934255 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fced81598a0 0x7fced80ce7e0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:34.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.800+0000 7fcedddc1700 1 -- 172.21.15.98:0/3138934255 >> 172.21.15.98:0/3138934255 conn(0x7fced8089e50 msgr2=0x7fced8099770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:34.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.800+0000 7fcedddc1700 1 -- 172.21.15.98:0/3138934255 shutdown_connections 2023-10-06T05:07:34.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.800+0000 7fcedddc1700 1 -- 172.21.15.98:0/3138934255 wait complete. 2023-10-06T05:07:34.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.801+0000 7fcedddc1700 1 Processor -- start 2023-10-06T05:07:34.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.801+0000 7fcedddc1700 1 -- start start 2023-10-06T05:07:34.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.802+0000 7fcedddc1700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fced80cbc70 0x7fced816b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:34.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.802+0000 7fcedddc1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fced81594d0 0x7fced8157ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:34.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.802+0000 7fcedddc1700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fced81598a0 0x7fced8158020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:34.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.802+0000 7fcedddc1700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fced8156ad0 con 0x7fced81594d0 2023-10-06T05:07:34.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.802+0000 7fcedddc1700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fced8157650 con 0x7fced81598a0 2023-10-06T05:07:34.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.802+0000 7fcedddc1700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fced8156d20 con 0x7fced80cbc70 2023-10-06T05:07:34.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.802+0000 7fced6ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fced81594d0 0x7fced8157ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:34.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.802+0000 7fced6ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fced81594d0 0x7fced8157ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45286/0 (socket says 172.21.15.98:45286) 2023-10-06T05:07:34.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.802+0000 7fced6ffd700 1 -- 172.21.15.98:0/66653005 learned_addr learned my addr 172.21.15.98:0/66653005 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:34.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.802+0000 7fced7fff700 1 --2- 172.21.15.98:0/66653005 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fced81598a0 0x7fced8158020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:34.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.802+0000 7fced77fe700 1 --2- 172.21.15.98:0/66653005 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fced80cbc70 0x7fced816b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:34.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.802+0000 7fced6ffd700 1 -- 172.21.15.98:0/66653005 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fced80cbc70 msgr2=0x7fced816b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:34.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.802+0000 7fced6ffd700 1 --2- 172.21.15.98:0/66653005 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fced80cbc70 0x7fced816b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:34.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.802+0000 7fced6ffd700 1 -- 172.21.15.98:0/66653005 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fced81598a0 msgr2=0x7fced8158020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:34.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.802+0000 7fced6ffd700 1 --2- 172.21.15.98:0/66653005 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fced81598a0 0x7fced8158020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:34.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.802+0000 7fced6ffd700 1 -- 172.21.15.98:0/66653005 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcec00094f0 con 0x7fced81594d0 2023-10-06T05:07:34.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.802+0000 7fced6ffd700 1 --2- 172.21.15.98:0/66653005 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fced81594d0 0x7fced8157ae0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fcec00030c0 tx=0x7fcec0016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:34.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.802+0000 7fcecf7fe700 1 -- 172.21.15.98:0/66653005 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcec0011bf0 con 0x7fced81594d0 2023-10-06T05:07:34.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.802+0000 7fcecf7fe700 1 -- 172.21.15.98:0/66653005 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcec0020070 con 0x7fced81594d0 2023-10-06T05:07:34.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.802+0000 7fcedddc1700 1 -- 172.21.15.98:0/66653005 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fced8157000 con 0x7fced81594d0 2023-10-06T05:07:34.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.802+0000 7fcecf7fe700 1 -- 172.21.15.98:0/66653005 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcec001b450 con 0x7fced81594d0 2023-10-06T05:07:34.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.803+0000 7fcedddc1700 1 -- 172.21.15.98:0/66653005 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fced816b790 con 0x7fced81594d0 2023-10-06T05:07:34.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.803+0000 7fcedddc1700 1 -- 172.21.15.98:0/66653005 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fced81577e0 con 0x7fced81594d0 2023-10-06T05:07:34.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.804+0000 7fcecf7fe700 1 -- 172.21.15.98:0/66653005 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fcec0022030 con 0x7fced81594d0 2023-10-06T05:07:34.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.804+0000 7fcecf7fe700 1 --2- 172.21.15.98:0/66653005 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fceb8066f30 0x7fceb80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:34.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.804+0000 7fcecf7fe700 1 -- 172.21.15.98:0/66653005 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fcec0086770 con 0x7fced81594d0 2023-10-06T05:07:34.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.804+0000 7fced77fe700 1 --2- 172.21.15.98:0/66653005 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fceb8066f30 0x7fceb80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:34.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.805+0000 7fced77fe700 1 --2- 172.21.15.98:0/66653005 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fceb8066f30 0x7fceb80693e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fcec8007fd0 tx=0x7fcec80026a0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:34.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.808+0000 7fcecf7fe700 1 -- 172.21.15.98:0/66653005 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcec0053590 con 0x7fced81594d0 2023-10-06T05:07:34.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.993+0000 7fcedddc1700 1 -- 172.21.15.98:0/66653005 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fced8157790 con 0x7fced81594d0 2023-10-06T05:07:34.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.993+0000 7fcecf7fe700 1 -- 172.21.15.98:0/66653005 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v29) v1 ==== 104+0+11 (secure 0 0 0) 0x7fcec0056bb0 con 0x7fced81594d0 2023-10-06T05:07:34.994 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2023-10-06T05:07:34.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.995+0000 7fcedddc1700 1 -- 172.21.15.98:0/66653005 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fceb8066f30 msgr2=0x7fceb80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:34.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.995+0000 7fcedddc1700 1 --2- 172.21.15.98:0/66653005 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fceb8066f30 0x7fceb80693e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fcec8007fd0 tx=0x7fcec80026a0 comp rx=0 tx=0).stop 2023-10-06T05:07:34.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.995+0000 7fcedddc1700 1 -- 172.21.15.98:0/66653005 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fced81594d0 msgr2=0x7fced8157ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:34.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.995+0000 7fcedddc1700 1 --2- 172.21.15.98:0/66653005 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fced81594d0 0x7fced8157ae0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fcec00030c0 tx=0x7fcec0016000 comp rx=0 tx=0).stop 2023-10-06T05:07:34.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.995+0000 7fcedddc1700 1 -- 172.21.15.98:0/66653005 shutdown_connections 2023-10-06T05:07:34.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.995+0000 7fcedddc1700 1 --2- 172.21.15.98:0/66653005 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fced81598a0 0x7fced8158020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:34.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.995+0000 7fcedddc1700 1 --2- 172.21.15.98:0/66653005 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fced80cbc70 0x7fced816b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:34.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.995+0000 7fcedddc1700 1 --2- 172.21.15.98:0/66653005 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fceb8066f30 0x7fceb80693e0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:34.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.995+0000 7fcedddc1700 1 --2- 172.21.15.98:0/66653005 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fced81594d0 0x7fced8157ae0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:34.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.995+0000 7fcedddc1700 1 -- 172.21.15.98:0/66653005 >> 172.21.15.98:0/66653005 conn(0x7fced8089e50 msgr2=0x7fced815db20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:34.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.995+0000 7fcedddc1700 1 -- 172.21.15.98:0/66653005 shutdown_connections 2023-10-06T05:07:34.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:34.995+0000 7fcedddc1700 1 -- 172.21.15.98:0/66653005 wait complete. 2023-10-06T05:07:35.008 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-06T05:07:35.126 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.124+0000 7f5325c8c700 1 Processor -- start 2023-10-06T05:07:35.126 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.124+0000 7f5325c8c700 1 -- start start 2023-10-06T05:07:35.128 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.125+0000 7f5325c8c700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5320159840 0x7f53200c6170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:35.128 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.125+0000 7f5325c8c700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f53200cf090 0x7f53200c66b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:35.128 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.125+0000 7f5325c8c700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53200cbb80 con 0x7f5320159470 2023-10-06T05:07:35.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.125+0000 7f5325c8c700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53200c6f00 con 0x7f53200cf090 2023-10-06T05:07:35.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.125+0000 7f5325c8c700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53200cb530 con 0x7f5320159840 2023-10-06T05:07:35.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.125+0000 7f531f7fe700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f5320159470 0x7f53200ce980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:57528/0 (socket says 172.21.15.98:57528) 2023-10-06T05:07:35.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.125+0000 7f531f7fe700 1 -- 172.21.15.98:0/2506717884 learned_addr learned my addr 172.21.15.98:0/2506717884 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:07:35.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.125+0000 7f531ffff700 1 --2- 172.21.15.98:0/2506717884 >> v2:172.21.15.174:3300/0 conn(0x7f53200cf090 0x7f53200c66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:35.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.125+0000 7f531effd700 1 --2- 172.21.15.98:0/2506717884 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5320159840 0x7f53200c6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:35.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.125+0000 7f531e7fc700 1 -- 172.21.15.98:0/2506717884 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1588139801 0 0) 0x7f53200cbb80 con 0x7f5320159470 2023-10-06T05:07:35.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.125+0000 7f531e7fc700 1 -- 172.21.15.98:0/2506717884 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5308003570 con 0x7f5320159470 2023-10-06T05:07:35.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.126+0000 7f531e7fc700 1 -- 172.21.15.98:0/2506717884 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1410133759 0 0) 0x7f5308003570 con 0x7f5320159470 2023-10-06T05:07:35.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.126+0000 7f531effd700 1 -- 172.21.15.98:0/2506717884 >> v2:172.21.15.174:3300/0 conn(0x7f53200cf090 msgr2=0x7f53200c66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:35.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.126+0000 7f531effd700 1 --2- 172.21.15.98:0/2506717884 >> v2:172.21.15.174:3300/0 conn(0x7f53200cf090 0x7f53200c66b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:35.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.126+0000 7f531effd700 1 -- 172.21.15.98:0/2506717884 >> v1:172.21.15.98:6789/0 conn(0x7f5320159470 legacy=0x7f53200ce980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:35.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.126+0000 7f531effd700 1 -- 172.21.15.98:0/2506717884 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53200cb810 con 0x7f5320159840 2023-10-06T05:07:35.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.126+0000 7f531ffff700 1 --2- 172.21.15.98:0/2506717884 >> v2:172.21.15.174:3300/0 conn(0x7f53200cf090 0x7f53200c66b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T05:07:35.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.126+0000 7f531effd700 1 --2- 172.21.15.98:0/2506717884 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5320159840 0x7f53200c6170 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f53140098e0 tx=0x7f53140091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8cbb92eb2a6914d4 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:35.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.126+0000 7f531e7fc700 1 -- 172.21.15.98:0/2506717884 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5310001970 con 0x7f5320159840 2023-10-06T05:07:35.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.126+0000 7f531e7fc700 1 -- 172.21.15.98:0/2506717884 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5314011420 con 0x7f5320159840 2023-10-06T05:07:35.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.126+0000 7f531e7fc700 1 -- 172.21.15.98:0/2506717884 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53140117a0 con 0x7f5320159840 2023-10-06T05:07:35.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.126+0000 7f5325c8c700 1 -- 172.21.15.98:0/2506717884 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5320159840 msgr2=0x7f53200c6170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:35.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.126+0000 7f5325c8c700 1 --2- 172.21.15.98:0/2506717884 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5320159840 0x7f53200c6170 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f53140098e0 tx=0x7f53140091a0 comp rx=0 tx=0).stop 2023-10-06T05:07:35.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.127+0000 7f5325c8c700 1 -- 172.21.15.98:0/2506717884 shutdown_connections 2023-10-06T05:07:35.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.127+0000 7f5325c8c700 1 --2- 172.21.15.98:0/2506717884 >> v2:172.21.15.174:3300/0 conn(0x7f53200cf090 0x7f53200c66b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:35.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.127+0000 7f5325c8c700 1 --2- 172.21.15.98:0/2506717884 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5320159840 0x7f53200c6170 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:35.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.127+0000 7f5325c8c700 1 -- 172.21.15.98:0/2506717884 >> 172.21.15.98:0/2506717884 conn(0x7f53200996e0 msgr2=0x7f532006f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:35.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.127+0000 7f5325c8c700 1 -- 172.21.15.98:0/2506717884 shutdown_connections 2023-10-06T05:07:35.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.127+0000 7f5325c8c700 1 -- 172.21.15.98:0/2506717884 wait complete. 2023-10-06T05:07:35.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.128+0000 7f5325c8c700 1 Processor -- start 2023-10-06T05:07:35.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.128+0000 7f5325c8c700 1 -- start start 2023-10-06T05:07:35.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.129+0000 7f5325c8c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53200cf090 0x7f53201560e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:35.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.129+0000 7f5325c8c700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5320159470 0x7f5320155a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:35.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.129+0000 7f5325c8c700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5320159840 0x7f532015b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:35.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.129+0000 7f5325c8c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5320154fc0 con 0x7f53200cf090 2023-10-06T05:07:35.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.129+0000 7f5325c8c700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5320155580 con 0x7f5320159470 2023-10-06T05:07:35.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.129+0000 7f5325c8c700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5320156700 con 0x7f5320159840 2023-10-06T05:07:35.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.129+0000 7f531f7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53200cf090 0x7f53201560e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:35.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.129+0000 7f531f7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53200cf090 0x7f53201560e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45290/0 (socket says 172.21.15.98:45290) 2023-10-06T05:07:35.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.129+0000 7f531f7fe700 1 -- 172.21.15.98:0/1227538468 learned_addr learned my addr 172.21.15.98:0/1227538468 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:35.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.129+0000 7f531effd700 1 --2- 172.21.15.98:0/1227538468 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5320159470 0x7f5320155a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:35.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.129+0000 7f531ffff700 1 --2- 172.21.15.98:0/1227538468 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5320159840 0x7f532015b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:35.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.129+0000 7f531f7fe700 1 -- 172.21.15.98:0/1227538468 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5320159840 msgr2=0x7f532015b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:35.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.129+0000 7f531f7fe700 1 --2- 172.21.15.98:0/1227538468 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5320159840 0x7f532015b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:35.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.129+0000 7f531f7fe700 1 -- 172.21.15.98:0/1227538468 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5320159470 msgr2=0x7f5320155a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:35.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.129+0000 7f531f7fe700 1 --2- 172.21.15.98:0/1227538468 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5320159470 0x7f5320155a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:35.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.129+0000 7f531f7fe700 1 -- 172.21.15.98:0/1227538468 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5318009420 con 0x7f53200cf090 2023-10-06T05:07:35.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.129+0000 7f531effd700 1 --2- 172.21.15.98:0/1227538468 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5320159470 0x7f5320155a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:07:35.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.129+0000 7f531ffff700 1 --2- 172.21.15.98:0/1227538468 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5320159840 0x7f532015b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:07:35.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.129+0000 7f531f7fe700 1 --2- 172.21.15.98:0/1227538468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53200cf090 0x7f53201560e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f531000ccc0 tx=0x7f53100012f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:35.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.129+0000 7f53077fe700 1 -- 172.21.15.98:0/1227538468 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5310005d80 con 0x7f53200cf090 2023-10-06T05:07:35.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.129+0000 7f53077fe700 1 -- 172.21.15.98:0/1227538468 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f531000a920 con 0x7f53200cf090 2023-10-06T05:07:35.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.129+0000 7f5325c8c700 1 -- 172.21.15.98:0/1227538468 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5314009420 con 0x7f53200cf090 2023-10-06T05:07:35.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.129+0000 7f53077fe700 1 -- 172.21.15.98:0/1227538468 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f531000ace0 con 0x7f53200cf090 2023-10-06T05:07:35.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.129+0000 7f5325c8c700 1 -- 172.21.15.98:0/1227538468 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5320155210 con 0x7f53200cf090 2023-10-06T05:07:35.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.130+0000 7f5325c8c700 1 -- 172.21.15.98:0/1227538468 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5320155860 con 0x7f53200cf090 2023-10-06T05:07:35.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.131+0000 7f53077fe700 1 -- 172.21.15.98:0/1227538468 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f5310007300 con 0x7f53200cf090 2023-10-06T05:07:35.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.131+0000 7f53077fe700 1 --2- 172.21.15.98:0/1227538468 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f530c066f30 0x7f530c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:35.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.131+0000 7f53077fe700 1 -- 172.21.15.98:0/1227538468 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f5310084b30 con 0x7f53200cf090 2023-10-06T05:07:35.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.131+0000 7f531effd700 1 --2- 172.21.15.98:0/1227538468 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f530c066f30 0x7f530c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:35.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.131+0000 7f531effd700 1 --2- 172.21.15.98:0/1227538468 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f530c066f30 0x7f530c0693e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f532006f690 tx=0x7f5314016000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:35.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.135+0000 7f53077fe700 1 -- 172.21.15.98:0/1227538468 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f53100519d0 con 0x7f53200cf090 2023-10-06T05:07:35.323 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.322+0000 7f5325c8c700 1 -- 172.21.15.98:0/1227538468 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5320064a90 con 0x7f53200cf090 2023-10-06T05:07:35.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.323+0000 7f53077fe700 1 -- 172.21.15.98:0/1227538468 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v29) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5310054ff0 con 0x7f53200cf090 2023-10-06T05:07:35.325 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2023-10-06T05:07:35.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.325+0000 7f5325c8c700 1 -- 172.21.15.98:0/1227538468 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f530c066f30 msgr2=0x7f530c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:35.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.325+0000 7f5325c8c700 1 --2- 172.21.15.98:0/1227538468 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f530c066f30 0x7f530c0693e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f532006f690 tx=0x7f5314016000 comp rx=0 tx=0).stop 2023-10-06T05:07:35.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.325+0000 7f5325c8c700 1 -- 172.21.15.98:0/1227538468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53200cf090 msgr2=0x7f53201560e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:35.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.325+0000 7f5325c8c700 1 --2- 172.21.15.98:0/1227538468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53200cf090 0x7f53201560e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f531000ccc0 tx=0x7f53100012f0 comp rx=0 tx=0).stop 2023-10-06T05:07:35.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.325+0000 7f5325c8c700 1 -- 172.21.15.98:0/1227538468 shutdown_connections 2023-10-06T05:07:35.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.325+0000 7f5325c8c700 1 --2- 172.21.15.98:0/1227538468 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5320159840 0x7f532015b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:35.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.325+0000 7f5325c8c700 1 --2- 172.21.15.98:0/1227538468 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f530c066f30 0x7f530c0693e0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:35.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.325+0000 7f5325c8c700 1 --2- 172.21.15.98:0/1227538468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53200cf090 0x7f53201560e0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:35.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.325+0000 7f5325c8c700 1 --2- 172.21.15.98:0/1227538468 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5320159470 0x7f5320155a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:35.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.325+0000 7f5325c8c700 1 -- 172.21.15.98:0/1227538468 >> 172.21.15.98:0/1227538468 conn(0x7f53200996e0 msgr2=0x7f53200ce890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:35.328 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.325+0000 7f5325c8c700 1 -- 172.21.15.98:0/1227538468 shutdown_connections 2023-10-06T05:07:35.328 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.325+0000 7f5325c8c700 1 -- 172.21.15.98:0/1227538468 wait complete. 2023-10-06T05:07:35.338 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-06T05:07:35.455 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.454+0000 7f5528cc0700 1 Processor -- start 2023-10-06T05:07:35.455 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.454+0000 7f5528cc0700 1 -- start start 2023-10-06T05:07:35.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.455+0000 7f5528cc0700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f55240cd3c0 0x7f55240ceaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:35.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.455+0000 7f5528cc0700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f55240b29f0 0x7f55240cefe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:35.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.455+0000 7f5528cc0700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55240cc970 con 0x7f55240b2620 2023-10-06T05:07:35.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.455+0000 7f5528cc0700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55240cf520 con 0x7f55240cd3c0 2023-10-06T05:07:35.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.455+0000 7f5528cc0700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55240cbd60 con 0x7f55240b29f0 2023-10-06T05:07:35.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.455+0000 7f5522ffd700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f55240cd3c0 0x7f55240ceaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:35.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.455+0000 7f55237fe700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f55240b2620 0x7f55240cccb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:57542/0 (socket says 172.21.15.98:57542) 2023-10-06T05:07:35.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.455+0000 7f55237fe700 1 -- 172.21.15.98:0/1806044470 learned_addr learned my addr 172.21.15.98:0/1806044470 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:07:35.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.455+0000 7f5522ffd700 1 -- 172.21.15.98:0/1806044470 >> v2:172.21.15.174:3300/0 conn(0x7f55240b29f0 msgr2=0x7f55240cefe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:35.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.455+0000 7f5523fff700 1 --2- 172.21.15.98:0/1806044470 >> v2:172.21.15.174:3300/0 conn(0x7f55240b29f0 0x7f55240cefe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:35.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.455+0000 7f5522ffd700 1 --2- 172.21.15.98:0/1806044470 >> v2:172.21.15.174:3300/0 conn(0x7f55240b29f0 0x7f55240cefe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:35.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.455+0000 7f5522ffd700 1 -- 172.21.15.98:0/1806044470 >> v1:172.21.15.98:6789/0 conn(0x7f55240b2620 legacy=0x7f55240cccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:35.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.455+0000 7f55227fc700 1 -- 172.21.15.98:0/1806044470 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3892246059 0 0) 0x7f55240cc970 con 0x7f55240b2620 2023-10-06T05:07:35.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.455+0000 7f5522ffd700 1 -- 172.21.15.98:0/1806044470 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55240cc040 con 0x7f55240cd3c0 2023-10-06T05:07:35.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.455+0000 7f5523fff700 1 --2- 172.21.15.98:0/1806044470 >> v2:172.21.15.174:3300/0 conn(0x7f55240b29f0 0x7f55240cefe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:07:35.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.456+0000 7f5522ffd700 1 --2- 172.21.15.98:0/1806044470 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55240cd3c0 0x7f55240ceaa0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f5524099b90 tx=0x7f55180091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3723c8f9406595c server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:35.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.456+0000 7f55227fc700 1 -- 172.21.15.98:0/1806044470 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f551800b6b0 con 0x7f55240cd3c0 2023-10-06T05:07:35.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.456+0000 7f55227fc700 1 -- 172.21.15.98:0/1806044470 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f551800b810 con 0x7f55240cd3c0 2023-10-06T05:07:35.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.456+0000 7f55227fc700 1 -- 172.21.15.98:0/1806044470 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f551800bbb0 con 0x7f55240cd3c0 2023-10-06T05:07:35.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.456+0000 7f5528cc0700 1 -- 172.21.15.98:0/1806044470 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55240cd3c0 msgr2=0x7f55240ceaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:35.460 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.456+0000 7f5528cc0700 1 --2- 172.21.15.98:0/1806044470 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55240cd3c0 0x7f55240ceaa0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f5524099b90 tx=0x7f55180091a0 comp rx=0 tx=0).stop 2023-10-06T05:07:35.460 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.456+0000 7f5528cc0700 1 -- 172.21.15.98:0/1806044470 shutdown_connections 2023-10-06T05:07:35.460 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.456+0000 7f5528cc0700 1 --2- 172.21.15.98:0/1806044470 >> v2:172.21.15.174:3300/0 conn(0x7f55240b29f0 0x7f55240cefe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:35.461 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.456+0000 7f5528cc0700 1 --2- 172.21.15.98:0/1806044470 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55240cd3c0 0x7f55240ceaa0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:35.461 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.456+0000 7f5528cc0700 1 -- 172.21.15.98:0/1806044470 >> 172.21.15.98:0/1806044470 conn(0x7f5524099520 msgr2=0x7f552406f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:35.461 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.456+0000 7f5528cc0700 1 -- 172.21.15.98:0/1806044470 shutdown_connections 2023-10-06T05:07:35.461 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.456+0000 7f5528cc0700 1 -- 172.21.15.98:0/1806044470 wait complete. 2023-10-06T05:07:35.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.458+0000 7f5528cc0700 1 Processor -- start 2023-10-06T05:07:35.464 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.458+0000 7f5528cc0700 1 -- start start 2023-10-06T05:07:35.464 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.458+0000 7f5528cc0700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f55240b2620 0x7f5524146e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:35.464 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.458+0000 7f5528cc0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55240b29f0 0x7f5524169820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:35.464 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.458+0000 7f5528cc0700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f55240cd3c0 0x7f5524159bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:35.464 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.458+0000 7f5528cc0700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f552416a860 con 0x7f55240b29f0 2023-10-06T05:07:35.464 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.458+0000 7f5528cc0700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55240b4570 con 0x7f55240b2620 2023-10-06T05:07:35.464 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.458+0000 7f5528cc0700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55241473b0 con 0x7f55240cd3c0 2023-10-06T05:07:35.464 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.458+0000 7f5522ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55240b29f0 0x7f5524169820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:35.464 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.458+0000 7f5522ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55240b29f0 0x7f5524169820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45306/0 (socket says 172.21.15.98:45306) 2023-10-06T05:07:35.464 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.458+0000 7f5522ffd700 1 -- 172.21.15.98:0/732827681 learned_addr learned my addr 172.21.15.98:0/732827681 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:35.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.459+0000 7f5523fff700 1 --2- 172.21.15.98:0/732827681 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f55240cd3c0 0x7f5524159bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:35.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.459+0000 7f5522ffd700 1 -- 172.21.15.98:0/732827681 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f55240cd3c0 msgr2=0x7f5524159bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:35.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.459+0000 7f5522ffd700 1 --2- 172.21.15.98:0/732827681 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f55240cd3c0 0x7f5524159bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:35.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.459+0000 7f55237fe700 1 --2- 172.21.15.98:0/732827681 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f55240b2620 0x7f5524146e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:35.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.459+0000 7f5522ffd700 1 -- 172.21.15.98:0/732827681 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f55240b2620 msgr2=0x7f5524146e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:35.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.459+0000 7f5522ffd700 1 --2- 172.21.15.98:0/732827681 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f55240b2620 0x7f5524146e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:35.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.459+0000 7f5522ffd700 1 -- 172.21.15.98:0/732827681 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5518009420 con 0x7f55240b29f0 2023-10-06T05:07:35.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.459+0000 7f5522ffd700 1 --2- 172.21.15.98:0/732827681 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55240b29f0 0x7f5524169820 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f552406fcc0 tx=0x7f5518017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:35.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.459+0000 7f550b7fe700 1 -- 172.21.15.98:0/732827681 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5518011420 con 0x7f55240b29f0 2023-10-06T05:07:35.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.459+0000 7f550b7fe700 1 -- 172.21.15.98:0/732827681 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5518011580 con 0x7f55240b29f0 2023-10-06T05:07:35.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.459+0000 7f5528cc0700 1 -- 172.21.15.98:0/732827681 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f552416a2a0 con 0x7f55240b29f0 2023-10-06T05:07:35.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.459+0000 7f550b7fe700 1 -- 172.21.15.98:0/732827681 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5518011940 con 0x7f55240b29f0 2023-10-06T05:07:35.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.459+0000 7f5528cc0700 1 -- 172.21.15.98:0/732827681 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55240b4200 con 0x7f55240b29f0 2023-10-06T05:07:35.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.460+0000 7f5528cc0700 1 -- 172.21.15.98:0/732827681 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55240b4060 con 0x7f55240b29f0 2023-10-06T05:07:35.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.460+0000 7f550b7fe700 1 -- 172.21.15.98:0/732827681 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f5518011cb0 con 0x7f55240b29f0 2023-10-06T05:07:35.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.461+0000 7f550b7fe700 1 --2- 172.21.15.98:0/732827681 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f5510066f30 0x7f55100693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:35.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.461+0000 7f55237fe700 1 --2- 172.21.15.98:0/732827681 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f5510066f30 0x7f55100693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:35.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.461+0000 7f550b7fe700 1 -- 172.21.15.98:0/732827681 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f55180863e0 con 0x7f55240b29f0 2023-10-06T05:07:35.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.461+0000 7f55237fe700 1 --2- 172.21.15.98:0/732827681 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f5510066f30 0x7f55100693e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5514001730 tx=0x7f5514009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:35.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.465+0000 7f550b7fe700 1 -- 172.21.15.98:0/732827681 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f55240b4060 con 0x7f55240b29f0 2023-10-06T05:07:35.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.649+0000 7f5528cc0700 1 -- 172.21.15.98:0/732827681 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f55240b4060 con 0x7f55240b29f0 2023-10-06T05:07:35.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.649+0000 7f550b7fe700 1 -- 172.21.15.98:0/732827681 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v29) v1 ==== 103+0+11 (secure 0 0 0) 0x7f55240b4060 con 0x7f55240b29f0 2023-10-06T05:07:35.651 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2023-10-06T05:07:35.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.652+0000 7f5528cc0700 1 -- 172.21.15.98:0/732827681 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f5510066f30 msgr2=0x7f55100693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:35.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.652+0000 7f5528cc0700 1 --2- 172.21.15.98:0/732827681 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f5510066f30 0x7f55100693e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5514001730 tx=0x7f5514009040 comp rx=0 tx=0).stop 2023-10-06T05:07:35.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.652+0000 7f5528cc0700 1 -- 172.21.15.98:0/732827681 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55240b29f0 msgr2=0x7f5524169820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:35.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.652+0000 7f5528cc0700 1 --2- 172.21.15.98:0/732827681 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55240b29f0 0x7f5524169820 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f552406fcc0 tx=0x7f5518017000 comp rx=0 tx=0).stop 2023-10-06T05:07:35.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.652+0000 7f5528cc0700 1 -- 172.21.15.98:0/732827681 shutdown_connections 2023-10-06T05:07:35.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.652+0000 7f5528cc0700 1 --2- 172.21.15.98:0/732827681 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f55240cd3c0 0x7f5524159bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:35.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.652+0000 7f5528cc0700 1 --2- 172.21.15.98:0/732827681 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f55240b2620 0x7f5524146e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:35.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.652+0000 7f5528cc0700 1 --2- 172.21.15.98:0/732827681 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f5510066f30 0x7f55100693e0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:35.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.652+0000 7f5528cc0700 1 --2- 172.21.15.98:0/732827681 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55240b29f0 0x7f5524169820 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:35.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.652+0000 7f5528cc0700 1 -- 172.21.15.98:0/732827681 >> 172.21.15.98:0/732827681 conn(0x7f5524099520 msgr2=0x7f55240c6970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:35.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.652+0000 7f5528cc0700 1 -- 172.21.15.98:0/732827681 shutdown_connections 2023-10-06T05:07:35.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.652+0000 7f5528cc0700 1 -- 172.21.15.98:0/732827681 wait complete. 2023-10-06T05:07:35.665 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T05:07:35.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.779+0000 7f437be0f700 1 Processor -- start 2023-10-06T05:07:35.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.779+0000 7f437be0f700 1 -- start start 2023-10-06T05:07:35.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.780+0000 7f437be0f700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f4374159560 0x7f43740aa530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:35.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.780+0000 7f437be0f700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f43740b49a0 0x7f43740aaa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:35.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.780+0000 7f437be0f700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43740b0c60 con 0x7f43740b0760 2023-10-06T05:07:35.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.780+0000 7f437be0f700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43740b0f40 con 0x7f43740b49a0 2023-10-06T05:07:35.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.780+0000 7f437be0f700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43740b1220 con 0x7f4374159560 2023-10-06T05:07:35.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.780+0000 7f43793aa700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f43740b49a0 0x7f43740aaa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:35.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.780+0000 7f43793aa700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f43740b49a0 0x7f43740aaa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45320/0 (socket says 172.21.15.98:45320) 2023-10-06T05:07:35.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.780+0000 7f43793aa700 1 -- 172.21.15.98:0/176783178 learned_addr learned my addr 172.21.15.98:0/176783178 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:35.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.780+0000 7f4379bab700 1 --2- 172.21.15.98:0/176783178 >> v2:172.21.15.174:3300/0 conn(0x7f4374159560 0x7f43740aa530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:35.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.780+0000 7f43793aa700 1 -- 172.21.15.98:0/176783178 >> v2:172.21.15.174:3300/0 conn(0x7f4374159560 msgr2=0x7f43740aa530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:35.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.780+0000 7f43793aa700 1 --2- 172.21.15.98:0/176783178 >> v2:172.21.15.174:3300/0 conn(0x7f4374159560 0x7f43740aa530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:35.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.780+0000 7f43793aa700 1 -- 172.21.15.98:0/176783178 >> v1:172.21.15.174:6789/0 conn(0x7f43740b0760 legacy=0x7f43740b3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:35.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.780+0000 7f43793aa700 1 -- 172.21.15.98:0/176783178 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43740b1500 con 0x7f43740b49a0 2023-10-06T05:07:35.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.780+0000 7f43793aa700 1 --2- 172.21.15.98:0/176783178 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43740b49a0 0x7f43740aaa70 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f43640099b0 tx=0x7f4364009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7510a687dc097ba3 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:35.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.781+0000 7f4378ba9700 1 -- 172.21.15.98:0/176783178 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f436400b7e0 con 0x7f43740b49a0 2023-10-06T05:07:35.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.781+0000 7f4378ba9700 1 -- 172.21.15.98:0/176783178 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f436400b940 con 0x7f43740b49a0 2023-10-06T05:07:35.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.781+0000 7f4378ba9700 1 -- 172.21.15.98:0/176783178 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f436400bce0 con 0x7f43740b49a0 2023-10-06T05:07:35.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.781+0000 7f437be0f700 1 -- 172.21.15.98:0/176783178 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43740b49a0 msgr2=0x7f43740aaa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:35.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.781+0000 7f437be0f700 1 --2- 172.21.15.98:0/176783178 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43740b49a0 0x7f43740aaa70 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f43640099b0 tx=0x7f4364009250 comp rx=0 tx=0).stop 2023-10-06T05:07:35.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.781+0000 7f437be0f700 1 -- 172.21.15.98:0/176783178 shutdown_connections 2023-10-06T05:07:35.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.781+0000 7f437be0f700 1 --2- 172.21.15.98:0/176783178 >> v2:172.21.15.174:3300/0 conn(0x7f4374159560 0x7f43740aa530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:35.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.781+0000 7f437be0f700 1 --2- 172.21.15.98:0/176783178 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43740b49a0 0x7f43740aaa70 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:35.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.781+0000 7f437be0f700 1 -- 172.21.15.98:0/176783178 >> 172.21.15.98:0/176783178 conn(0x7f43740997f0 msgr2=0x7f437406f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:35.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.781+0000 7f437be0f700 1 -- 172.21.15.98:0/176783178 shutdown_connections 2023-10-06T05:07:35.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.781+0000 7f437be0f700 1 -- 172.21.15.98:0/176783178 wait complete. 2023-10-06T05:07:35.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.782+0000 7f437be0f700 1 Processor -- start 2023-10-06T05:07:35.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.782+0000 7f437be0f700 1 -- start start 2023-10-06T05:07:35.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.783+0000 7f437be0f700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f43740b0760 0x7f43740aee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:35.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.783+0000 7f437be0f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43740b49a0 0x7f43740cda50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:35.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.783+0000 7f437be0f700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4374159560 0x7f43740cdf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:35.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.783+0000 7f437be0f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4374155880 con 0x7f43740b49a0 2023-10-06T05:07:35.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.783+0000 7f437be0f700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f437409cfe0 con 0x7f4374159560 2023-10-06T05:07:35.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.783+0000 7f437be0f700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f437409d2c0 con 0x7f43740b0760 2023-10-06T05:07:35.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.783+0000 7f43793aa700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43740b49a0 0x7f43740cda50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:35.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.783+0000 7f43793aa700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43740b49a0 0x7f43740cda50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45326/0 (socket says 172.21.15.98:45326) 2023-10-06T05:07:35.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.783+0000 7f43793aa700 1 -- 172.21.15.98:0/796510302 learned_addr learned my addr 172.21.15.98:0/796510302 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:35.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.783+0000 7f43793aa700 1 -- 172.21.15.98:0/796510302 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f43740b0760 msgr2=0x7f43740aee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:35.790 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.783+0000 7f43793aa700 1 --2- 172.21.15.98:0/796510302 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f43740b0760 0x7f43740aee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:35.790 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.783+0000 7f43793aa700 1 -- 172.21.15.98:0/796510302 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4374159560 msgr2=0x7f43740cdf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:35.790 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.783+0000 7f437a3ac700 1 --2- 172.21.15.98:0/796510302 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4374159560 0x7f43740cdf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:35.790 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.783+0000 7f43793aa700 1 --2- 172.21.15.98:0/796510302 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4374159560 0x7f43740cdf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:35.790 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.783+0000 7f43793aa700 1 -- 172.21.15.98:0/796510302 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43640094f0 con 0x7f43740b49a0 2023-10-06T05:07:35.790 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.783+0000 7f437a3ac700 1 --2- 172.21.15.98:0/796510302 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4374159560 0x7f43740cdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:07:35.790 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.784+0000 7f43793aa700 1 --2- 172.21.15.98:0/796510302 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43740b49a0 0x7f43740cda50 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f43640099b0 tx=0x7f4364016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:35.790 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.784+0000 7f4369ffb700 1 -- 172.21.15.98:0/796510302 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f436401b450 con 0x7f43740b49a0 2023-10-06T05:07:35.790 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.784+0000 7f4369ffb700 1 -- 172.21.15.98:0/796510302 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4364020070 con 0x7f43740b49a0 2023-10-06T05:07:35.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.784+0000 7f437be0f700 1 -- 172.21.15.98:0/796510302 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f437409d5a0 con 0x7f43740b49a0 2023-10-06T05:07:35.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.784+0000 7f4369ffb700 1 -- 172.21.15.98:0/796510302 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f436401b800 con 0x7f43740b49a0 2023-10-06T05:07:35.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.784+0000 7f437be0f700 1 -- 172.21.15.98:0/796510302 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f437409db60 con 0x7f43740b49a0 2023-10-06T05:07:35.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.785+0000 7f437be0f700 1 -- 172.21.15.98:0/796510302 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f437409dd50 con 0x7f43740b49a0 2023-10-06T05:07:35.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.785+0000 7f4369ffb700 1 -- 172.21.15.98:0/796510302 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f4364022030 con 0x7f43740b49a0 2023-10-06T05:07:35.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.786+0000 7f4369ffb700 1 --2- 172.21.15.98:0/796510302 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f4358066f30 0x7f43580693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:35.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.786+0000 7f4369ffb700 1 -- 172.21.15.98:0/796510302 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f4364086950 con 0x7f43740b49a0 2023-10-06T05:07:35.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.786+0000 7f4379bab700 1 --2- 172.21.15.98:0/796510302 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f4358066f30 0x7f43580693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:35.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.786+0000 7f4379bab700 1 --2- 172.21.15.98:0/796510302 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f4358066f30 0x7f43580693e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f4370006fd0 tx=0x7f4370008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:35.792 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.790+0000 7f4369ffb700 1 -- 172.21.15.98:0/796510302 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4364053770 con 0x7f43740b49a0 2023-10-06T05:07:35.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.974+0000 7f437be0f700 1 -- 172.21.15.98:0/796510302 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f437409dd50 con 0x7f43740b49a0 2023-10-06T05:07:35.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.975+0000 7f4369ffb700 1 -- 172.21.15.98:0/796510302 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+18707 (secure 0 0 0) 0x7f4364056d90 con 0x7f43740b49a0 2023-10-06T05:07:35.976 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:07:35.976 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":29,"fsid":"f50c5df5-cad5-47a4-9bb1-8c791d4311ec","created":"2023-10-06T05:06:32.971175+0000","modified":"2023-10-06T05:07:02.131672+0000","last_up_change":"2023-10-06T05:06:44.016489+0000","last_in_change":"2023-10-06T05:06:35.544511+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-06T05:06:44.559671+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-06T05:06:51.432161+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-10-06T05:06:55.589057+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-10-06T05:06:56.041687+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-10-06T05:06:59.754099+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d9495d95-da75-4c3b-8927-e9bae6a8b093","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":90627},{"type":"v1","addr":"172.21.15.98:6801","nonce":90627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":90627},{"type":"v1","addr":"172.21.15.98:6803","nonce":90627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":90627},{"type":"v1","addr":"172.21.15.98:6807","nonce":90627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":90627},{"type":"v1","addr":"172.21.15.98:6805","nonce":90627}]},"public_addr":"172.21.15.98:6801/90627","cluster_addr":"172.21.15.98:6803/90627","heartbeat_back_addr":"172.21.15.98:6807/90627","heartbeat_front_addr":"172.21.15.98:6805/90627","state":["exists","up"]},{"osd":1,"uuid":"f25ba5be-d65e-4707-a134-154092deb754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":90628},{"type":"v1","addr":"172.21.15.98:6809","nonce":90628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":90628},{"type":"v1","addr":"172.21.15.98:6811","nonce":90628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":90628},{"type":"v1","addr":"172.21.15.98:6815","nonce":90628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":90628},{"type":"v1","addr":"172.21.15.98:6813","nonce":90628}]},"public_addr":"172.21.15.98:6809/90628","cluster_addr":"172.21.15.98:6811/90628","heartbeat_back_addr":"172.21.15.98:6815/90628","heartbeat_front_addr":"172.21.15.98:6813/90628","state":["exists","up"]},{"osd":2,"uuid":"5ab9e028-945c-43e2-8c95-3b0b4cf38e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":90662},{"type":"v1","addr":"172.21.15.98:6818","nonce":90662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":90662},{"type":"v1","addr":"172.21.15.98:6822","nonce":90662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":90662},{"type":"v1","addr":"172.21.15.98:6830","nonce":90662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":90662},{"type":"v1","addr":"172.21.15.98:6826","nonce":90662}]},"public_addr":"172.21.15.98:6818/90662","cluster_addr":"172.21.15.98:6822/90662","heartbeat_back_addr":"172.21.15.98:6830/90662","heartbeat_front_addr":"172.21.15.98:6826/90662","state":["exists","up"]},{"osd":3,"uuid":"3bb803ff-1738-46bf-be62-2267bf378822","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6817","nonce":90663},{"type":"v1","addr":"172.21.15.98:6819","nonce":90663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6821","nonce":90663},{"type":"v1","addr":"172.21.15.98:6823","nonce":90663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6829","nonce":90663},{"type":"v1","addr":"172.21.15.98:6831","nonce":90663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6825","nonce":90663},{"type":"v1","addr":"172.21.15.98:6827","nonce":90663}]},"public_addr":"172.21.15.98:6819/90663","cluster_addr":"172.21.15.98:6823/90663","heartbeat_back_addr":"172.21.15.98:6831/90663","heartbeat_front_addr":"172.21.15.98:6827/90663","state":["exists","up"]},{"osd":4,"uuid":"7d57a4c9-6462-43f6-b64a-7f9e9b4353d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":90847},{"type":"v1","addr":"172.21.15.174:6801","nonce":90847}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":90847},{"type":"v1","addr":"172.21.15.174:6803","nonce":90847}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":90847},{"type":"v1","addr":"172.21.15.174:6807","nonce":90847}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":90847},{"type":"v1","addr":"172.21.15.174:6805","nonce":90847}]},"public_addr":"172.21.15.174:6801/90847","cluster_addr":"172.21.15.174:6803/90847","heartbeat_back_addr":"172.21.15.174:6807/90847","heartbeat_front_addr":"172.21.15.174:6805/90847","state":["exists","up"]},{"osd":5,"uuid":"83287ea4-e44d-4af9-bc77-36d47c60d4a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":90848},{"type":"v1","addr":"172.21.15.174:6809","nonce":90848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":90848},{"type":"v1","addr":"172.21.15.174:6811","nonce":90848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":90848},{"type":"v1","addr":"172.21.15.174:6815","nonce":90848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":90848},{"type":"v1","addr":"172.21.15.174:6813","nonce":90848}]},"public_addr":"172.21.15.174:6809/90848","cluster_addr":"172.21.15.174:6811/90848","heartbeat_back_addr":"172.21.15.174:6815/90848","heartbeat_front_addr":"172.21.15.174:6813/90848","state":["exists","up"]},{"osd":6,"uuid":"496a6272-892e-43c4-a145-206623c22f99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":90877},{"type":"v1","addr":"172.21.15.174:6817","nonce":90877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":90877},{"type":"v1","addr":"172.21.15.174:6819","nonce":90877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":90877},{"type":"v1","addr":"172.21.15.174:6823","nonce":90877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":90877},{"type":"v1","addr":"172.21.15.174:6821","nonce":90877}]},"public_addr":"172.21.15.174:6817/90877","cluster_addr":"172.21.15.174:6819/90877","heartbeat_back_addr":"172.21.15.174:6823/90877","heartbeat_front_addr":"172.21.15.174:6821/90877","state":["exists","up"]},{"osd":7,"uuid":"49967e34-8b07-4bab-a13e-0c16d6a0f3a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":90878},{"type":"v1","addr":"172.21.15.174:6825","nonce":90878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":90878},{"type":"v1","addr":"172.21.15.174:6827","nonce":90878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":90878},{"type":"v1","addr":"172.21.15.174:6831","nonce":90878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":90878},{"type":"v1","addr":"172.21.15.174:6829","nonce":90878}]},"public_addr":"172.21.15.174:6825/90878","cluster_addr":"172.21.15.174:6827/90878","heartbeat_back_addr":"172.21.15.174:6831/90878","heartbeat_front_addr":"172.21.15.174:6829/90878","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.446479+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.514266+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.794108+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.810894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.731919+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.717756+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.737435+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.508682+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T05:07:35.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.977+0000 7f437be0f700 1 -- 172.21.15.98:0/796510302 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f4358066f30 msgr2=0x7f43580693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:35.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.977+0000 7f437be0f700 1 --2- 172.21.15.98:0/796510302 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f4358066f30 0x7f43580693e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f4370006fd0 tx=0x7f4370008040 comp rx=0 tx=0).stop 2023-10-06T05:07:35.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.977+0000 7f437be0f700 1 -- 172.21.15.98:0/796510302 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43740b49a0 msgr2=0x7f43740cda50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:35.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.977+0000 7f437be0f700 1 --2- 172.21.15.98:0/796510302 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43740b49a0 0x7f43740cda50 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f43640099b0 tx=0x7f4364016000 comp rx=0 tx=0).stop 2023-10-06T05:07:35.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.977+0000 7f437be0f700 1 -- 172.21.15.98:0/796510302 shutdown_connections 2023-10-06T05:07:35.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.977+0000 7f437be0f700 1 --2- 172.21.15.98:0/796510302 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4374159560 0x7f43740cdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:35.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.977+0000 7f437be0f700 1 --2- 172.21.15.98:0/796510302 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f43740b0760 0x7f43740aee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:35.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.977+0000 7f437be0f700 1 --2- 172.21.15.98:0/796510302 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f4358066f30 0x7f43580693e0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:35.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.977+0000 7f437be0f700 1 --2- 172.21.15.98:0/796510302 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43740b49a0 0x7f43740cda50 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:35.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.977+0000 7f437be0f700 1 -- 172.21.15.98:0/796510302 >> 172.21.15.98:0/796510302 conn(0x7f43740997f0 msgr2=0x7f43740ccbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:35.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.977+0000 7f437be0f700 1 -- 172.21.15.98:0/796510302 shutdown_connections 2023-10-06T05:07:35.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:35.977+0000 7f437be0f700 1 -- 172.21.15.98:0/796510302 wait complete. 2023-10-06T05:07:35.991 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-06T05:06:44.559671+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-10-06T05:06:51.432161+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-06T05:06:55.589057+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-10-06T05:06:56.041687+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-06T05:06:59.754099+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '29', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-06T05:07:35.991 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-06T05:07:36.108 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.107+0000 7f00de0cc700 1 Processor -- start 2023-10-06T05:07:36.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.107+0000 7f00de0cc700 1 -- start start 2023-10-06T05:07:36.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.108+0000 7f00de0cc700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f00d81594d0 0x7f00d80cb560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:36.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.108+0000 7f00de0cc700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f00d81598a0 0x7f00d80cbaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:36.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.108+0000 7f00de0cc700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f00d80ce7e0 0x7f00d80cec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:36.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.108+0000 7f00de0cc700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00d80cf5d0 con 0x7f00d81594d0 2023-10-06T05:07:36.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.108+0000 7f00de0cc700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00d80cf8b0 con 0x7f00d80ce7e0 2023-10-06T05:07:36.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.108+0000 7f00de0cc700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00d80cfb90 con 0x7f00d81598a0 2023-10-06T05:07:36.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.108+0000 7f00d77fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f00d81594d0 0x7f00d80cb560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:36.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.108+0000 7f00d77fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f00d81594d0 0x7f00d80cb560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45336/0 (socket says 172.21.15.98:45336) 2023-10-06T05:07:36.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.108+0000 7f00d77fe700 1 -- 172.21.15.98:0/1340258964 learned_addr learned my addr 172.21.15.98:0/1340258964 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:36.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.108+0000 7f00d7fff700 1 --2- 172.21.15.98:0/1340258964 >> v2:172.21.15.174:3300/0 conn(0x7f00d80ce7e0 0x7f00d80cec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:36.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.108+0000 7f00d6ffd700 1 --2- 172.21.15.98:0/1340258964 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f00d81598a0 0x7f00d80cbaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:36.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.108+0000 7f00d77fe700 1 -- 172.21.15.98:0/1340258964 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f00d81598a0 msgr2=0x7f00d80cbaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:36.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.108+0000 7f00d77fe700 1 --2- 172.21.15.98:0/1340258964 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f00d81598a0 0x7f00d80cbaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:36.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.109+0000 7f00d77fe700 1 -- 172.21.15.98:0/1340258964 >> v2:172.21.15.174:3300/0 conn(0x7f00d80ce7e0 msgr2=0x7f00d80cec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:36.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.109+0000 7f00d77fe700 1 --2- 172.21.15.98:0/1340258964 >> v2:172.21.15.174:3300/0 conn(0x7f00d80ce7e0 0x7f00d80cec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:36.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.109+0000 7f00d77fe700 1 -- 172.21.15.98:0/1340258964 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00d80cfe70 con 0x7f00d81594d0 2023-10-06T05:07:36.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.109+0000 7f00d7fff700 1 --2- 172.21.15.98:0/1340258964 >> v2:172.21.15.174:3300/0 conn(0x7f00d80ce7e0 0x7f00d80cec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:07:36.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.109+0000 7f00d6ffd700 1 --2- 172.21.15.98:0/1340258964 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f00d81598a0 0x7f00d80cbaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:07:36.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.109+0000 7f00d77fe700 1 --2- 172.21.15.98:0/1340258964 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00d81594d0 0x7f00d80cb560 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f00d808ac70 tx=0x7f00c8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13905bfe86ce28c9 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:36.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.109+0000 7f00d67fc700 1 -- 172.21.15.98:0/1340258964 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00c800e070 con 0x7f00d81594d0 2023-10-06T05:07:36.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.109+0000 7f00d67fc700 1 -- 172.21.15.98:0/1340258964 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00c800ba50 con 0x7f00d81594d0 2023-10-06T05:07:36.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.109+0000 7f00d67fc700 1 -- 172.21.15.98:0/1340258964 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00c80092e0 con 0x7f00d81594d0 2023-10-06T05:07:36.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.109+0000 7f00de0cc700 1 -- 172.21.15.98:0/1340258964 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00d81594d0 msgr2=0x7f00d80cb560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:36.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.109+0000 7f00de0cc700 1 --2- 172.21.15.98:0/1340258964 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00d81594d0 0x7f00d80cb560 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f00d808ac70 tx=0x7f00c8009840 comp rx=0 tx=0).stop 2023-10-06T05:07:36.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.109+0000 7f00de0cc700 1 -- 172.21.15.98:0/1340258964 shutdown_connections 2023-10-06T05:07:36.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.109+0000 7f00de0cc700 1 --2- 172.21.15.98:0/1340258964 >> v2:172.21.15.174:3300/0 conn(0x7f00d80ce7e0 0x7f00d80cec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:36.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.109+0000 7f00de0cc700 1 --2- 172.21.15.98:0/1340258964 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00d81594d0 0x7f00d80cb560 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:36.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.109+0000 7f00de0cc700 1 --2- 172.21.15.98:0/1340258964 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f00d81598a0 0x7f00d80cbaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:36.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.109+0000 7f00de0cc700 1 -- 172.21.15.98:0/1340258964 >> 172.21.15.98:0/1340258964 conn(0x7f00d8089e50 msgr2=0x7f00d8099770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:36.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.109+0000 7f00de0cc700 1 -- 172.21.15.98:0/1340258964 shutdown_connections 2023-10-06T05:07:36.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.109+0000 7f00de0cc700 1 -- 172.21.15.98:0/1340258964 wait complete. 2023-10-06T05:07:36.115 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.111+0000 7f00de0cc700 1 Processor -- start 2023-10-06T05:07:36.115 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.111+0000 7f00de0cc700 1 -- start start 2023-10-06T05:07:36.117 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.111+0000 7f00de0cc700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f00d80ce7e0 0x7f00d8158380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:36.117 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.111+0000 7f00de0cc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00d81594d0 0x7f00d80cad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:36.117 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.111+0000 7f00de0cc700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f00d81598a0 0x7f00d80cb270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:36.117 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.111+0000 7f00de0cc700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00d8169b90 con 0x7f00d81594d0 2023-10-06T05:07:36.117 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.111+0000 7f00de0cc700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00d8169e70 con 0x7f00d80ce7e0 2023-10-06T05:07:36.118 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.111+0000 7f00de0cc700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00d816a150 con 0x7f00d81598a0 2023-10-06T05:07:36.118 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.111+0000 7f00d6ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00d81594d0 0x7f00d80cad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:36.118 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.111+0000 7f00d6ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00d81594d0 0x7f00d80cad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45346/0 (socket says 172.21.15.98:45346) 2023-10-06T05:07:36.118 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.111+0000 7f00d6ffd700 1 -- 172.21.15.98:0/2161104920 learned_addr learned my addr 172.21.15.98:0/2161104920 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:36.118 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.112+0000 7f00d77fe700 1 --2- 172.21.15.98:0/2161104920 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f00d80ce7e0 0x7f00d8158380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:36.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.112+0000 7f00d7fff700 1 --2- 172.21.15.98:0/2161104920 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f00d81598a0 0x7f00d80cb270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:36.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.112+0000 7f00d6ffd700 1 -- 172.21.15.98:0/2161104920 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f00d81598a0 msgr2=0x7f00d80cb270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:36.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.112+0000 7f00d6ffd700 1 --2- 172.21.15.98:0/2161104920 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f00d81598a0 0x7f00d80cb270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:36.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.112+0000 7f00d6ffd700 1 -- 172.21.15.98:0/2161104920 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f00d80ce7e0 msgr2=0x7f00d8158380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:36.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.112+0000 7f00d6ffd700 1 --2- 172.21.15.98:0/2161104920 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f00d80ce7e0 0x7f00d8158380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:36.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.112+0000 7f00d6ffd700 1 -- 172.21.15.98:0/2161104920 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00c8009ae0 con 0x7f00d81594d0 2023-10-06T05:07:36.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.112+0000 7f00d7fff700 1 --2- 172.21.15.98:0/2161104920 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f00d81598a0 0x7f00d80cb270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:07:36.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.112+0000 7f00d6ffd700 1 --2- 172.21.15.98:0/2161104920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00d81594d0 0x7f00d80cad30 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f00cc006fd0 tx=0x7f00cc008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:36.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.112+0000 7f00bf7fe700 1 -- 172.21.15.98:0/2161104920 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00cc013410 con 0x7f00d81594d0 2023-10-06T05:07:36.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.112+0000 7f00bf7fe700 1 -- 172.21.15.98:0/2161104920 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00cc013570 con 0x7f00d81594d0 2023-10-06T05:07:36.120 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.112+0000 7f00de0cc700 1 -- 172.21.15.98:0/2161104920 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00d816a430 con 0x7f00d81594d0 2023-10-06T05:07:36.120 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.112+0000 7f00bf7fe700 1 -- 172.21.15.98:0/2161104920 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00cc013910 con 0x7f00d81594d0 2023-10-06T05:07:36.120 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.112+0000 7f00de0cc700 1 -- 172.21.15.98:0/2161104920 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00d81588c0 con 0x7f00d81594d0 2023-10-06T05:07:36.120 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.113+0000 7f00de0cc700 1 -- 172.21.15.98:0/2161104920 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00d8064a90 con 0x7f00d81594d0 2023-10-06T05:07:36.120 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.113+0000 7f00bf7fe700 1 -- 172.21.15.98:0/2161104920 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f00cc013c80 con 0x7f00d81594d0 2023-10-06T05:07:36.120 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.114+0000 7f00bf7fe700 1 --2- 172.21.15.98:0/2161104920 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f00c4066f30 0x7f00c40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:36.120 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.114+0000 7f00bf7fe700 1 -- 172.21.15.98:0/2161104920 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f00cc0846b0 con 0x7f00d81594d0 2023-10-06T05:07:36.120 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.114+0000 7f00d77fe700 1 --2- 172.21.15.98:0/2161104920 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f00c4066f30 0x7f00c40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:36.121 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.114+0000 7f00d77fe700 1 --2- 172.21.15.98:0/2161104920 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f00c4066f30 0x7f00c40693e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f00c8009880 tx=0x7f00c8018000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:36.121 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.118+0000 7f00bf7fe700 1 -- 172.21.15.98:0/2161104920 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f00cc054f60 con 0x7f00d81594d0 2023-10-06T05:07:36.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.300+0000 7f00de0cc700 1 -- 172.21.15.98:0/2161104920 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f00d80cf140 con 0x7f00d81594d0 2023-10-06T05:07:36.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.301+0000 7f00bf7fe700 1 -- 172.21.15.98:0/2161104920 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v29) v1 ==== 93+0+10 (secure 0 0 0) 0x7f00cc054af0 con 0x7f00d81594d0 2023-10-06T05:07:36.302 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2023-10-06T05:07:36.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.303+0000 7f00de0cc700 1 -- 172.21.15.98:0/2161104920 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f00c4066f30 msgr2=0x7f00c40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:36.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.303+0000 7f00de0cc700 1 --2- 172.21.15.98:0/2161104920 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f00c4066f30 0x7f00c40693e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f00c8009880 tx=0x7f00c8018000 comp rx=0 tx=0).stop 2023-10-06T05:07:36.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.303+0000 7f00de0cc700 1 -- 172.21.15.98:0/2161104920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00d81594d0 msgr2=0x7f00d80cad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:36.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.303+0000 7f00de0cc700 1 --2- 172.21.15.98:0/2161104920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00d81594d0 0x7f00d80cad30 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f00cc006fd0 tx=0x7f00cc008040 comp rx=0 tx=0).stop 2023-10-06T05:07:36.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.303+0000 7f00de0cc700 1 -- 172.21.15.98:0/2161104920 shutdown_connections 2023-10-06T05:07:36.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.303+0000 7f00de0cc700 1 --2- 172.21.15.98:0/2161104920 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f00d81598a0 0x7f00d80cb270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:36.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.303+0000 7f00de0cc700 1 --2- 172.21.15.98:0/2161104920 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f00d80ce7e0 0x7f00d8158380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:36.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.303+0000 7f00de0cc700 1 --2- 172.21.15.98:0/2161104920 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f00c4066f30 0x7f00c40693e0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:36.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.303+0000 7f00de0cc700 1 --2- 172.21.15.98:0/2161104920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00d81594d0 0x7f00d80cad30 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:36.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.303+0000 7f00de0cc700 1 -- 172.21.15.98:0/2161104920 >> 172.21.15.98:0/2161104920 conn(0x7f00d8089e50 msgr2=0x7f00d814f750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:36.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.303+0000 7f00de0cc700 1 -- 172.21.15.98:0/2161104920 shutdown_connections 2023-10-06T05:07:36.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.303+0000 7f00de0cc700 1 -- 172.21.15.98:0/2161104920 wait complete. 2023-10-06T05:07:36.316 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-10-06T05:07:36.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.432+0000 7fd2a6dcd700 1 Processor -- start 2023-10-06T05:07:36.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.432+0000 7fd2a6dcd700 1 -- start start 2023-10-06T05:07:36.436 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.432+0000 7fd2a6dcd700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd2a00cd940 0x7fd2a0159bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:36.436 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.432+0000 7fd2a6dcd700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd2a00b4a20 0x7fd2a00d02d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:36.436 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.432+0000 7fd2a6dcd700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2a00b5ef0 con 0x7fd2a00b4e70 2023-10-06T05:07:36.436 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.432+0000 7fd2a6dcd700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2a00b3600 con 0x7fd2a00cd940 2023-10-06T05:07:36.436 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.432+0000 7fd2a6dcd700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a00b6380 con 0x7fd2a00b4a20 2023-10-06T05:07:36.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.433+0000 7fd2a536a700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fd2a00b4e70 0x7fd2a015ada0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:57546/0 (socket says 172.21.15.98:57546) 2023-10-06T05:07:36.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.433+0000 7fd2a4b69700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd2a00cd940 0x7fd2a0159bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:36.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.433+0000 7fd2a536a700 1 -- 172.21.15.98:0/4195097430 learned_addr learned my addr 172.21.15.98:0/4195097430 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:07:36.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.433+0000 7fd297fff700 1 --2- 172.21.15.98:0/4195097430 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd2a00b4a20 0x7fd2a00d02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:36.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.433+0000 7fd2977fe700 1 -- 172.21.15.98:0/4195097430 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3288563131 0 0) 0x7fd2a00b5ef0 con 0x7fd2a00b4e70 2023-10-06T05:07:36.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.433+0000 7fd2977fe700 1 -- 172.21.15.98:0/4195097430 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd288003570 con 0x7fd2a00b4e70 2023-10-06T05:07:36.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.433+0000 7fd2a4b69700 1 -- 172.21.15.98:0/4195097430 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd2a00b4a20 msgr2=0x7fd2a00d02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:36.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.433+0000 7fd2a4b69700 1 --2- 172.21.15.98:0/4195097430 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd2a00b4a20 0x7fd2a00d02d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:36.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.433+0000 7fd2a4b69700 1 -- 172.21.15.98:0/4195097430 >> v1:172.21.15.98:6789/0 conn(0x7fd2a00b4e70 legacy=0x7fd2a015ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:36.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.433+0000 7fd2a4b69700 1 -- 172.21.15.98:0/4195097430 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2a00b52e0 con 0x7fd2a00cd940 2023-10-06T05:07:36.438 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.433+0000 7fd2a4b69700 1 --2- 172.21.15.98:0/4195097430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2a00cd940 0x7fd2a0159bd0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fd29c009e60 tx=0x7fd29c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62bbca76a2e4d746 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:36.438 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.433+0000 7fd2977fe700 1 -- 172.21.15.98:0/4195097430 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd29c00e070 con 0x7fd2a00cd940 2023-10-06T05:07:36.438 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.433+0000 7fd2977fe700 1 -- 172.21.15.98:0/4195097430 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd29c00bab0 con 0x7fd2a00cd940 2023-10-06T05:07:36.438 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.433+0000 7fd2977fe700 1 -- 172.21.15.98:0/4195097430 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd29c00be70 con 0x7fd2a00cd940 2023-10-06T05:07:36.438 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.434+0000 7fd2a6dcd700 1 -- 172.21.15.98:0/4195097430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2a00cd940 msgr2=0x7fd2a0159bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:36.438 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.434+0000 7fd2a6dcd700 1 --2- 172.21.15.98:0/4195097430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2a00cd940 0x7fd2a0159bd0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fd29c009e60 tx=0x7fd29c009780 comp rx=0 tx=0).stop 2023-10-06T05:07:36.438 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.434+0000 7fd2a6dcd700 1 -- 172.21.15.98:0/4195097430 shutdown_connections 2023-10-06T05:07:36.438 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.434+0000 7fd2a6dcd700 1 --2- 172.21.15.98:0/4195097430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2a00cd940 0x7fd2a0159bd0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:36.438 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.434+0000 7fd2a6dcd700 1 --2- 172.21.15.98:0/4195097430 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd2a00b4a20 0x7fd2a00d02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:36.439 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.434+0000 7fd2a6dcd700 1 -- 172.21.15.98:0/4195097430 >> 172.21.15.98:0/4195097430 conn(0x7fd2a006ece0 msgr2=0x7fd2a006f0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:36.439 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.434+0000 7fd2a6dcd700 1 -- 172.21.15.98:0/4195097430 shutdown_connections 2023-10-06T05:07:36.439 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.434+0000 7fd2a6dcd700 1 -- 172.21.15.98:0/4195097430 wait complete. 2023-10-06T05:07:36.439 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.435+0000 7fd2a6dcd700 1 Processor -- start 2023-10-06T05:07:36.439 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.435+0000 7fd2a6dcd700 1 -- start start 2023-10-06T05:07:36.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.436+0000 7fd2a6dcd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2a00b4a20 0x7fd2a016d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:36.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.436+0000 7fd2a6dcd700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd2a00b4e70 0x7fd2a01486b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:36.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.436+0000 7fd2a6dcd700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd2a00cd940 0x7fd2a0159a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:36.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.436+0000 7fd2a6dcd700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a00cde40 con 0x7fd2a00b4a20 2023-10-06T05:07:36.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.436+0000 7fd2a6dcd700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a00ce120 con 0x7fd2a00cd940 2023-10-06T05:07:36.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.436+0000 7fd2a6dcd700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a00ce400 con 0x7fd2a00b4e70 2023-10-06T05:07:36.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.436+0000 7fd2a4b69700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2a00b4a20 0x7fd2a016d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:36.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.436+0000 7fd2a4b69700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2a00b4a20 0x7fd2a016d1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45362/0 (socket says 172.21.15.98:45362) 2023-10-06T05:07:36.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.436+0000 7fd2a4b69700 1 -- 172.21.15.98:0/359578929 learned_addr learned my addr 172.21.15.98:0/359578929 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:36.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.436+0000 7fd297fff700 1 --2- 172.21.15.98:0/359578929 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd2a00b4e70 0x7fd2a01486b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:36.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.436+0000 7fd2a536a700 1 --2- 172.21.15.98:0/359578929 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd2a00cd940 0x7fd2a0159a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:36.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.436+0000 7fd2a4b69700 1 -- 172.21.15.98:0/359578929 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd2a00b4e70 msgr2=0x7fd2a01486b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:36.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.436+0000 7fd2a4b69700 1 --2- 172.21.15.98:0/359578929 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd2a00b4e70 0x7fd2a01486b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:36.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.436+0000 7fd2a4b69700 1 -- 172.21.15.98:0/359578929 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd2a00cd940 msgr2=0x7fd2a0159a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:36.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.436+0000 7fd2a4b69700 1 --2- 172.21.15.98:0/359578929 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd2a00cd940 0x7fd2a0159a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:36.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.436+0000 7fd2a4b69700 1 -- 172.21.15.98:0/359578929 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd29c009a20 con 0x7fd2a00b4a20 2023-10-06T05:07:36.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.436+0000 7fd297fff700 1 --2- 172.21.15.98:0/359578929 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd2a00b4e70 0x7fd2a01486b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:07:36.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.436+0000 7fd2a536a700 1 --2- 172.21.15.98:0/359578929 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd2a00cd940 0x7fd2a0159a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:07:36.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.436+0000 7fd2a4b69700 1 --2- 172.21.15.98:0/359578929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2a00b4a20 0x7fd2a016d1c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fd29c0097c0 tx=0x7fd29c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:36.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.436+0000 7fd294ff9700 1 -- 172.21.15.98:0/359578929 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd29c012d00 con 0x7fd2a00b4a20 2023-10-06T05:07:36.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.436+0000 7fd294ff9700 1 -- 172.21.15.98:0/359578929 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd29c012e80 con 0x7fd2a00b4a20 2023-10-06T05:07:36.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.436+0000 7fd294ff9700 1 -- 172.21.15.98:0/359578929 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd29c0276c0 con 0x7fd2a00b4a20 2023-10-06T05:07:36.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.436+0000 7fd2a6dcd700 1 -- 172.21.15.98:0/359578929 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2a00ce6e0 con 0x7fd2a00b4a20 2023-10-06T05:07:36.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.436+0000 7fd2a6dcd700 1 -- 172.21.15.98:0/359578929 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2a0156630 con 0x7fd2a00b4a20 2023-10-06T05:07:36.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.437+0000 7fd2a6dcd700 1 -- 172.21.15.98:0/359578929 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2a00ceb80 con 0x7fd2a00b4a20 2023-10-06T05:07:36.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.438+0000 7fd294ff9700 1 -- 172.21.15.98:0/359578929 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fd29c024070 con 0x7fd2a00b4a20 2023-10-06T05:07:36.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.438+0000 7fd294ff9700 1 --2- 172.21.15.98:0/359578929 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fd28c066f60 0x7fd28c069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:36.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.438+0000 7fd297fff700 1 --2- 172.21.15.98:0/359578929 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fd28c066f60 0x7fd28c069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:36.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.438+0000 7fd294ff9700 1 -- 172.21.15.98:0/359578929 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fd29c086eb0 con 0x7fd2a00b4a20 2023-10-06T05:07:36.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.438+0000 7fd297fff700 1 --2- 172.21.15.98:0/359578929 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fd28c066f60 0x7fd28c069410 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd290006f00 tx=0x7fd290008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:36.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.442+0000 7fd294ff9700 1 -- 172.21.15.98:0/359578929 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd29c053cd0 con 0x7fd2a00b4a20 2023-10-06T05:07:36.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.624+0000 7fd2a6dcd700 1 -- 172.21.15.98:0/359578929 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fd2a0169c30 con 0x7fd2a00b4a20 2023-10-06T05:07:36.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.624+0000 7fd294ff9700 1 -- 172.21.15.98:0/359578929 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v29) v1 ==== 92+0+10 (secure 0 0 0) 0x7fd29c02c0a0 con 0x7fd2a00b4a20 2023-10-06T05:07:36.626 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 8 2023-10-06T05:07:36.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.626+0000 7fd2a6dcd700 1 -- 172.21.15.98:0/359578929 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fd28c066f60 msgr2=0x7fd28c069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:36.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.626+0000 7fd2a6dcd700 1 --2- 172.21.15.98:0/359578929 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fd28c066f60 0x7fd28c069410 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd290006f00 tx=0x7fd290008040 comp rx=0 tx=0).stop 2023-10-06T05:07:36.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.626+0000 7fd2a6dcd700 1 -- 172.21.15.98:0/359578929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2a00b4a20 msgr2=0x7fd2a016d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:36.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.626+0000 7fd2a6dcd700 1 --2- 172.21.15.98:0/359578929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2a00b4a20 0x7fd2a016d1c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fd29c0097c0 tx=0x7fd29c019000 comp rx=0 tx=0).stop 2023-10-06T05:07:36.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.626+0000 7fd2a6dcd700 1 -- 172.21.15.98:0/359578929 shutdown_connections 2023-10-06T05:07:36.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.626+0000 7fd2a6dcd700 1 --2- 172.21.15.98:0/359578929 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd2a00cd940 0x7fd2a0159a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:36.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.626+0000 7fd2a6dcd700 1 --2- 172.21.15.98:0/359578929 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fd28c066f60 0x7fd28c069410 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:36.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.626+0000 7fd2a6dcd700 1 --2- 172.21.15.98:0/359578929 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2a00b4a20 0x7fd2a016d1c0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:36.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.626+0000 7fd2a6dcd700 1 --2- 172.21.15.98:0/359578929 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd2a00b4e70 0x7fd2a01486b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:36.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.626+0000 7fd2a6dcd700 1 -- 172.21.15.98:0/359578929 >> 172.21.15.98:0/359578929 conn(0x7fd2a006ece0 msgr2=0x7fd2a015ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:36.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.626+0000 7fd2a6dcd700 1 -- 172.21.15.98:0/359578929 shutdown_connections 2023-10-06T05:07:36.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.626+0000 7fd2a6dcd700 1 -- 172.21.15.98:0/359578929 wait complete. 2023-10-06T05:07:36.639 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-06T05:07:36.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.757+0000 7fd0ad646700 1 Processor -- start 2023-10-06T05:07:36.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.757+0000 7fd0ad646700 1 -- start start 2023-10-06T05:07:36.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.758+0000 7fd0ad646700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fd0a815da80 0x7fd0a80c8320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:36.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.758+0000 7fd0ad646700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0a80c8860 con 0x7fd0a81593b0 2023-10-06T05:07:36.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.758+0000 7fd0ad646700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0a80c5f40 con 0x7fd0a80685c0 2023-10-06T05:07:36.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.758+0000 7fd0ad646700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0a80c9050 con 0x7fd0a815da80 2023-10-06T05:07:36.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.758+0000 7fd0a6ffd700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fd0a81593b0 0x7fd0a80c63d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:57552/0 (socket says 172.21.15.98:57552) 2023-10-06T05:07:36.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.758+0000 7fd0a6ffd700 1 -- 172.21.15.98:0/1661828752 learned_addr learned my addr 172.21.15.98:0/1661828752 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:07:36.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.758+0000 7fd0a67fc700 1 -- 172.21.15.98:0/1661828752 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3083428844 0 0) 0x7fd0a80c8860 con 0x7fd0a81593b0 2023-10-06T05:07:36.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.758+0000 7fd0a67fc700 1 -- 172.21.15.98:0/1661828752 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd094003570 con 0x7fd0a81593b0 2023-10-06T05:07:36.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.759+0000 7fd09e7fc700 1 --2- 172.21.15.98:0/1661828752 >> v2:172.21.15.174:3300/0 conn(0x7fd0a815da80 0x7fd0a80c8320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:36.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.759+0000 7fd0a67fc700 1 -- 172.21.15.98:0/1661828752 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1143291667 0 0) 0x7fd094003570 con 0x7fd0a81593b0 2023-10-06T05:07:36.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.759+0000 7fd0a67fc700 1 -- 172.21.15.98:0/1661828752 >> v2:172.21.15.174:3300/0 conn(0x7fd0a815da80 msgr2=0x7fd0a80c8320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:36.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.759+0000 7fd0a67fc700 1 --2- 172.21.15.98:0/1661828752 >> v2:172.21.15.174:3300/0 conn(0x7fd0a815da80 0x7fd0a80c8320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:36.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.759+0000 7fd0a67fc700 1 -- 172.21.15.98:0/1661828752 >> v1:172.21.15.174:6789/0 conn(0x7fd0a80685c0 legacy=0x7fd0a806add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:36.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.759+0000 7fd0a67fc700 1 -- 172.21.15.98:0/1661828752 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0a80c8a90 con 0x7fd0a81593b0 2023-10-06T05:07:36.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.759+0000 7fd0a67fc700 1 -- 172.21.15.98:0/1661828752 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1543991876 0 0) 0x7fd098001470 con 0x7fd0a81593b0 2023-10-06T05:07:36.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.759+0000 7fd0a67fc700 1 -- 172.21.15.98:0/1661828752 >> v1:172.21.15.98:6789/0 conn(0x7fd0a81593b0 legacy=0x7fd0a80c63d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:36.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.759+0000 7fd0a67fc700 1 --2- 172.21.15.98:0/1661828752 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd094003a60 0x7fd094003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:36.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.759+0000 7fd0a67fc700 1 --2- 172.21.15.98:0/1661828752 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd094004810 0x7fd094004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:36.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.759+0000 7fd0a67fc700 1 --2- 172.21.15.98:0/1661828752 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0940051c0 0x7fd09400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:36.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.759+0000 7fd0a67fc700 1 -- 172.21.15.98:0/1661828752 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0a80c9050 con 0x7fd0940051c0 2023-10-06T05:07:36.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.759+0000 7fd0a67fc700 1 -- 172.21.15.98:0/1661828752 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd09400c0d0 con 0x7fd094003a60 2023-10-06T05:07:36.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.759+0000 7fd0a67fc700 1 -- 172.21.15.98:0/1661828752 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd09400c290 con 0x7fd094004810 2023-10-06T05:07:36.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.759+0000 7fd0a77fe700 1 --2- 172.21.15.98:0/1661828752 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0940051c0 0x7fd09400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:36.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.759+0000 7fd0a6ffd700 1 --2- 172.21.15.98:0/1661828752 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd094003a60 0x7fd094003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:36.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.759+0000 7fd09e7fc700 1 --2- 172.21.15.98:0/1661828752 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd094004810 0x7fd094004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:36.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.759+0000 7fd0a77fe700 1 -- 172.21.15.98:0/1661828752 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd094004810 msgr2=0x7fd094004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:36.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.759+0000 7fd0a77fe700 1 --2- 172.21.15.98:0/1661828752 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd094004810 0x7fd094004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:36.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.759+0000 7fd0a77fe700 1 -- 172.21.15.98:0/1661828752 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd094003a60 msgr2=0x7fd094003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:36.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.759+0000 7fd0a77fe700 1 --2- 172.21.15.98:0/1661828752 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd094003a60 0x7fd094003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:36.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.759+0000 7fd0a77fe700 1 -- 172.21.15.98:0/1661828752 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd094003570 con 0x7fd0940051c0 2023-10-06T05:07:36.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.759+0000 7fd0a77fe700 1 -- 172.21.15.98:0/1661828752 reap_dead start 2023-10-06T05:07:36.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.759+0000 7fd0a77fe700 1 --2- 172.21.15.98:0/1661828752 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0940051c0 0x7fd09400ba60 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fd0a0001fc0 tx=0x7fd0a0001640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72a8e1d590634612 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:36.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.760+0000 7fd0a67fc700 1 -- 172.21.15.98:0/1661828752 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0a0001950 con 0x7fd0940051c0 2023-10-06T05:07:36.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.760+0000 7fd0a67fc700 1 -- 172.21.15.98:0/1661828752 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0a000c8b0 con 0x7fd0940051c0 2023-10-06T05:07:36.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.760+0000 7fd0a67fc700 1 -- 172.21.15.98:0/1661828752 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0a000cc90 con 0x7fd0940051c0 2023-10-06T05:07:36.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.760+0000 7fd0ad646700 1 -- 172.21.15.98:0/1661828752 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0940051c0 msgr2=0x7fd09400ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:36.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.760+0000 7fd0ad646700 1 --2- 172.21.15.98:0/1661828752 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0940051c0 0x7fd09400ba60 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fd0a0001fc0 tx=0x7fd0a0001640 comp rx=0 tx=0).stop 2023-10-06T05:07:36.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.760+0000 7fd0ad646700 1 -- 172.21.15.98:0/1661828752 shutdown_connections 2023-10-06T05:07:36.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.760+0000 7fd0ad646700 1 --2- 172.21.15.98:0/1661828752 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0940051c0 0x7fd09400ba60 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:36.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.760+0000 7fd0ad646700 1 -- 172.21.15.98:0/1661828752 >> 172.21.15.98:0/1661828752 conn(0x7fd0a80c9c90 msgr2=0x7fd0a806da00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:36.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.760+0000 7fd0ad646700 1 -- 172.21.15.98:0/1661828752 shutdown_connections 2023-10-06T05:07:36.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.760+0000 7fd0ad646700 1 -- 172.21.15.98:0/1661828752 wait complete. 2023-10-06T05:07:36.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.761+0000 7fd0ad646700 1 Processor -- start 2023-10-06T05:07:36.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.761+0000 7fd0ad646700 1 -- start start 2023-10-06T05:07:36.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.762+0000 7fd0ad646700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd0940051c0 0x7fd0a81556c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:36.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.762+0000 7fd0ad646700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd0a80aafc0 0x7fd0a80b1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:36.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.762+0000 7fd0ad646700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0a80c6d70 0x7fd0a80c71f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:36.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.762+0000 7fd0ad646700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0a80c7730 con 0x7fd0a80c6d70 2023-10-06T05:07:36.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.762+0000 7fd0ad646700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0a816aaf0 con 0x7fd0a80aafc0 2023-10-06T05:07:36.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.762+0000 7fd0ad646700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0a8169f20 con 0x7fd0940051c0 2023-10-06T05:07:36.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.762+0000 7fd0a77fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0a80c6d70 0x7fd0a80c71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:36.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.762+0000 7fd0a77fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0a80c6d70 0x7fd0a80c71f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45372/0 (socket says 172.21.15.98:45372) 2023-10-06T05:07:36.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.762+0000 7fd0a77fe700 1 -- 172.21.15.98:0/2740772015 learned_addr learned my addr 172.21.15.98:0/2740772015 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:36.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.762+0000 7fd09e7fc700 1 --2- 172.21.15.98:0/2740772015 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd0a80aafc0 0x7fd0a80b1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:36.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.762+0000 7fd0a6ffd700 1 --2- 172.21.15.98:0/2740772015 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd0940051c0 0x7fd0a81556c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:36.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.763+0000 7fd0a77fe700 1 -- 172.21.15.98:0/2740772015 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd0940051c0 msgr2=0x7fd0a81556c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:36.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.763+0000 7fd0a77fe700 1 --2- 172.21.15.98:0/2740772015 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd0940051c0 0x7fd0a81556c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:36.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.763+0000 7fd0a77fe700 1 -- 172.21.15.98:0/2740772015 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd0a80aafc0 msgr2=0x7fd0a80b1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:36.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.763+0000 7fd0a77fe700 1 --2- 172.21.15.98:0/2740772015 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd0a80aafc0 0x7fd0a80b1c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:36.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.763+0000 7fd0a77fe700 1 -- 172.21.15.98:0/2740772015 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0a816a1c0 con 0x7fd0a80c6d70 2023-10-06T05:07:36.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.763+0000 7fd0a77fe700 1 --2- 172.21.15.98:0/2740772015 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0a80c6d70 0x7fd0a80c71f0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fd0a000c6a0 tx=0x7fd0a0015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:36.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.763+0000 7fd09f7fe700 1 -- 172.21.15.98:0/2740772015 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0a0001950 con 0x7fd0a80c6d70 2023-10-06T05:07:36.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.763+0000 7fd09f7fe700 1 -- 172.21.15.98:0/2740772015 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0a000c6d0 con 0x7fd0a80c6d70 2023-10-06T05:07:36.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.763+0000 7fd0ad646700 1 -- 172.21.15.98:0/2740772015 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0a816a4a0 con 0x7fd0a80c6d70 2023-10-06T05:07:36.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.763+0000 7fd09f7fe700 1 -- 172.21.15.98:0/2740772015 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0a001e850 con 0x7fd0a80c6d70 2023-10-06T05:07:36.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.763+0000 7fd0ad646700 1 -- 172.21.15.98:0/2740772015 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0a80b21b0 con 0x7fd0a80c6d70 2023-10-06T05:07:36.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.764+0000 7fd0ad646700 1 -- 172.21.15.98:0/2740772015 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0a816ac80 con 0x7fd0a80c6d70 2023-10-06T05:07:36.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.764+0000 7fd09f7fe700 1 -- 172.21.15.98:0/2740772015 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fd0a001ec40 con 0x7fd0a80c6d70 2023-10-06T05:07:36.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.764+0000 7fd09f7fe700 1 --2- 172.21.15.98:0/2740772015 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fd088066f30 0x7fd0880693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:36.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.765+0000 7fd09f7fe700 1 -- 172.21.15.98:0/2740772015 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fd0a0085e00 con 0x7fd0a80c6d70 2023-10-06T05:07:36.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.765+0000 7fd0a6ffd700 1 --2- 172.21.15.98:0/2740772015 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fd088066f30 0x7fd0880693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:36.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.765+0000 7fd0a6ffd700 1 --2- 172.21.15.98:0/2740772015 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fd088066f30 0x7fd0880693e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd0a80c7af0 tx=0x7fd098001e50 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:36.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.769+0000 7fd09f7fe700 1 -- 172.21.15.98:0/2740772015 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd0a0052c20 con 0x7fd0a80c6d70 2023-10-06T05:07:36.955 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.954+0000 7fd0ad646700 1 -- 172.21.15.98:0/2740772015 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd0a805f230 con 0x7fd0a80c6d70 2023-10-06T05:07:36.955 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.954+0000 7fd09f7fe700 1 -- 172.21.15.98:0/2740772015 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v29) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd0a0017020 con 0x7fd0a80c6d70 2023-10-06T05:07:36.956 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2023-10-06T05:07:36.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.956+0000 7fd0ad646700 1 -- 172.21.15.98:0/2740772015 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fd088066f30 msgr2=0x7fd0880693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:36.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.956+0000 7fd0ad646700 1 --2- 172.21.15.98:0/2740772015 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fd088066f30 0x7fd0880693e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd0a80c7af0 tx=0x7fd098001e50 comp rx=0 tx=0).stop 2023-10-06T05:07:36.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.956+0000 7fd0ad646700 1 -- 172.21.15.98:0/2740772015 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0a80c6d70 msgr2=0x7fd0a80c71f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:36.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.956+0000 7fd0ad646700 1 --2- 172.21.15.98:0/2740772015 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0a80c6d70 0x7fd0a80c71f0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fd0a000c6a0 tx=0x7fd0a0015000 comp rx=0 tx=0).stop 2023-10-06T05:07:36.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.956+0000 7fd0ad646700 1 -- 172.21.15.98:0/2740772015 shutdown_connections 2023-10-06T05:07:36.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.956+0000 7fd0ad646700 1 --2- 172.21.15.98:0/2740772015 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fd088066f30 0x7fd0880693e0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:36.958 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.956+0000 7fd0ad646700 1 --2- 172.21.15.98:0/2740772015 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd0a80c6d70 0x7fd0a80c71f0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:36.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.956+0000 7fd0ad646700 1 --2- 172.21.15.98:0/2740772015 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd0940051c0 0x7fd0a81556c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:36.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.957+0000 7fd0ad646700 1 --2- 172.21.15.98:0/2740772015 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd0a80aafc0 0x7fd0a80b1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:36.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.957+0000 7fd0ad646700 1 -- 172.21.15.98:0/2740772015 >> 172.21.15.98:0/2740772015 conn(0x7fd0a80c9c90 msgr2=0x7fd0a80ab830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:36.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.957+0000 7fd0ad646700 1 -- 172.21.15.98:0/2740772015 shutdown_connections 2023-10-06T05:07:36.959 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:36.957+0000 7fd0ad646700 1 -- 172.21.15.98:0/2740772015 wait complete. 2023-10-06T05:07:36.970 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-06T05:07:37.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.086+0000 7fab53751700 1 Processor -- start 2023-10-06T05:07:37.087 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.086+0000 7fab53751700 1 -- start start 2023-10-06T05:07:37.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.087+0000 7fab53751700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fab4c1594d0 0x7fab4c0cb560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:37.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.087+0000 7fab53751700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab4c0cbec0 con 0x7fab4c1598a0 2023-10-06T05:07:37.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.087+0000 7fab53751700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab4c0cf5d0 con 0x7fab4c0cbaa0 2023-10-06T05:07:37.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.087+0000 7fab53751700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab4c0cf8b0 con 0x7fab4c1594d0 2023-10-06T05:07:37.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.087+0000 7fab50cec700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fab4c1598a0 0x7fab4c0ce7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:57556/0 (socket says 172.21.15.98:57556) 2023-10-06T05:07:37.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.087+0000 7fab50cec700 1 -- 172.21.15.98:0/3463540195 learned_addr learned my addr 172.21.15.98:0/3463540195 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:07:37.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.087+0000 7fab514ed700 1 --2- 172.21.15.98:0/3463540195 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fab4c1594d0 0x7fab4c0cb560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:37.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.087+0000 7fab43fff700 1 -- 172.21.15.98:0/3463540195 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1716832448 0 0) 0x7fab4c0cbec0 con 0x7fab4c1598a0 2023-10-06T05:07:37.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.087+0000 7fab43fff700 1 -- 172.21.15.98:0/3463540195 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab38003570 con 0x7fab4c1598a0 2023-10-06T05:07:37.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.087+0000 7fab43fff700 1 -- 172.21.15.98:0/3463540195 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 987186039 0 0) 0x7fab4c0cf5d0 con 0x7fab4c0cbaa0 2023-10-06T05:07:37.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.087+0000 7fab43fff700 1 -- 172.21.15.98:0/3463540195 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab4c0cbec0 con 0x7fab4c0cbaa0 2023-10-06T05:07:37.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.087+0000 7fab514ed700 1 -- 172.21.15.98:0/3463540195 >> v1:172.21.15.174:6789/0 conn(0x7fab4c0cbaa0 legacy=0x7fab4c0c5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:37.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.087+0000 7fab514ed700 1 -- 172.21.15.98:0/3463540195 >> v1:172.21.15.98:6789/0 conn(0x7fab4c1598a0 legacy=0x7fab4c0ce7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:37.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.087+0000 7fab514ed700 1 -- 172.21.15.98:0/3463540195 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab4c0cfb90 con 0x7fab4c1594d0 2023-10-06T05:07:37.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.087+0000 7fab514ed700 1 --2- 172.21.15.98:0/3463540195 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fab4c1594d0 0x7fab4c0cb560 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fab48009e90 tx=0x7fab48009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3a46d063151ba4b1 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:37.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.088+0000 7fab43fff700 1 -- 172.21.15.98:0/3463540195 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab4800e070 con 0x7fab4c1594d0 2023-10-06T05:07:37.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.088+0000 7fab43fff700 1 -- 172.21.15.98:0/3463540195 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab4800bb80 con 0x7fab4c1594d0 2023-10-06T05:07:37.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.088+0000 7fab43fff700 1 -- 172.21.15.98:0/3463540195 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab48012430 con 0x7fab4c1594d0 2023-10-06T05:07:37.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.088+0000 7fab53751700 1 -- 172.21.15.98:0/3463540195 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fab4c1594d0 msgr2=0x7fab4c0cb560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:37.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.088+0000 7fab53751700 1 --2- 172.21.15.98:0/3463540195 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fab4c1594d0 0x7fab4c0cb560 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fab48009e90 tx=0x7fab48009780 comp rx=0 tx=0).stop 2023-10-06T05:07:37.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.088+0000 7fab53751700 1 -- 172.21.15.98:0/3463540195 shutdown_connections 2023-10-06T05:07:37.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.088+0000 7fab53751700 1 --2- 172.21.15.98:0/3463540195 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fab4c1594d0 0x7fab4c0cb560 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:37.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.088+0000 7fab53751700 1 -- 172.21.15.98:0/3463540195 >> 172.21.15.98:0/3463540195 conn(0x7fab4c089e50 msgr2=0x7fab4c099770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:37.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.088+0000 7fab53751700 1 -- 172.21.15.98:0/3463540195 shutdown_connections 2023-10-06T05:07:37.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.088+0000 7fab53751700 1 -- 172.21.15.98:0/3463540195 wait complete. 2023-10-06T05:07:37.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.089+0000 7fab53751700 1 Processor -- start 2023-10-06T05:07:37.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.089+0000 7fab53751700 1 -- start start 2023-10-06T05:07:37.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.090+0000 7fab53751700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab4c0cbaa0 0x7fab4c14ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:37.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.090+0000 7fab53751700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fab4c1594d0 0x7fab4c069780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:37.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.090+0000 7fab53751700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fab4c1598a0 0x7fab4c069cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:37.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.090+0000 7fab53751700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab4c0c9f20 con 0x7fab4c0cbaa0 2023-10-06T05:07:37.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.090+0000 7fab53751700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab4c14f3b0 con 0x7fab4c1598a0 2023-10-06T05:07:37.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.090+0000 7fab53751700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab4c09ae30 con 0x7fab4c1594d0 2023-10-06T05:07:37.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.090+0000 7fab514ed700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab4c0cbaa0 0x7fab4c14ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:37.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.090+0000 7fab514ed700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab4c0cbaa0 0x7fab4c14ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45374/0 (socket says 172.21.15.98:45374) 2023-10-06T05:07:37.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.090+0000 7fab514ed700 1 -- 172.21.15.98:0/783074164 learned_addr learned my addr 172.21.15.98:0/783074164 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:37.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.090+0000 7fab50cec700 1 --2- 172.21.15.98:0/783074164 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fab4c1594d0 0x7fab4c069780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:37.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.090+0000 7fab514ed700 1 -- 172.21.15.98:0/783074164 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fab4c1594d0 msgr2=0x7fab4c069780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:37.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.090+0000 7fab514ed700 1 --2- 172.21.15.98:0/783074164 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fab4c1594d0 0x7fab4c069780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:37.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.090+0000 7fab514ed700 1 -- 172.21.15.98:0/783074164 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fab4c1598a0 msgr2=0x7fab4c069cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:37.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.090+0000 7fab51cee700 1 --2- 172.21.15.98:0/783074164 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fab4c1598a0 0x7fab4c069cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:37.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.090+0000 7fab514ed700 1 --2- 172.21.15.98:0/783074164 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fab4c1598a0 0x7fab4c069cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:37.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.091+0000 7fab514ed700 1 -- 172.21.15.98:0/783074164 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab48009a20 con 0x7fab4c0cbaa0 2023-10-06T05:07:37.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.091+0000 7fab514ed700 1 --2- 172.21.15.98:0/783074164 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab4c0cbaa0 0x7fab4c14ee70 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fab480097c0 tx=0x7fab48019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:37.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.091+0000 7fab417fa700 1 -- 172.21.15.98:0/783074164 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab48012e30 con 0x7fab4c0cbaa0 2023-10-06T05:07:37.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.091+0000 7fab417fa700 1 -- 172.21.15.98:0/783074164 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab4801b420 con 0x7fab4c0cbaa0 2023-10-06T05:07:37.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.091+0000 7fab53751700 1 -- 172.21.15.98:0/783074164 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab4c09b110 con 0x7fab4c0cbaa0 2023-10-06T05:07:37.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.091+0000 7fab417fa700 1 -- 172.21.15.98:0/783074164 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab4801b7e0 con 0x7fab4c0cbaa0 2023-10-06T05:07:37.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.091+0000 7fab53751700 1 -- 172.21.15.98:0/783074164 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab4c0c9bb0 con 0x7fab4c0cbaa0 2023-10-06T05:07:37.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.092+0000 7fab53751700 1 -- 172.21.15.98:0/783074164 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab4c0ca200 con 0x7fab4c0cbaa0 2023-10-06T05:07:37.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.092+0000 7fab417fa700 1 -- 172.21.15.98:0/783074164 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fab4801bb90 con 0x7fab4c0cbaa0 2023-10-06T05:07:37.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.093+0000 7fab417fa700 1 --2- 172.21.15.98:0/783074164 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fab30066f30 0x7fab300693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:37.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.093+0000 7fab417fa700 1 -- 172.21.15.98:0/783074164 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fab480865e0 con 0x7fab4c0cbaa0 2023-10-06T05:07:37.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.093+0000 7fab50cec700 1 --2- 172.21.15.98:0/783074164 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fab30066f30 0x7fab300693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:37.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.093+0000 7fab50cec700 1 --2- 172.21.15.98:0/783074164 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fab30066f30 0x7fab300693e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fab3c0058d0 tx=0x7fab3c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:37.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.097+0000 7fab417fa700 1 -- 172.21.15.98:0/783074164 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fab48056e90 con 0x7fab4c0cbaa0 2023-10-06T05:07:37.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.278+0000 7fab53751700 1 -- 172.21.15.98:0/783074164 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fab4c064a90 con 0x7fab4c0cbaa0 2023-10-06T05:07:37.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.278+0000 7fab417fa700 1 -- 172.21.15.98:0/783074164 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v29) v1 ==== 100+0+11 (secure 0 0 0) 0x7fab48056a20 con 0x7fab4c0cbaa0 2023-10-06T05:07:37.279 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2023-10-06T05:07:37.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.280+0000 7fab53751700 1 -- 172.21.15.98:0/783074164 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fab30066f30 msgr2=0x7fab300693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:37.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.280+0000 7fab53751700 1 --2- 172.21.15.98:0/783074164 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fab30066f30 0x7fab300693e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fab3c0058d0 tx=0x7fab3c009040 comp rx=0 tx=0).stop 2023-10-06T05:07:37.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.280+0000 7fab53751700 1 -- 172.21.15.98:0/783074164 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab4c0cbaa0 msgr2=0x7fab4c14ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:37.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.280+0000 7fab53751700 1 --2- 172.21.15.98:0/783074164 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab4c0cbaa0 0x7fab4c14ee70 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fab480097c0 tx=0x7fab48019000 comp rx=0 tx=0).stop 2023-10-06T05:07:37.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.280+0000 7fab53751700 1 -- 172.21.15.98:0/783074164 shutdown_connections 2023-10-06T05:07:37.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.280+0000 7fab53751700 1 --2- 172.21.15.98:0/783074164 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fab4c1598a0 0x7fab4c069cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:37.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.280+0000 7fab53751700 1 --2- 172.21.15.98:0/783074164 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fab30066f30 0x7fab300693e0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:37.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.280+0000 7fab53751700 1 --2- 172.21.15.98:0/783074164 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fab4c0cbaa0 0x7fab4c14ee70 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:37.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.280+0000 7fab53751700 1 --2- 172.21.15.98:0/783074164 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fab4c1594d0 0x7fab4c069780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:37.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.280+0000 7fab53751700 1 -- 172.21.15.98:0/783074164 >> 172.21.15.98:0/783074164 conn(0x7fab4c089e50 msgr2=0x7fab4c068370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:37.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.280+0000 7fab53751700 1 -- 172.21.15.98:0/783074164 shutdown_connections 2023-10-06T05:07:37.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.280+0000 7fab53751700 1 -- 172.21.15.98:0/783074164 wait complete. 2023-10-06T05:07:37.293 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-06T05:07:37.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.409+0000 7fe1cac13700 1 Processor -- start 2023-10-06T05:07:37.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.409+0000 7fe1cac13700 1 -- start start 2023-10-06T05:07:37.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.410+0000 7fe1cac13700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe1c40b0640 0x7fe1c40cecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:37.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.410+0000 7fe1cac13700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe1c40b0a10 0x7fe1c40cbee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:37.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.410+0000 7fe1cac13700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fe1c40cc420 0x7fe1c40c3b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:37.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.410+0000 7fe1cac13700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1c40ca790 con 0x7fe1c40b0a10 2023-10-06T05:07:37.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.410+0000 7fe1cac13700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1c40c53c0 con 0x7fe1c40cc420 2023-10-06T05:07:37.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.410+0000 7fe1cac13700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c40ccd00 con 0x7fe1c40b0640 2023-10-06T05:07:37.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.410+0000 7fe1c3fff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe1c40b0a10 0x7fe1c40cbee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:37.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.410+0000 7fe1c3fff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe1c40b0a10 0x7fe1c40cbee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45388/0 (socket says 172.21.15.98:45388) 2023-10-06T05:07:37.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.410+0000 7fe1c3fff700 1 -- 172.21.15.98:0/805488722 learned_addr learned my addr 172.21.15.98:0/805488722 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:37.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.410+0000 7fe1c89af700 1 --2- 172.21.15.98:0/805488722 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe1c40b0640 0x7fe1c40cecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:37.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.410+0000 7fe1c91b0700 1 --2- 172.21.15.98:0/805488722 >> v2:172.21.15.174:3300/0 conn(0x7fe1c40cc420 0x7fe1c40c3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:37.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.411+0000 7fe1c3fff700 1 -- 172.21.15.98:0/805488722 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe1c40b0640 msgr2=0x7fe1c40cecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:37.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.411+0000 7fe1c3fff700 1 --2- 172.21.15.98:0/805488722 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe1c40b0640 0x7fe1c40cecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:37.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.411+0000 7fe1c3fff700 1 -- 172.21.15.98:0/805488722 >> v2:172.21.15.174:3300/0 conn(0x7fe1c40cc420 msgr2=0x7fe1c40c3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:37.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.411+0000 7fe1c3fff700 1 --2- 172.21.15.98:0/805488722 >> v2:172.21.15.174:3300/0 conn(0x7fe1c40cc420 0x7fe1c40c3b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:37.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.411+0000 7fe1c3fff700 1 -- 172.21.15.98:0/805488722 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1c40ccfe0 con 0x7fe1c40b0a10 2023-10-06T05:07:37.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.411+0000 7fe1c89af700 1 --2- 172.21.15.98:0/805488722 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe1c40b0640 0x7fe1c40cecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:07:37.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.411+0000 7fe1c91b0700 1 --2- 172.21.15.98:0/805488722 >> v2:172.21.15.174:3300/0 conn(0x7fe1c40cc420 0x7fe1c40c3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:07:37.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.411+0000 7fe1c3fff700 1 --2- 172.21.15.98:0/805488722 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1c40b0a10 0x7fe1c40cbee0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fe1c40c3a70 tx=0x7fe1b8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db1784a0ddae2ac3 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:37.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.411+0000 7fe1c37fe700 1 -- 172.21.15.98:0/805488722 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1b800b860 con 0x7fe1c40b0a10 2023-10-06T05:07:37.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.411+0000 7fe1c37fe700 1 -- 172.21.15.98:0/805488722 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1b800b9c0 con 0x7fe1c40b0a10 2023-10-06T05:07:37.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.411+0000 7fe1c37fe700 1 -- 172.21.15.98:0/805488722 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1b800bd60 con 0x7fe1c40b0a10 2023-10-06T05:07:37.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.411+0000 7fe1cac13700 1 -- 172.21.15.98:0/805488722 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1c40b0a10 msgr2=0x7fe1c40cbee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:37.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.411+0000 7fe1cac13700 1 --2- 172.21.15.98:0/805488722 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1c40b0a10 0x7fe1c40cbee0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fe1c40c3a70 tx=0x7fe1b8009250 comp rx=0 tx=0).stop 2023-10-06T05:07:37.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.411+0000 7fe1cac13700 1 -- 172.21.15.98:0/805488722 shutdown_connections 2023-10-06T05:07:37.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.411+0000 7fe1cac13700 1 --2- 172.21.15.98:0/805488722 >> v2:172.21.15.174:3300/0 conn(0x7fe1c40cc420 0x7fe1c40c3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:37.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.411+0000 7fe1cac13700 1 --2- 172.21.15.98:0/805488722 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe1c40b0640 0x7fe1c40cecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:37.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.411+0000 7fe1cac13700 1 --2- 172.21.15.98:0/805488722 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1c40b0a10 0x7fe1c40cbee0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:37.419 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.411+0000 7fe1cac13700 1 -- 172.21.15.98:0/805488722 >> 172.21.15.98:0/805488722 conn(0x7fe1c4087fc0 msgr2=0x7fe1c40ce310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:37.419 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.411+0000 7fe1cac13700 1 -- 172.21.15.98:0/805488722 shutdown_connections 2023-10-06T05:07:37.419 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.412+0000 7fe1cac13700 1 -- 172.21.15.98:0/805488722 wait complete. 2023-10-06T05:07:37.419 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.413+0000 7fe1cac13700 1 Processor -- start 2023-10-06T05:07:37.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.413+0000 7fe1cac13700 1 -- start start 2023-10-06T05:07:37.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.413+0000 7fe1cac13700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe1c40b0640 0x7fe1c4158220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:37.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.413+0000 7fe1cac13700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe1c40b0a10 0x7fe1c4169f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:37.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.413+0000 7fe1cac13700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1c40cc420 0x7fe1c416a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:37.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.413+0000 7fe1cac13700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c40686c0 con 0x7fe1c40cc420 2023-10-06T05:07:37.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.413+0000 7fe1cac13700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c4068c80 con 0x7fe1c40b0640 2023-10-06T05:07:37.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.413+0000 7fe1cac13700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c416aaf0 con 0x7fe1c40b0a10 2023-10-06T05:07:37.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.414+0000 7fe1c91b0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1c40cc420 0x7fe1c416a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:37.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.414+0000 7fe1c91b0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1c40cc420 0x7fe1c416a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45404/0 (socket says 172.21.15.98:45404) 2023-10-06T05:07:37.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.414+0000 7fe1c91b0700 1 -- 172.21.15.98:0/3196328330 learned_addr learned my addr 172.21.15.98:0/3196328330 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:37.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.414+0000 7fe1c3fff700 1 --2- 172.21.15.98:0/3196328330 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe1c40b0a10 0x7fe1c4169f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:37.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.414+0000 7fe1c89af700 1 --2- 172.21.15.98:0/3196328330 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe1c40b0640 0x7fe1c4158220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:37.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.414+0000 7fe1c91b0700 1 -- 172.21.15.98:0/3196328330 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe1c40b0a10 msgr2=0x7fe1c4169f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:37.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.414+0000 7fe1c91b0700 1 --2- 172.21.15.98:0/3196328330 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe1c40b0a10 0x7fe1c4169f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:37.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.414+0000 7fe1c91b0700 1 -- 172.21.15.98:0/3196328330 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe1c40b0640 msgr2=0x7fe1c4158220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:37.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.414+0000 7fe1c91b0700 1 --2- 172.21.15.98:0/3196328330 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe1c40b0640 0x7fe1c4158220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:37.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.414+0000 7fe1c91b0700 1 -- 172.21.15.98:0/3196328330 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1b80094f0 con 0x7fe1c40cc420 2023-10-06T05:07:37.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.414+0000 7fe1c91b0700 1 --2- 172.21.15.98:0/3196328330 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1c40cc420 0x7fe1c416a4d0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fe1bc00e8d0 tx=0x7fe1bc008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:37.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.414+0000 7fe1c0ff9700 1 -- 172.21.15.98:0/3196328330 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1bc00c070 con 0x7fe1c40cc420 2023-10-06T05:07:37.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.414+0000 7fe1c0ff9700 1 -- 172.21.15.98:0/3196328330 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1bc013410 con 0x7fe1c40cc420 2023-10-06T05:07:37.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.414+0000 7fe1cac13700 1 -- 172.21.15.98:0/3196328330 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1c416add0 con 0x7fe1c40cc420 2023-10-06T05:07:37.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.414+0000 7fe1c0ff9700 1 -- 172.21.15.98:0/3196328330 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1bc0137b0 con 0x7fe1c40cc420 2023-10-06T05:07:37.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.414+0000 7fe1cac13700 1 -- 172.21.15.98:0/3196328330 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1c4158760 con 0x7fe1c40cc420 2023-10-06T05:07:37.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.415+0000 7fe1cac13700 1 -- 172.21.15.98:0/3196328330 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1c4068e10 con 0x7fe1c40cc420 2023-10-06T05:07:37.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.416+0000 7fe1c0ff9700 1 -- 172.21.15.98:0/3196328330 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fe1bc013ba0 con 0x7fe1c40cc420 2023-10-06T05:07:37.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.416+0000 7fe1c0ff9700 1 --2- 172.21.15.98:0/3196328330 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fe1b0066f30 0x7fe1b00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:37.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.416+0000 7fe1c0ff9700 1 -- 172.21.15.98:0/3196328330 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fe1bc084550 con 0x7fe1c40cc420 2023-10-06T05:07:37.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.416+0000 7fe1c89af700 1 --2- 172.21.15.98:0/3196328330 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fe1b0066f30 0x7fe1b00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:37.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.416+0000 7fe1c89af700 1 --2- 172.21.15.98:0/3196328330 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fe1b0066f30 0x7fe1b00693e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe1b4007700 tx=0x7fe1b4009980 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:37.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.420+0000 7fe1c0ff9700 1 -- 172.21.15.98:0/3196328330 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe1bc051370 con 0x7fe1c40cc420 2023-10-06T05:07:37.606 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.605+0000 7fe1cac13700 1 -- 172.21.15.98:0/3196328330 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fe1c4064a90 con 0x7fe1c40cc420 2023-10-06T05:07:37.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.606+0000 7fe1c0ff9700 1 -- 172.21.15.98:0/3196328330 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v29) v1 ==== 103+0+11 (secure 0 0 0) 0x7fe1bc054990 con 0x7fe1c40cc420 2023-10-06T05:07:37.607 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2023-10-06T05:07:37.609 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.607+0000 7fe1cac13700 1 -- 172.21.15.98:0/3196328330 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fe1b0066f30 msgr2=0x7fe1b00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:37.609 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.607+0000 7fe1cac13700 1 --2- 172.21.15.98:0/3196328330 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fe1b0066f30 0x7fe1b00693e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe1b4007700 tx=0x7fe1b4009980 comp rx=0 tx=0).stop 2023-10-06T05:07:37.609 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.607+0000 7fe1cac13700 1 -- 172.21.15.98:0/3196328330 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1c40cc420 msgr2=0x7fe1c416a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:37.609 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.607+0000 7fe1cac13700 1 --2- 172.21.15.98:0/3196328330 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1c40cc420 0x7fe1c416a4d0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fe1bc00e8d0 tx=0x7fe1bc008040 comp rx=0 tx=0).stop 2023-10-06T05:07:37.609 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.607+0000 7fe1cac13700 1 -- 172.21.15.98:0/3196328330 shutdown_connections 2023-10-06T05:07:37.609 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.608+0000 7fe1cac13700 1 --2- 172.21.15.98:0/3196328330 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fe1b0066f30 0x7fe1b00693e0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:37.610 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.608+0000 7fe1cac13700 1 --2- 172.21.15.98:0/3196328330 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1c40cc420 0x7fe1c416a4d0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:37.610 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.608+0000 7fe1cac13700 1 --2- 172.21.15.98:0/3196328330 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe1c40b0640 0x7fe1c4158220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:37.610 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.608+0000 7fe1cac13700 1 --2- 172.21.15.98:0/3196328330 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe1c40b0a10 0x7fe1c4169f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:37.610 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.608+0000 7fe1cac13700 1 -- 172.21.15.98:0/3196328330 >> 172.21.15.98:0/3196328330 conn(0x7fe1c4087fc0 msgr2=0x7fe1c4156cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:37.610 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.608+0000 7fe1cac13700 1 -- 172.21.15.98:0/3196328330 shutdown_connections 2023-10-06T05:07:37.610 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.608+0000 7fe1cac13700 1 -- 172.21.15.98:0/3196328330 wait complete. 2023-10-06T05:07:37.650 INFO:tasks.cephfs_test_runner:Starting test: test_flush (tasks.cephfs.test_readahead.TestReadahead) 2023-10-06T05:07:37.650 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_readahead.TestReadahead.test_flush' 2023-10-06T05:07:37.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.765+0000 7efd0cd6a700 1 Processor -- start 2023-10-06T05:07:37.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.765+0000 7efd0cd6a700 1 -- start start 2023-10-06T05:07:37.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.766+0000 7efd0cd6a700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efd080b49a0 0x7efd080aac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:37.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.766+0000 7efd0cd6a700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7efd080b0760 0x7efd080b1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:37.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.766+0000 7efd0cd6a700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd080b0c60 con 0x7efd08159560 2023-10-06T05:07:37.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.766+0000 7efd0cd6a700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd080b0f40 con 0x7efd080b0760 2023-10-06T05:07:37.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.766+0000 7efd0cd6a700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd080b1220 con 0x7efd080b49a0 2023-10-06T05:07:37.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.766+0000 7efd05d9b700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efd080b49a0 0x7efd080aac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:37.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.766+0000 7efd06d9d700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7efd080b0760 0x7efd080b1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:37.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.766+0000 7efd05d9b700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efd080b49a0 0x7efd080aac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.98:51366/0 (socket says 172.21.15.98:51366) 2023-10-06T05:07:37.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.766+0000 7efd05d9b700 1 -- 172.21.15.98:0/2261384931 learned_addr learned my addr 172.21.15.98:0/2261384931 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:37.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.766+0000 7efd05d9b700 1 -- 172.21.15.98:0/2261384931 >> v2:172.21.15.174:3300/0 conn(0x7efd080b0760 msgr2=0x7efd080b1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:37.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.766+0000 7efd05d9b700 1 --2- 172.21.15.98:0/2261384931 >> v2:172.21.15.174:3300/0 conn(0x7efd080b0760 0x7efd080b1bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:37.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.766+0000 7efd05d9b700 1 -- 172.21.15.98:0/2261384931 >> v1:172.21.15.174:6789/0 conn(0x7efd08159560 legacy=0x7efd080aa530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:37.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.766+0000 7efd05d9b700 1 -- 172.21.15.98:0/2261384931 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd080b1500 con 0x7efd080b49a0 2023-10-06T05:07:37.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.766+0000 7efd0559a700 1 -- 172.21.15.98:0/2261384931 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 727833059 0 0) 0x7efd080b0c60 con 0x7efd08159560 2023-10-06T05:07:37.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.766+0000 7efd06d9d700 1 --2- 172.21.15.98:0/2261384931 >> v2:172.21.15.174:3300/0 conn(0x7efd080b0760 0x7efd080b1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T05:07:37.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.767+0000 7efd05d9b700 1 --2- 172.21.15.98:0/2261384931 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efd080b49a0 0x7efd080aac40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7efcfc0099b0 tx=0x7efcfc009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c439ae3a3cd7e7e server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:37.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.767+0000 7efd0559a700 1 -- 172.21.15.98:0/2261384931 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcfc009ea0 con 0x7efd080b49a0 2023-10-06T05:07:37.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.767+0000 7efd0559a700 1 -- 172.21.15.98:0/2261384931 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcfc00b860 con 0x7efd080b49a0 2023-10-06T05:07:37.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.767+0000 7efd0559a700 1 -- 172.21.15.98:0/2261384931 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcfc00bc00 con 0x7efd080b49a0 2023-10-06T05:07:37.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.767+0000 7efd0cd6a700 1 -- 172.21.15.98:0/2261384931 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efd080b49a0 msgr2=0x7efd080aac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:37.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.767+0000 7efd0cd6a700 1 --2- 172.21.15.98:0/2261384931 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efd080b49a0 0x7efd080aac40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7efcfc0099b0 tx=0x7efcfc009250 comp rx=0 tx=0).stop 2023-10-06T05:07:37.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.767+0000 7efd0cd6a700 1 -- 172.21.15.98:0/2261384931 shutdown_connections 2023-10-06T05:07:37.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.767+0000 7efd0cd6a700 1 --2- 172.21.15.98:0/2261384931 >> v2:172.21.15.174:3300/0 conn(0x7efd080b0760 0x7efd080b1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:37.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.767+0000 7efd0cd6a700 1 --2- 172.21.15.98:0/2261384931 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efd080b49a0 0x7efd080aac40 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:37.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.767+0000 7efd0cd6a700 1 -- 172.21.15.98:0/2261384931 >> 172.21.15.98:0/2261384931 conn(0x7efd080997f0 msgr2=0x7efd0806f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:37.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.767+0000 7efd0cd6a700 1 -- 172.21.15.98:0/2261384931 shutdown_connections 2023-10-06T05:07:37.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.767+0000 7efd0cd6a700 1 -- 172.21.15.98:0/2261384931 wait complete. 2023-10-06T05:07:37.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.768+0000 7efd0cd6a700 1 Processor -- start 2023-10-06T05:07:37.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.768+0000 7efd0cd6a700 1 -- start start 2023-10-06T05:07:37.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.769+0000 7efd0cd6a700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efd080b0760 0x7efd0815b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:37.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.769+0000 7efd0cd6a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd080b49a0 0x7efd0809d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:37.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.769+0000 7efd0cd6a700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efd08159560 0x7efd0815c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:37.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.769+0000 7efd0cd6a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd0816a6f0 con 0x7efd080b49a0 2023-10-06T05:07:37.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.769+0000 7efd0cd6a700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd08159930 con 0x7efd08159560 2023-10-06T05:07:37.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.769+0000 7efd0cd6a700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd08169ae0 con 0x7efd080b0760 2023-10-06T05:07:37.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.769+0000 7efd05d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd080b49a0 0x7efd0809d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:37.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.769+0000 7efd05d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd080b49a0 0x7efd0809d610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45416/0 (socket says 172.21.15.98:45416) 2023-10-06T05:07:37.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.769+0000 7efd05d9b700 1 -- 172.21.15.98:0/1802157029 learned_addr learned my addr 172.21.15.98:0/1802157029 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:37.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.769+0000 7efd06d9d700 1 --2- 172.21.15.98:0/1802157029 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efd08159560 0x7efd0815c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:37.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.769+0000 7efd0659c700 1 --2- 172.21.15.98:0/1802157029 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efd080b0760 0x7efd0815b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:37.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.770+0000 7efd05d9b700 1 -- 172.21.15.98:0/1802157029 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efd080b0760 msgr2=0x7efd0815b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:37.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.770+0000 7efd05d9b700 1 --2- 172.21.15.98:0/1802157029 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efd080b0760 0x7efd0815b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:37.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.770+0000 7efd05d9b700 1 -- 172.21.15.98:0/1802157029 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efd08159560 msgr2=0x7efd0815c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:37.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.770+0000 7efd05d9b700 1 --2- 172.21.15.98:0/1802157029 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efd08159560 0x7efd0815c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:37.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.770+0000 7efd05d9b700 1 -- 172.21.15.98:0/1802157029 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd00009420 con 0x7efd080b49a0 2023-10-06T05:07:37.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.770+0000 7efd05d9b700 1 --2- 172.21.15.98:0/1802157029 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd080b49a0 0x7efd0809d610 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7efd080b1360 tx=0x7efcfc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:37.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.770+0000 7efcf67fc700 1 -- 172.21.15.98:0/1802157029 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcfc00bbe0 con 0x7efd080b49a0 2023-10-06T05:07:37.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.770+0000 7efcf67fc700 1 -- 172.21.15.98:0/1802157029 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcfc011420 con 0x7efd080b49a0 2023-10-06T05:07:37.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.770+0000 7efd0cd6a700 1 -- 172.21.15.98:0/1802157029 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcfc0094f0 con 0x7efd080b49a0 2023-10-06T05:07:37.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.770+0000 7efcf67fc700 1 -- 172.21.15.98:0/1802157029 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcfc0117c0 con 0x7efd080b49a0 2023-10-06T05:07:37.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.770+0000 7efd0cd6a700 1 -- 172.21.15.98:0/1802157029 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd0816a0a0 con 0x7efd080b49a0 2023-10-06T05:07:37.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.771+0000 7efd0cd6a700 1 -- 172.21.15.98:0/1802157029 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd08064a90 con 0x7efd080b49a0 2023-10-06T05:07:37.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.771+0000 7efcf67fc700 1 -- 172.21.15.98:0/1802157029 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7efcfc011b70 con 0x7efd080b49a0 2023-10-06T05:07:37.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.772+0000 7efcf67fc700 1 --2- 172.21.15.98:0/1802157029 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7efce8066f30 0x7efce80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:37.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.772+0000 7efd0659c700 1 --2- 172.21.15.98:0/1802157029 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7efce8066f30 0x7efce80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:37.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.772+0000 7efcf67fc700 1 -- 172.21.15.98:0/1802157029 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6809+0+0 (secure 0 0 0) 0x7efcfc086340 con 0x7efd080b49a0 2023-10-06T05:07:37.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.772+0000 7efd0659c700 1 --2- 172.21.15.98:0/1802157029 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7efce8066f30 0x7efce80693e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7efcf800e160 tx=0x7efcf8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:37.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:37.776+0000 7efcf67fc700 1 -- 172.21.15.98:0/1802157029 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7efcfc053160 con 0x7efd080b49a0 2023-10-06T05:07:38.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:38.040+0000 7efd0cd6a700 1 -- 172.21.15.98:0/1802157029 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_readahead.TestReadahead.test_flush"]} v 0) v1 -- 0x7efd080b17e0 con 0x7efd080b49a0 2023-10-06T05:07:38.947 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:38.946+0000 7efcf67fc700 1 -- 172.21.15.98:0/1802157029 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_readahead.TestReadahead.test_flush"]}]=0 v63) v1 ==== 134+0+0 (secure 0 0 0) 0x7efcfc056780 con 0x7efd080b49a0 2023-10-06T05:07:38.950 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:38.949+0000 7efd0cd6a700 1 -- 172.21.15.98:0/1802157029 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7efce8066f30 msgr2=0x7efce80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:38.950 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:38.949+0000 7efd0cd6a700 1 --2- 172.21.15.98:0/1802157029 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7efce8066f30 0x7efce80693e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7efcf800e160 tx=0x7efcf8009040 comp rx=0 tx=0).stop 2023-10-06T05:07:38.951 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:38.949+0000 7efd0cd6a700 1 -- 172.21.15.98:0/1802157029 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd080b49a0 msgr2=0x7efd0809d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:38.951 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:38.949+0000 7efd0cd6a700 1 --2- 172.21.15.98:0/1802157029 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd080b49a0 0x7efd0809d610 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7efd080b1360 tx=0x7efcfc017000 comp rx=0 tx=0).stop 2023-10-06T05:07:38.951 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:38.949+0000 7efd0cd6a700 1 -- 172.21.15.98:0/1802157029 shutdown_connections 2023-10-06T05:07:38.951 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:38.949+0000 7efd0cd6a700 1 --2- 172.21.15.98:0/1802157029 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efd08159560 0x7efd0815c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:38.951 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:38.949+0000 7efd0cd6a700 1 --2- 172.21.15.98:0/1802157029 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efd080b0760 0x7efd0815b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:38.951 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:38.949+0000 7efd0cd6a700 1 --2- 172.21.15.98:0/1802157029 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7efce8066f30 0x7efce80693e0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:38.951 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:38.949+0000 7efd0cd6a700 1 --2- 172.21.15.98:0/1802157029 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd080b49a0 0x7efd0809d610 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:38.951 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:38.949+0000 7efd0cd6a700 1 -- 172.21.15.98:0/1802157029 >> 172.21.15.98:0/1802157029 conn(0x7efd080997f0 msgr2=0x7efd08069be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:38.951 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:38.949+0000 7efd0cd6a700 1 -- 172.21.15.98:0/1802157029 shutdown_connections 2023-10-06T05:07:38.951 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:38.949+0000 7efd0cd6a700 1 -- 172.21.15.98:0/1802157029 wait complete. 2023-10-06T05:07:38.963 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-06T05:07:39.081 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.079+0000 7fccb2549700 1 Processor -- start 2023-10-06T05:07:39.081 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.080+0000 7fccb2549700 1 -- start start 2023-10-06T05:07:39.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.080+0000 7fccb2549700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fccac0cbc70 0x7fccac0ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:39.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.080+0000 7fccb2549700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccac0cf5d0 con 0x7fccac1594d0 2023-10-06T05:07:39.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.080+0000 7fccb2549700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccac0cf8b0 con 0x7fccac1598a0 2023-10-06T05:07:39.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.080+0000 7fccb2549700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccac0cfb90 con 0x7fccac0cbc70 2023-10-06T05:07:39.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.080+0000 7fccabfff700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fccac1594d0 0x7fccac0cb560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:57562/0 (socket says 172.21.15.98:57562) 2023-10-06T05:07:39.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.080+0000 7fccabfff700 1 -- 172.21.15.98:0/1722097293 learned_addr learned my addr 172.21.15.98:0/1722097293 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:07:39.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.081+0000 7fccaaffd700 1 -- 172.21.15.98:0/1722097293 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2249607741 0 0) 0x7fccac0cf5d0 con 0x7fccac1594d0 2023-10-06T05:07:39.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.081+0000 7fccaaffd700 1 -- 172.21.15.98:0/1722097293 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc94003570 con 0x7fccac1594d0 2023-10-06T05:07:39.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.081+0000 7fccb0ae6700 1 --2- 172.21.15.98:0/1722097293 >> v2:172.21.15.174:3300/0 conn(0x7fccac0cbc70 0x7fccac0ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:39.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.081+0000 7fccaaffd700 1 -- 172.21.15.98:0/1722097293 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3795508372 0 0) 0x7fcc94003570 con 0x7fccac1594d0 2023-10-06T05:07:39.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.081+0000 7fccaaffd700 1 -- 172.21.15.98:0/1722097293 >> v2:172.21.15.174:3300/0 conn(0x7fccac0cbc70 msgr2=0x7fccac0ceef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:39.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.081+0000 7fccaaffd700 1 --2- 172.21.15.98:0/1722097293 >> v2:172.21.15.174:3300/0 conn(0x7fccac0cbc70 0x7fccac0ceef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:39.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.081+0000 7fccaaffd700 1 -- 172.21.15.98:0/1722097293 >> v1:172.21.15.174:6789/0 conn(0x7fccac1598a0 legacy=0x7fccac0ce7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:39.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.081+0000 7fccaaffd700 1 -- 172.21.15.98:0/1722097293 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccac0cfe70 con 0x7fccac1594d0 2023-10-06T05:07:39.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.081+0000 7fccaaffd700 1 -- 172.21.15.98:0/1722097293 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1543991876 0 0) 0x7fcc9c001470 con 0x7fccac1594d0 2023-10-06T05:07:39.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.081+0000 7fccaaffd700 1 -- 172.21.15.98:0/1722097293 >> v1:172.21.15.98:6789/0 conn(0x7fccac1594d0 legacy=0x7fccac0cb560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:39.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.081+0000 7fccaaffd700 1 --2- 172.21.15.98:0/1722097293 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc94003a60 0x7fcc94003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:39.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.081+0000 7fccaaffd700 1 --2- 172.21.15.98:0/1722097293 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcc94004810 0x7fcc94004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:39.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.081+0000 7fccaaffd700 1 --2- 172.21.15.98:0/1722097293 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcc940052e0 0x7fcc9400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:39.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.081+0000 7fccaaffd700 1 -- 172.21.15.98:0/1722097293 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccac0cfb90 con 0x7fcc94003a60 2023-10-06T05:07:39.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.081+0000 7fccaaffd700 1 -- 172.21.15.98:0/1722097293 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc9400c190 con 0x7fcc94004810 2023-10-06T05:07:39.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.081+0000 7fccaaffd700 1 -- 172.21.15.98:0/1722097293 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc9400c350 con 0x7fcc940052e0 2023-10-06T05:07:39.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.081+0000 7fccabfff700 1 --2- 172.21.15.98:0/1722097293 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc94003a60 0x7fcc94003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:39.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.081+0000 7fccab7fe700 1 --2- 172.21.15.98:0/1722097293 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcc94004810 0x7fcc94004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:39.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.081+0000 7fccb0ae6700 1 --2- 172.21.15.98:0/1722097293 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcc940052e0 0x7fcc9400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:39.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.081+0000 7fccabfff700 1 -- 172.21.15.98:0/1722097293 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcc940052e0 msgr2=0x7fcc9400ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:39.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.081+0000 7fccabfff700 1 --2- 172.21.15.98:0/1722097293 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcc940052e0 0x7fcc9400ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:39.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.081+0000 7fccabfff700 1 -- 172.21.15.98:0/1722097293 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcc94004810 msgr2=0x7fcc94004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:39.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.081+0000 7fccabfff700 1 --2- 172.21.15.98:0/1722097293 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcc94004810 0x7fcc94004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:39.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.081+0000 7fccabfff700 1 -- 172.21.15.98:0/1722097293 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc94003570 con 0x7fcc94003a60 2023-10-06T05:07:39.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.081+0000 7fccb0ae6700 1 -- 172.21.15.98:0/1722097293 reap_dead start 2023-10-06T05:07:39.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.081+0000 7fccab7fe700 1 --2- 172.21.15.98:0/1722097293 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcc94004810 0x7fcc94004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:07:39.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.082+0000 7fccabfff700 1 --2- 172.21.15.98:0/1722097293 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc94003a60 0x7fcc94003e70 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fcc9c00c3c0 tx=0x7fcc9c001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=368a74069d65989f server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:39.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.082+0000 7fccaaffd700 1 -- 172.21.15.98:0/1722097293 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc9c00cc90 con 0x7fcc94003a60 2023-10-06T05:07:39.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.082+0000 7fccaaffd700 1 -- 172.21.15.98:0/1722097293 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc9c00ce10 con 0x7fcc94003a60 2023-10-06T05:07:39.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.082+0000 7fccaaffd700 1 -- 172.21.15.98:0/1722097293 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc9c005b70 con 0x7fcc94003a60 2023-10-06T05:07:39.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.082+0000 7fccb2549700 1 -- 172.21.15.98:0/1722097293 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc94003a60 msgr2=0x7fcc94003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:39.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.082+0000 7fccb2549700 1 --2- 172.21.15.98:0/1722097293 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc94003a60 0x7fcc94003e70 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fcc9c00c3c0 tx=0x7fcc9c001e50 comp rx=0 tx=0).stop 2023-10-06T05:07:39.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.082+0000 7fccb2549700 1 -- 172.21.15.98:0/1722097293 shutdown_connections 2023-10-06T05:07:39.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.082+0000 7fccb2549700 1 --2- 172.21.15.98:0/1722097293 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcc94003a60 0x7fcc94003e70 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:39.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.082+0000 7fccb2549700 1 -- 172.21.15.98:0/1722097293 >> 172.21.15.98:0/1722097293 conn(0x7fccac089e50 msgr2=0x7fccac099770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:39.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.082+0000 7fccb2549700 1 -- 172.21.15.98:0/1722097293 shutdown_connections 2023-10-06T05:07:39.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.082+0000 7fccb2549700 1 -- 172.21.15.98:0/1722097293 wait complete. 2023-10-06T05:07:39.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.083+0000 7fccb2549700 1 Processor -- start 2023-10-06T05:07:39.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.084+0000 7fccb2549700 1 -- start start 2023-10-06T05:07:39.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.084+0000 7fccb2549700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcc94003a60 0x7fccac14afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:39.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.084+0000 7fccb2549700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fccac0c88e0 0x7fccac0c8d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:39.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.084+0000 7fccb2549700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fccac0b3ec0 0x7fccac0b4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:39.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.084+0000 7fccb2549700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccac14dae0 con 0x7fccac0c88e0 2023-10-06T05:07:39.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.084+0000 7fccb2549700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccac0b48a0 con 0x7fcc94003a60 2023-10-06T05:07:39.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.084+0000 7fccb2549700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccac16b660 con 0x7fccac0b3ec0 2023-10-06T05:07:39.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.084+0000 7fccab7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fccac0c88e0 0x7fccac0c8d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:39.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.084+0000 7fccab7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fccac0c88e0 0x7fccac0c8d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45436/0 (socket says 172.21.15.98:45436) 2023-10-06T05:07:39.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.084+0000 7fccab7fe700 1 -- 172.21.15.98:0/2995369476 learned_addr learned my addr 172.21.15.98:0/2995369476 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:39.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.085+0000 7fccb0ae6700 1 --2- 172.21.15.98:0/2995369476 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fccac0b3ec0 0x7fccac0b4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:39.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.085+0000 7fccab7fe700 1 -- 172.21.15.98:0/2995369476 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fccac0b3ec0 msgr2=0x7fccac0b4360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:39.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.085+0000 7fccabfff700 1 --2- 172.21.15.98:0/2995369476 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcc94003a60 0x7fccac14afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:39.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.085+0000 7fccab7fe700 1 --2- 172.21.15.98:0/2995369476 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fccac0b3ec0 0x7fccac0b4360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:39.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.085+0000 7fccab7fe700 1 -- 172.21.15.98:0/2995369476 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcc94003a60 msgr2=0x7fccac14afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:39.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.085+0000 7fccab7fe700 1 --2- 172.21.15.98:0/2995369476 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcc94003a60 0x7fccac14afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:39.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.085+0000 7fccab7fe700 1 -- 172.21.15.98:0/2995369476 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccac16b380 con 0x7fccac0c88e0 2023-10-06T05:07:39.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.085+0000 7fccab7fe700 1 --2- 172.21.15.98:0/2995369476 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fccac0c88e0 0x7fccac0c8d50 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fcca000e510 tx=0x7fcca0009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:39.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.085+0000 7fcc93fff700 1 -- 172.21.15.98:0/2995369476 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca00114f0 con 0x7fccac0c88e0 2023-10-06T05:07:39.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.085+0000 7fcc93fff700 1 -- 172.21.15.98:0/2995369476 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcca0011650 con 0x7fccac0c88e0 2023-10-06T05:07:39.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.085+0000 7fccb2549700 1 -- 172.21.15.98:0/2995369476 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccac0b33e0 con 0x7fccac0c88e0 2023-10-06T05:07:39.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.085+0000 7fcc93fff700 1 -- 172.21.15.98:0/2995369476 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca00119f0 con 0x7fccac0c88e0 2023-10-06T05:07:39.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.085+0000 7fccb2549700 1 -- 172.21.15.98:0/2995369476 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccac0b39a0 con 0x7fccac0c88e0 2023-10-06T05:07:39.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.086+0000 7fccb2549700 1 -- 172.21.15.98:0/2995369476 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccac064a90 con 0x7fccac0c88e0 2023-10-06T05:07:39.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.086+0000 7fcc93fff700 1 -- 172.21.15.98:0/2995369476 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fcca0014480 con 0x7fccac0c88e0 2023-10-06T05:07:39.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.087+0000 7fcc93fff700 1 --2- 172.21.15.98:0/2995369476 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fcc98066f30 0x7fcc980693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:39.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.087+0000 7fcc93fff700 1 -- 172.21.15.98:0/2995369476 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6867+0+0 (secure 0 0 0) 0x7fcca0084220 con 0x7fccac0c88e0 2023-10-06T05:07:39.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.087+0000 7fccabfff700 1 --2- 172.21.15.98:0/2995369476 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fcc98066f30 0x7fcc980693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:39.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.087+0000 7fccabfff700 1 --2- 172.21.15.98:0/2995369476 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fcc98066f30 0x7fcc980693e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fccac15d250 tx=0x7fcc9c015000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:39.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.091+0000 7fcc93fff700 1 -- 172.21.15.98:0/2995369476 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcca004d5a0 con 0x7fccac0c88e0 2023-10-06T05:07:39.274 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.273+0000 7fccb2549700 1 -- 172.21.15.98:0/2995369476 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fccac0cb560 con 0x7fccac0c88e0 2023-10-06T05:07:39.278 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.277+0000 7fcc93fff700 1 -- 172.21.15.98:0/2995369476 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7fcca00510b0 con 0x7fccac0c88e0 2023-10-06T05:07:39.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.279+0000 7fccb2549700 1 -- 172.21.15.98:0/2995369476 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fcc98066f30 msgr2=0x7fcc980693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:39.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.279+0000 7fccb2549700 1 --2- 172.21.15.98:0/2995369476 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fcc98066f30 0x7fcc980693e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fccac15d250 tx=0x7fcc9c015000 comp rx=0 tx=0).stop 2023-10-06T05:07:39.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.279+0000 7fccb2549700 1 -- 172.21.15.98:0/2995369476 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fccac0c88e0 msgr2=0x7fccac0c8d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:39.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.279+0000 7fccb2549700 1 --2- 172.21.15.98:0/2995369476 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fccac0c88e0 0x7fccac0c8d50 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fcca000e510 tx=0x7fcca0009040 comp rx=0 tx=0).stop 2023-10-06T05:07:39.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.279+0000 7fccb2549700 1 -- 172.21.15.98:0/2995369476 shutdown_connections 2023-10-06T05:07:39.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.279+0000 7fccb2549700 1 --2- 172.21.15.98:0/2995369476 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fccac0b3ec0 0x7fccac0b4360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:39.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.279+0000 7fccb2549700 1 --2- 172.21.15.98:0/2995369476 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcc94003a60 0x7fccac14afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:39.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.279+0000 7fccb2549700 1 --2- 172.21.15.98:0/2995369476 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fcc98066f30 0x7fcc980693e0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:39.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.279+0000 7fccb2549700 1 --2- 172.21.15.98:0/2995369476 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fccac0c88e0 0x7fccac0c8d50 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:39.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.279+0000 7fccb2549700 1 -- 172.21.15.98:0/2995369476 >> 172.21.15.98:0/2995369476 conn(0x7fccac089e50 msgr2=0x7fccac0c63d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:39.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.279+0000 7fccb2549700 1 -- 172.21.15.98:0/2995369476 shutdown_connections 2023-10-06T05:07:39.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.279+0000 7fccb2549700 1 -- 172.21.15.98:0/2995369476 wait complete. 2023-10-06T05:07:39.292 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-06T05:07:39.292 DEBUG:teuthology.orchestra.run.smithi098:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-06T05:07:39.332 DEBUG:teuthology.orchestra.run.smithi174:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-06T05:07:39.385 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-06T05:07:39.385 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-06T05:07:39.385 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T05:07:39.385 DEBUG:teuthology.orchestra.run.smithi098:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2023-10-06T05:07:39.424 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi098.front.sepia.ceph.com 2023-10-06T05:07:39.424 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:07:39.425 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-06T05:07:39.480 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2023-10-06T05:07:39.480 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T05:07:39.480 DEBUG:teuthology.orchestra.run.smithi174:> sudo umount /home/ubuntu/cephtest/mnt.1 -f 2023-10-06T05:07:39.522 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi174.front.sepia.ceph.com 2023-10-06T05:07:39.522 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:07:39.522 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-10-06T05:07:39.580 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.2... 2023-10-06T05:07:39.580 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T05:07:39.580 DEBUG:teuthology.orchestra.run.smithi098:> sudo umount /home/ubuntu/cephtest/mnt.2 -f 2023-10-06T05:07:39.618 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi098.front.sepia.ceph.com 2023-10-06T05:07:39.618 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:07:39.618 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-10-06T05:07:39.673 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.3... 2023-10-06T05:07:39.673 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T05:07:39.673 DEBUG:teuthology.orchestra.run.smithi174:> sudo umount /home/ubuntu/cephtest/mnt.3 -f 2023-10-06T05:07:39.709 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi174.front.sepia.ceph.com 2023-10-06T05:07:39.709 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:07:39.709 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-10-06T05:07:39.767 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-06T05:07:39.767 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-06T05:07:39.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.880+0000 7f3621d95700 1 Processor -- start 2023-10-06T05:07:39.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.880+0000 7f3621d95700 1 -- start start 2023-10-06T05:07:39.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.881+0000 7f3621d95700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f361c159560 0x7f361c0aa530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:39.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.881+0000 7f3621d95700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f361c0b0760 0x7f361c0b1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:39.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.881+0000 7f3621d95700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f361c0b0c60 con 0x7f361c0b49a0 2023-10-06T05:07:39.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.881+0000 7f3621d95700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f361c0b0f40 con 0x7f361c0b0760 2023-10-06T05:07:39.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.881+0000 7f3621d95700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f361c0b1220 con 0x7f361c159560 2023-10-06T05:07:39.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.881+0000 7f361bfff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f361c0b0760 0x7f361c0b1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:39.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.881+0000 7f361bfff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f361c0b0760 0x7f361c0b1bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60236/0 (socket says 172.21.15.98:60236) 2023-10-06T05:07:39.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.881+0000 7f361bfff700 1 -- 172.21.15.98:0/2371866299 learned_addr learned my addr 172.21.15.98:0/2371866299 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:39.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.881+0000 7f361b7fe700 1 --2- 172.21.15.98:0/2371866299 >> v2:172.21.15.174:3300/0 conn(0x7f361c159560 0x7f361c0aa530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:39.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.882+0000 7f361bfff700 1 -- 172.21.15.98:0/2371866299 >> v2:172.21.15.174:3300/0 conn(0x7f361c159560 msgr2=0x7f361c0aa530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:39.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.882+0000 7f361bfff700 1 --2- 172.21.15.98:0/2371866299 >> v2:172.21.15.174:3300/0 conn(0x7f361c159560 0x7f361c0aa530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:39.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.882+0000 7f361bfff700 1 -- 172.21.15.98:0/2371866299 >> v1:172.21.15.174:6789/0 conn(0x7f361c0b49a0 legacy=0x7f361c0aaa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:39.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.882+0000 7f361bfff700 1 -- 172.21.15.98:0/2371866299 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f361c0b1500 con 0x7f361c0b0760 2023-10-06T05:07:39.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.882+0000 7f361bfff700 1 --2- 172.21.15.98:0/2371866299 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f361c0b0760 0x7f361c0b1bc0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f36140099b0 tx=0x7f3614009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d228d13522de0061 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:39.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.882+0000 7f361a7fc700 1 -- 172.21.15.98:0/2371866299 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3614009ea0 con 0x7f361c0b0760 2023-10-06T05:07:39.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.882+0000 7f361a7fc700 1 -- 172.21.15.98:0/2371866299 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f361400b820 con 0x7f361c0b0760 2023-10-06T05:07:39.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.882+0000 7f361a7fc700 1 -- 172.21.15.98:0/2371866299 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f361400bbc0 con 0x7f361c0b0760 2023-10-06T05:07:39.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.882+0000 7f3621d95700 1 -- 172.21.15.98:0/2371866299 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f361c0b0760 msgr2=0x7f361c0b1bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:39.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.882+0000 7f3621d95700 1 --2- 172.21.15.98:0/2371866299 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f361c0b0760 0x7f361c0b1bc0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f36140099b0 tx=0x7f3614009250 comp rx=0 tx=0).stop 2023-10-06T05:07:39.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.882+0000 7f3621d95700 1 -- 172.21.15.98:0/2371866299 shutdown_connections 2023-10-06T05:07:39.886 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.882+0000 7f3621d95700 1 --2- 172.21.15.98:0/2371866299 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f361c0b0760 0x7f361c0b1bc0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:39.886 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.882+0000 7f3621d95700 1 --2- 172.21.15.98:0/2371866299 >> v2:172.21.15.174:3300/0 conn(0x7f361c159560 0x7f361c0aa530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:39.886 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.882+0000 7f3621d95700 1 -- 172.21.15.98:0/2371866299 >> 172.21.15.98:0/2371866299 conn(0x7f361c0997f0 msgr2=0x7f361c06f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:39.886 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.882+0000 7f3621d95700 1 -- 172.21.15.98:0/2371866299 shutdown_connections 2023-10-06T05:07:39.886 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.882+0000 7f3621d95700 1 -- 172.21.15.98:0/2371866299 wait complete. 2023-10-06T05:07:39.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.884+0000 7f3621d95700 1 Processor -- start 2023-10-06T05:07:39.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.884+0000 7f3621d95700 1 -- start start 2023-10-06T05:07:39.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.884+0000 7f3621d95700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f361c0b0760 0x7f361c157540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:39.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.884+0000 7f3621d95700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f361c0b49a0 0x7f361c09aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:39.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.884+0000 7f3621d95700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f361c159560 0x7f361c0acc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:39.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.884+0000 7f3621d95700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f361c0af200 con 0x7f361c159560 2023-10-06T05:07:39.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.884+0000 7f3621d95700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f361c0ab200 con 0x7f361c0b49a0 2023-10-06T05:07:39.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.884+0000 7f3621d95700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f361c0ae1f0 con 0x7f361c0b0760 2023-10-06T05:07:39.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.884+0000 7f361bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f361c159560 0x7f361c0acc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:39.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.885+0000 7f361bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f361c159560 0x7f361c0acc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60242/0 (socket says 172.21.15.98:60242) 2023-10-06T05:07:39.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.885+0000 7f361bfff700 1 -- 172.21.15.98:0/1605839904 learned_addr learned my addr 172.21.15.98:0/1605839904 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:39.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.885+0000 7f361b7fe700 1 --2- 172.21.15.98:0/1605839904 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f361c0b0760 0x7f361c157540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:39.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.885+0000 7f361affd700 1 --2- 172.21.15.98:0/1605839904 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f361c0b49a0 0x7f361c09aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:39.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.885+0000 7f361bfff700 1 -- 172.21.15.98:0/1605839904 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f361c0b0760 msgr2=0x7f361c157540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:39.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.885+0000 7f361bfff700 1 --2- 172.21.15.98:0/1605839904 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f361c0b0760 0x7f361c157540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:39.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.885+0000 7f361bfff700 1 -- 172.21.15.98:0/1605839904 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f361c0b49a0 msgr2=0x7f361c09aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:39.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.885+0000 7f361bfff700 1 --2- 172.21.15.98:0/1605839904 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f361c0b49a0 0x7f361c09aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:39.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.885+0000 7f361bfff700 1 -- 172.21.15.98:0/1605839904 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36140094f0 con 0x7f361c159560 2023-10-06T05:07:39.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.885+0000 7f361bfff700 1 --2- 172.21.15.98:0/1605839904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f361c159560 0x7f361c0acc70 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f361c067c20 tx=0x7f3614017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:39.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.885+0000 7f36037fe700 1 -- 172.21.15.98:0/1605839904 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f361400bba0 con 0x7f361c159560 2023-10-06T05:07:39.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.885+0000 7f36037fe700 1 -- 172.21.15.98:0/1605839904 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3614011420 con 0x7f361c159560 2023-10-06T05:07:39.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.885+0000 7f36037fe700 1 -- 172.21.15.98:0/1605839904 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3614011800 con 0x7f361c159560 2023-10-06T05:07:39.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.885+0000 7f3621d95700 1 -- 172.21.15.98:0/1605839904 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f361c0aef20 con 0x7f361c159560 2023-10-06T05:07:39.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.885+0000 7f3621d95700 1 -- 172.21.15.98:0/1605839904 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f361c09a4d0 con 0x7f361c159560 2023-10-06T05:07:39.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.886+0000 7f3621d95700 1 -- 172.21.15.98:0/1605839904 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f361c09a050 con 0x7f361c159560 2023-10-06T05:07:39.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.886+0000 7f36037fe700 1 -- 172.21.15.98:0/1605839904 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f3614011b70 con 0x7f361c159560 2023-10-06T05:07:39.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.887+0000 7f36037fe700 1 --2- 172.21.15.98:0/1605839904 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f3608066f30 0x7f36080693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:39.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.887+0000 7f361b7fe700 1 --2- 172.21.15.98:0/1605839904 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f3608066f30 0x7f36080693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:39.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.887+0000 7f36037fe700 1 -- 172.21.15.98:0/1605839904 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6867+0+0 (secure 0 0 0) 0x7f3614086410 con 0x7f361c159560 2023-10-06T05:07:39.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.887+0000 7f361b7fe700 1 --2- 172.21.15.98:0/1605839904 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f3608066f30 0x7f36080693e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f360c00f510 tx=0x7f360c00b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:39.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:39.891+0000 7f36037fe700 1 -- 172.21.15.98:0/1605839904 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f361c09a050 con 0x7f361c159560 2023-10-06T05:07:40.124 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.123+0000 7f3621d95700 1 -- 172.21.15.98:0/1605839904 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f361c0ab370 con 0x7f361c159560 2023-10-06T05:07:40.125 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.124+0000 7f36037fe700 1 -- 172.21.15.98:0/1605839904 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4596 (secure 0 0 0) 0x7f361c0ab370 con 0x7f361c159560 2023-10-06T05:07:40.125 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:07:40.125 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4285,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6834/925250632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":925250632},{"type":"v1","addr":"172.21.15.174:6834","nonce":925250632}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4288,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/1629449810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6833","nonce":1629449810},{"type":"v1","addr":"172.21.15.174:6835","nonce":1629449810}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4299,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/3961327990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3961327990},{"type":"v1","addr":"172.21.15.98:6836","nonce":3961327990}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-06T05:06:57.086683+0000","modified":"2023-10-06T05:07:09.773654+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.98:6837/2683239094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":2683239094},{"type":"v1","addr":"172.21.15.98:6837","nonce":2683239094}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-06T05:07:40.127 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.126+0000 7f3621d95700 1 -- 172.21.15.98:0/1605839904 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f3608066f30 msgr2=0x7f36080693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:40.127 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.126+0000 7f3621d95700 1 --2- 172.21.15.98:0/1605839904 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f3608066f30 0x7f36080693e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f360c00f510 tx=0x7f360c00b040 comp rx=0 tx=0).stop 2023-10-06T05:07:40.127 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.126+0000 7f3621d95700 1 -- 172.21.15.98:0/1605839904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f361c159560 msgr2=0x7f361c0acc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:40.127 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.126+0000 7f3621d95700 1 --2- 172.21.15.98:0/1605839904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f361c159560 0x7f361c0acc70 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f361c067c20 tx=0x7f3614017000 comp rx=0 tx=0).stop 2023-10-06T05:07:40.127 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.126+0000 7f3621d95700 1 -- 172.21.15.98:0/1605839904 shutdown_connections 2023-10-06T05:07:40.128 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.126+0000 7f3621d95700 1 --2- 172.21.15.98:0/1605839904 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f3608066f30 0x7f36080693e0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:40.128 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.126+0000 7f3621d95700 1 --2- 172.21.15.98:0/1605839904 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f361c159560 0x7f361c0acc70 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:40.128 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.126+0000 7f3621d95700 1 --2- 172.21.15.98:0/1605839904 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f361c0b0760 0x7f361c157540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:40.128 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.126+0000 7f3621d95700 1 --2- 172.21.15.98:0/1605839904 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f361c0b49a0 0x7f361c09aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:40.128 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.126+0000 7f3621d95700 1 -- 172.21.15.98:0/1605839904 >> 172.21.15.98:0/1605839904 conn(0x7f361c0997f0 msgr2=0x7f361c158960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:40.128 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.126+0000 7f3621d95700 1 -- 172.21.15.98:0/1605839904 shutdown_connections 2023-10-06T05:07:40.128 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.126+0000 7f3621d95700 1 -- 172.21.15.98:0/1605839904 wait complete. 2023-10-06T05:07:40.128 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 8 2023-10-06T05:07:40.140 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T05:07:40.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.252+0000 7fd671589700 1 Processor -- start 2023-10-06T05:07:40.254 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.252+0000 7fd671589700 1 -- start start 2023-10-06T05:07:40.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.253+0000 7fd671589700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd66c0b2810 0x7fd66c0cc080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:40.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.253+0000 7fd671589700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd66c0cb7a0 con 0x7fd66c159440 2023-10-06T05:07:40.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.253+0000 7fd671589700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd66c0b0860 con 0x7fd66c0cbc30 2023-10-06T05:07:40.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.253+0000 7fd671589700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd66c0cb150 con 0x7fd66c0b2810 2023-10-06T05:07:40.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.253+0000 7fd66bfff700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fd66c159440 0x7fd66c0b3090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:40402/0 (socket says 172.21.15.98:40402) 2023-10-06T05:07:40.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.253+0000 7fd66bfff700 1 -- 172.21.15.98:0/2231048414 learned_addr learned my addr 172.21.15.98:0/2231048414 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:07:40.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.253+0000 7fd66affd700 1 -- 172.21.15.98:0/2231048414 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2132869394 0 0) 0x7fd66c0cb7a0 con 0x7fd66c159440 2023-10-06T05:07:40.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.253+0000 7fd66affd700 1 -- 172.21.15.98:0/2231048414 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd654003570 con 0x7fd66c159440 2023-10-06T05:07:40.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.253+0000 7fd670d88700 1 --2- 172.21.15.98:0/2231048414 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd66c0b2810 0x7fd66c0cc080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:40.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.254+0000 7fd66affd700 1 -- 172.21.15.98:0/2231048414 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2587396211 0 0) 0x7fd654003570 con 0x7fd66c159440 2023-10-06T05:07:40.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.254+0000 7fd66affd700 1 -- 172.21.15.98:0/2231048414 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd66c0b2810 msgr2=0x7fd66c0cc080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:40.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.254+0000 7fd66affd700 1 --2- 172.21.15.98:0/2231048414 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd66c0b2810 0x7fd66c0cc080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:40.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.254+0000 7fd66affd700 1 -- 172.21.15.98:0/2231048414 >> v1:172.21.15.174:6789/0 conn(0x7fd66c0cbc30 legacy=0x7fd66c0b2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:40.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.254+0000 7fd66affd700 1 -- 172.21.15.98:0/2231048414 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd66c0cb430 con 0x7fd66c159440 2023-10-06T05:07:40.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.254+0000 7fd66affd700 1 -- 172.21.15.98:0/2231048414 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1543991876 0 0) 0x7fd65c001470 con 0x7fd66c159440 2023-10-06T05:07:40.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.254+0000 7fd66affd700 1 -- 172.21.15.98:0/2231048414 >> v1:172.21.15.98:6789/0 conn(0x7fd66c159440 legacy=0x7fd66c0b3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:40.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.254+0000 7fd66affd700 1 --2- 172.21.15.98:0/2231048414 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd66c0b2810 0x7fd654003a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:40.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.254+0000 7fd66affd700 1 --2- 172.21.15.98:0/2231048414 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd6540040c0 0x7fd654004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:40.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.254+0000 7fd66affd700 1 --2- 172.21.15.98:0/2231048414 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd654004da0 0x7fd6540051f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:40.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.254+0000 7fd66affd700 1 -- 172.21.15.98:0/2231048414 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd66c0cb150 con 0x7fd654004da0 2023-10-06T05:07:40.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.254+0000 7fd66affd700 1 -- 172.21.15.98:0/2231048414 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd65400bc70 con 0x7fd6540040c0 2023-10-06T05:07:40.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.254+0000 7fd66affd700 1 -- 172.21.15.98:0/2231048414 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd65400be30 con 0x7fd66c0b2810 2023-10-06T05:07:40.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.254+0000 7fd670d88700 1 --2- 172.21.15.98:0/2231048414 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd654004da0 0x7fd6540051f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:40.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.254+0000 7fd66bfff700 1 --2- 172.21.15.98:0/2231048414 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd66c0b2810 0x7fd654003a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:40.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.254+0000 7fd66b7fe700 1 --2- 172.21.15.98:0/2231048414 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd6540040c0 0x7fd654004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:40.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.254+0000 7fd670d88700 1 -- 172.21.15.98:0/2231048414 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd66c0b2810 msgr2=0x7fd654003a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:40.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.254+0000 7fd670d88700 1 --2- 172.21.15.98:0/2231048414 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd66c0b2810 0x7fd654003a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:40.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.254+0000 7fd670d88700 1 -- 172.21.15.98:0/2231048414 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd6540040c0 msgr2=0x7fd654004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:40.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.254+0000 7fd670d88700 1 --2- 172.21.15.98:0/2231048414 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd6540040c0 0x7fd654004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:40.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.254+0000 7fd670d88700 1 -- 172.21.15.98:0/2231048414 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd654003570 con 0x7fd654004da0 2023-10-06T05:07:40.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.254+0000 7fd670d88700 1 --2- 172.21.15.98:0/2231048414 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd654004da0 0x7fd6540051f0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fd664007440 tx=0x7fd664009710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=450f90fb6db1e8a3 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:40.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.254+0000 7fd66affd700 1 -- 172.21.15.98:0/2231048414 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd664009a40 con 0x7fd654004da0 2023-10-06T05:07:40.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.254+0000 7fd66affd700 1 -- 172.21.15.98:0/2231048414 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd664007cf0 con 0x7fd654004da0 2023-10-06T05:07:40.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.254+0000 7fd66affd700 1 -- 172.21.15.98:0/2231048414 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd664014630 con 0x7fd654004da0 2023-10-06T05:07:40.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.255+0000 7fd671589700 1 -- 172.21.15.98:0/2231048414 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd654004da0 msgr2=0x7fd6540051f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:40.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.255+0000 7fd671589700 1 --2- 172.21.15.98:0/2231048414 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd654004da0 0x7fd6540051f0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fd664007440 tx=0x7fd664009710 comp rx=0 tx=0).stop 2023-10-06T05:07:40.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.255+0000 7fd670d88700 1 -- 172.21.15.98:0/2231048414 reap_dead start 2023-10-06T05:07:40.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.255+0000 7fd671589700 1 -- 172.21.15.98:0/2231048414 shutdown_connections 2023-10-06T05:07:40.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.255+0000 7fd671589700 1 -- 172.21.15.98:0/2231048414 >> 172.21.15.98:0/2231048414 conn(0x7fd66c06ece0 msgr2=0x7fd66c06f0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:40.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.255+0000 7fd671589700 1 -- 172.21.15.98:0/2231048414 shutdown_connections 2023-10-06T05:07:40.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.255+0000 7fd671589700 1 -- 172.21.15.98:0/2231048414 wait complete. 2023-10-06T05:07:40.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.256+0000 7fd671589700 1 Processor -- start 2023-10-06T05:07:40.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.256+0000 7fd671589700 1 -- start start 2023-10-06T05:07:40.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.257+0000 7fd671589700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd66c153950 0x7fd66c0c4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:40.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.257+0000 7fd671589700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd66c153d20 0x7fd66c0c5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:40.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.257+0000 7fd671589700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd66c0c5980 0x7fd66c152530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:40.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.257+0000 7fd671589700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd66c0c5fb0 con 0x7fd66c153950 2023-10-06T05:07:40.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.257+0000 7fd671589700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd66c0c5dd0 con 0x7fd66c0c5980 2023-10-06T05:07:40.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.257+0000 7fd671589700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd66c0d0cd0 con 0x7fd66c153d20 2023-10-06T05:07:40.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.257+0000 7fd66bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd66c153950 0x7fd66c0c4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:40.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.257+0000 7fd66bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd66c153950 0x7fd66c0c4b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60266/0 (socket says 172.21.15.98:60266) 2023-10-06T05:07:40.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.257+0000 7fd66bfff700 1 -- 172.21.15.98:0/294494438 learned_addr learned my addr 172.21.15.98:0/294494438 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:40.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.257+0000 7fd66b7fe700 1 --2- 172.21.15.98:0/294494438 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd66c153d20 0x7fd66c0c5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:40.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.257+0000 7fd66bfff700 1 -- 172.21.15.98:0/294494438 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd66c153d20 msgr2=0x7fd66c0c5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:40.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.257+0000 7fd66bfff700 1 --2- 172.21.15.98:0/294494438 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd66c153d20 0x7fd66c0c5440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:40.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.257+0000 7fd66bfff700 1 -- 172.21.15.98:0/294494438 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd66c0c5980 msgr2=0x7fd66c152530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:40.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.257+0000 7fd66bfff700 1 --2- 172.21.15.98:0/294494438 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd66c0c5980 0x7fd66c152530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:40.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.257+0000 7fd66bfff700 1 -- 172.21.15.98:0/294494438 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd66c0c3da0 con 0x7fd66c153950 2023-10-06T05:07:40.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.257+0000 7fd66bfff700 1 --2- 172.21.15.98:0/294494438 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd66c153950 0x7fd66c0c4b60 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fd65c0011e0 tx=0x7fd65c001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:40.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.258+0000 7fd653fff700 1 -- 172.21.15.98:0/294494438 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd65c018030 con 0x7fd66c153950 2023-10-06T05:07:40.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.258+0000 7fd653fff700 1 -- 172.21.15.98:0/294494438 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd65c00f930 con 0x7fd66c153950 2023-10-06T05:07:40.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.258+0000 7fd671589700 1 -- 172.21.15.98:0/294494438 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd66c0c4080 con 0x7fd66c153950 2023-10-06T05:07:40.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.258+0000 7fd653fff700 1 -- 172.21.15.98:0/294494438 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd65c00fcd0 con 0x7fd66c153950 2023-10-06T05:07:40.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.258+0000 7fd671589700 1 -- 172.21.15.98:0/294494438 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd66c0c4640 con 0x7fd66c153950 2023-10-06T05:07:40.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.258+0000 7fd671589700 1 -- 172.21.15.98:0/294494438 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd66c0c4860 con 0x7fd66c153950 2023-10-06T05:07:40.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.259+0000 7fd653fff700 1 -- 172.21.15.98:0/294494438 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fd65c01b070 con 0x7fd66c153950 2023-10-06T05:07:40.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.259+0000 7fd653fff700 1 --2- 172.21.15.98:0/294494438 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fd658066f60 0x7fd658069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:40.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.259+0000 7fd653fff700 1 -- 172.21.15.98:0/294494438 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6867+0+0 (secure 0 0 0) 0x7fd65c019070 con 0x7fd66c153950 2023-10-06T05:07:40.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.259+0000 7fd66b7fe700 1 --2- 172.21.15.98:0/294494438 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fd658066f60 0x7fd658069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:40.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.260+0000 7fd66b7fe700 1 --2- 172.21.15.98:0/294494438 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fd658066f60 0x7fd658069410 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd660007e40 tx=0x7fd660009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:40.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.263+0000 7fd653fff700 1 -- 172.21.15.98:0/294494438 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd65c055310 con 0x7fd66c153950 2023-10-06T05:07:40.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.450+0000 7fd671589700 1 -- 172.21.15.98:0/294494438 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd66c05f230 con 0x7fd66c153950 2023-10-06T05:07:40.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.450+0000 7fd653fff700 1 -- 172.21.15.98:0/294494438 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v31) v1 ==== 74+0+18801 (secure 0 0 0) 0x7fd65c054ea0 con 0x7fd66c153950 2023-10-06T05:07:40.452 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:07:40.452 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":31,"fsid":"f50c5df5-cad5-47a4-9bb1-8c791d4311ec","created":"2023-10-06T05:06:32.971175+0000","modified":"2023-10-06T05:07:39.938013+0000","last_up_change":"2023-10-06T05:06:44.016489+0000","last_in_change":"2023-10-06T05:06:35.544511+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-06T05:06:44.559671+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-06T05:06:51.432161+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-10-06T05:06:55.589057+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-10-06T05:06:56.041687+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-10-06T05:06:59.754099+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d9495d95-da75-4c3b-8927-e9bae6a8b093","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":90627},{"type":"v1","addr":"172.21.15.98:6801","nonce":90627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":90627},{"type":"v1","addr":"172.21.15.98:6803","nonce":90627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":90627},{"type":"v1","addr":"172.21.15.98:6807","nonce":90627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":90627},{"type":"v1","addr":"172.21.15.98:6805","nonce":90627}]},"public_addr":"172.21.15.98:6801/90627","cluster_addr":"172.21.15.98:6803/90627","heartbeat_back_addr":"172.21.15.98:6807/90627","heartbeat_front_addr":"172.21.15.98:6805/90627","state":["exists","up"]},{"osd":1,"uuid":"f25ba5be-d65e-4707-a134-154092deb754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":90628},{"type":"v1","addr":"172.21.15.98:6809","nonce":90628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":90628},{"type":"v1","addr":"172.21.15.98:6811","nonce":90628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":90628},{"type":"v1","addr":"172.21.15.98:6815","nonce":90628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":90628},{"type":"v1","addr":"172.21.15.98:6813","nonce":90628}]},"public_addr":"172.21.15.98:6809/90628","cluster_addr":"172.21.15.98:6811/90628","heartbeat_back_addr":"172.21.15.98:6815/90628","heartbeat_front_addr":"172.21.15.98:6813/90628","state":["exists","up"]},{"osd":2,"uuid":"5ab9e028-945c-43e2-8c95-3b0b4cf38e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":90662},{"type":"v1","addr":"172.21.15.98:6818","nonce":90662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":90662},{"type":"v1","addr":"172.21.15.98:6822","nonce":90662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":90662},{"type":"v1","addr":"172.21.15.98:6830","nonce":90662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":90662},{"type":"v1","addr":"172.21.15.98:6826","nonce":90662}]},"public_addr":"172.21.15.98:6818/90662","cluster_addr":"172.21.15.98:6822/90662","heartbeat_back_addr":"172.21.15.98:6830/90662","heartbeat_front_addr":"172.21.15.98:6826/90662","state":["exists","up"]},{"osd":3,"uuid":"3bb803ff-1738-46bf-be62-2267bf378822","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6817","nonce":90663},{"type":"v1","addr":"172.21.15.98:6819","nonce":90663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6821","nonce":90663},{"type":"v1","addr":"172.21.15.98:6823","nonce":90663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6829","nonce":90663},{"type":"v1","addr":"172.21.15.98:6831","nonce":90663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6825","nonce":90663},{"type":"v1","addr":"172.21.15.98:6827","nonce":90663}]},"public_addr":"172.21.15.98:6819/90663","cluster_addr":"172.21.15.98:6823/90663","heartbeat_back_addr":"172.21.15.98:6831/90663","heartbeat_front_addr":"172.21.15.98:6827/90663","state":["exists","up"]},{"osd":4,"uuid":"7d57a4c9-6462-43f6-b64a-7f9e9b4353d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":90847},{"type":"v1","addr":"172.21.15.174:6801","nonce":90847}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":90847},{"type":"v1","addr":"172.21.15.174:6803","nonce":90847}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":90847},{"type":"v1","addr":"172.21.15.174:6807","nonce":90847}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":90847},{"type":"v1","addr":"172.21.15.174:6805","nonce":90847}]},"public_addr":"172.21.15.174:6801/90847","cluster_addr":"172.21.15.174:6803/90847","heartbeat_back_addr":"172.21.15.174:6807/90847","heartbeat_front_addr":"172.21.15.174:6805/90847","state":["exists","up"]},{"osd":5,"uuid":"83287ea4-e44d-4af9-bc77-36d47c60d4a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":90848},{"type":"v1","addr":"172.21.15.174:6809","nonce":90848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":90848},{"type":"v1","addr":"172.21.15.174:6811","nonce":90848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":90848},{"type":"v1","addr":"172.21.15.174:6815","nonce":90848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":90848},{"type":"v1","addr":"172.21.15.174:6813","nonce":90848}]},"public_addr":"172.21.15.174:6809/90848","cluster_addr":"172.21.15.174:6811/90848","heartbeat_back_addr":"172.21.15.174:6815/90848","heartbeat_front_addr":"172.21.15.174:6813/90848","state":["exists","up"]},{"osd":6,"uuid":"496a6272-892e-43c4-a145-206623c22f99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":90877},{"type":"v1","addr":"172.21.15.174:6817","nonce":90877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":90877},{"type":"v1","addr":"172.21.15.174:6819","nonce":90877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":90877},{"type":"v1","addr":"172.21.15.174:6823","nonce":90877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":90877},{"type":"v1","addr":"172.21.15.174:6821","nonce":90877}]},"public_addr":"172.21.15.174:6817/90877","cluster_addr":"172.21.15.174:6819/90877","heartbeat_back_addr":"172.21.15.174:6823/90877","heartbeat_front_addr":"172.21.15.174:6821/90877","state":["exists","up"]},{"osd":7,"uuid":"49967e34-8b07-4bab-a13e-0c16d6a0f3a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":90878},{"type":"v1","addr":"172.21.15.174:6825","nonce":90878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":90878},{"type":"v1","addr":"172.21.15.174:6827","nonce":90878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":90878},{"type":"v1","addr":"172.21.15.174:6831","nonce":90878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":90878},{"type":"v1","addr":"172.21.15.174:6829","nonce":90878}]},"public_addr":"172.21.15.174:6825/90878","cluster_addr":"172.21.15.174:6827/90878","heartbeat_back_addr":"172.21.15.174:6831/90878","heartbeat_front_addr":"172.21.15.174:6829/90878","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.446479+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.514266+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.794108+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.810894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.731919+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.717756+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.737435+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.508682+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.9","mappings":[{"from":1,"to":5}]},{"pgid":"4.c","mappings":[{"from":3,"to":2}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T05:07:40.454 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.452+0000 7fd671589700 1 -- 172.21.15.98:0/294494438 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fd658066f60 msgr2=0x7fd658069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:40.454 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.452+0000 7fd671589700 1 --2- 172.21.15.98:0/294494438 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fd658066f60 0x7fd658069410 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd660007e40 tx=0x7fd660009040 comp rx=0 tx=0).stop 2023-10-06T05:07:40.454 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.453+0000 7fd671589700 1 -- 172.21.15.98:0/294494438 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd66c153950 msgr2=0x7fd66c0c4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:40.454 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.453+0000 7fd671589700 1 --2- 172.21.15.98:0/294494438 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd66c153950 0x7fd66c0c4b60 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fd65c0011e0 tx=0x7fd65c001e50 comp rx=0 tx=0).stop 2023-10-06T05:07:40.454 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.453+0000 7fd671589700 1 -- 172.21.15.98:0/294494438 shutdown_connections 2023-10-06T05:07:40.455 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.453+0000 7fd671589700 1 --2- 172.21.15.98:0/294494438 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd66c0c5980 0x7fd66c152530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:40.455 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.453+0000 7fd671589700 1 --2- 172.21.15.98:0/294494438 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fd658066f60 0x7fd658069410 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:40.455 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.453+0000 7fd671589700 1 --2- 172.21.15.98:0/294494438 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd66c153950 0x7fd66c0c4b60 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:40.455 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.453+0000 7fd671589700 1 --2- 172.21.15.98:0/294494438 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd66c153d20 0x7fd66c0c5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:40.455 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.453+0000 7fd671589700 1 -- 172.21.15.98:0/294494438 >> 172.21.15.98:0/294494438 conn(0x7fd66c06ece0 msgr2=0x7fd66c0b4a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:40.455 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.453+0000 7fd671589700 1 -- 172.21.15.98:0/294494438 shutdown_connections 2023-10-06T05:07:40.455 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.453+0000 7fd671589700 1 -- 172.21.15.98:0/294494438 wait complete. 2023-10-06T05:07:40.466 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-06T05:06:44.559671+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-10-06T05:06:51.432161+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-06T05:06:55.589057+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-10-06T05:06:56.041687+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-06T05:06:59.754099+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '29', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-06T05:07:40.466 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-06T05:07:40.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.582+0000 7f915ac7f700 1 Processor -- start 2023-10-06T05:07:40.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.582+0000 7f915ac7f700 1 -- start start 2023-10-06T05:07:40.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.583+0000 7f915ac7f700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f91541594d0 0x7f91540cb560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:40.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.583+0000 7f915ac7f700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f91541598a0 0x7f91540cbaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:40.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.583+0000 7f915ac7f700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f91540ce7e0 0x7f91540cebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:40.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.583+0000 7f915ac7f700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91540cf5d0 con 0x7f91541598a0 2023-10-06T05:07:40.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.583+0000 7f915ac7f700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91540cf8b0 con 0x7f91541594d0 2023-10-06T05:07:40.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.583+0000 7f915ac7f700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91540cfb90 con 0x7f91540ce7e0 2023-10-06T05:07:40.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.583+0000 7f9153fff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f91541598a0 0x7f91540cbaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:40.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.583+0000 7f9153fff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f91541598a0 0x7f91540cbaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60274/0 (socket says 172.21.15.98:60274) 2023-10-06T05:07:40.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.583+0000 7f9153fff700 1 -- 172.21.15.98:0/3410645473 learned_addr learned my addr 172.21.15.98:0/3410645473 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:40.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.583+0000 7f9158a1b700 1 --2- 172.21.15.98:0/3410645473 >> v2:172.21.15.174:3300/0 conn(0x7f91541594d0 0x7f91540cb560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:40.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.583+0000 7f915921c700 1 --2- 172.21.15.98:0/3410645473 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f91540ce7e0 0x7f91540cebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:40.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.583+0000 7f9153fff700 1 -- 172.21.15.98:0/3410645473 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f91540ce7e0 msgr2=0x7f91540cebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:40.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.583+0000 7f9153fff700 1 --2- 172.21.15.98:0/3410645473 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f91540ce7e0 0x7f91540cebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:40.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.583+0000 7f9153fff700 1 -- 172.21.15.98:0/3410645473 >> v2:172.21.15.174:3300/0 conn(0x7f91541594d0 msgr2=0x7f91540cb560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:40.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.583+0000 7f9153fff700 1 --2- 172.21.15.98:0/3410645473 >> v2:172.21.15.174:3300/0 conn(0x7f91541594d0 0x7f91540cb560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:40.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.583+0000 7f9153fff700 1 -- 172.21.15.98:0/3410645473 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91540cfe70 con 0x7f91541598a0 2023-10-06T05:07:40.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.584+0000 7f9153fff700 1 --2- 172.21.15.98:0/3410645473 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91541598a0 0x7f91540cbaa0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f9154088f70 tx=0x7f9148009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2ebc9b96066076f server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:40.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.584+0000 7f91537fe700 1 -- 172.21.15.98:0/3410645473 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9148009e70 con 0x7f91541598a0 2023-10-06T05:07:40.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.584+0000 7f91537fe700 1 -- 172.21.15.98:0/3410645473 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f914800b860 con 0x7f91541598a0 2023-10-06T05:07:40.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.584+0000 7f91537fe700 1 -- 172.21.15.98:0/3410645473 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f914800bc00 con 0x7f91541598a0 2023-10-06T05:07:40.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.584+0000 7f915ac7f700 1 -- 172.21.15.98:0/3410645473 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91541598a0 msgr2=0x7f91540cbaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:40.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.584+0000 7f915ac7f700 1 --2- 172.21.15.98:0/3410645473 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91541598a0 0x7f91540cbaa0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f9154088f70 tx=0x7f9148009250 comp rx=0 tx=0).stop 2023-10-06T05:07:40.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.584+0000 7f915ac7f700 1 -- 172.21.15.98:0/3410645473 shutdown_connections 2023-10-06T05:07:40.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.584+0000 7f915ac7f700 1 --2- 172.21.15.98:0/3410645473 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f91540ce7e0 0x7f91540cebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:40.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.584+0000 7f915ac7f700 1 --2- 172.21.15.98:0/3410645473 >> v2:172.21.15.174:3300/0 conn(0x7f91541594d0 0x7f91540cb560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:40.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.584+0000 7f915ac7f700 1 --2- 172.21.15.98:0/3410645473 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91541598a0 0x7f91540cbaa0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:40.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.584+0000 7f915ac7f700 1 -- 172.21.15.98:0/3410645473 >> 172.21.15.98:0/3410645473 conn(0x7f9154089e50 msgr2=0x7f9154099770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:40.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.584+0000 7f915ac7f700 1 -- 172.21.15.98:0/3410645473 shutdown_connections 2023-10-06T05:07:40.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.584+0000 7f915ac7f700 1 -- 172.21.15.98:0/3410645473 wait complete. 2023-10-06T05:07:40.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.585+0000 7f915ac7f700 1 Processor -- start 2023-10-06T05:07:40.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.586+0000 7f915ac7f700 1 -- start start 2023-10-06T05:07:40.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.586+0000 7f915ac7f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91540ce7e0 0x7f915416a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:40.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.586+0000 7f915ac7f700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f91541594d0 0x7f91540c6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:40.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.586+0000 7f915ac7f700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f91541598a0 0x7f91540c66c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:40.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.586+0000 7f915ac7f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f915414b320 con 0x7f91540ce7e0 2023-10-06T05:07:40.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.586+0000 7f915ac7f700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f915416abf0 con 0x7f91541598a0 2023-10-06T05:07:40.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.586+0000 7f915ac7f700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9154157f30 con 0x7f91541594d0 2023-10-06T05:07:40.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.586+0000 7f9158a1b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91540ce7e0 0x7f915416a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:40.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.586+0000 7f9158a1b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91540ce7e0 0x7f915416a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60280/0 (socket says 172.21.15.98:60280) 2023-10-06T05:07:40.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.586+0000 7f9158a1b700 1 -- 172.21.15.98:0/3214300296 learned_addr learned my addr 172.21.15.98:0/3214300296 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:40.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.586+0000 7f9153fff700 1 --2- 172.21.15.98:0/3214300296 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f91541594d0 0x7f91540c6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:40.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.586+0000 7f915921c700 1 --2- 172.21.15.98:0/3214300296 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f91541598a0 0x7f91540c66c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:40.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.587+0000 7f9158a1b700 1 -- 172.21.15.98:0/3214300296 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f91541594d0 msgr2=0x7f91540c6180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:40.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.587+0000 7f9158a1b700 1 --2- 172.21.15.98:0/3214300296 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f91541594d0 0x7f91540c6180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:40.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.587+0000 7f9158a1b700 1 -- 172.21.15.98:0/3214300296 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f91541598a0 msgr2=0x7f91540c66c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:40.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.587+0000 7f9158a1b700 1 --2- 172.21.15.98:0/3214300296 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f91541598a0 0x7f91540c66c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:40.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.587+0000 7f9158a1b700 1 -- 172.21.15.98:0/3214300296 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91480094f0 con 0x7f91540ce7e0 2023-10-06T05:07:40.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.587+0000 7f9153fff700 1 --2- 172.21.15.98:0/3214300296 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f91541594d0 0x7f91540c6180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:07:40.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.587+0000 7f915921c700 1 --2- 172.21.15.98:0/3214300296 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f91541598a0 0x7f91540c66c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:07:40.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.587+0000 7f9158a1b700 1 --2- 172.21.15.98:0/3214300296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91540ce7e0 0x7f915416a6b0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f914400ec80 tx=0x7f91440091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:40.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.587+0000 7f9150ff9700 1 -- 172.21.15.98:0/3214300296 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f914400c9d0 con 0x7f91540ce7e0 2023-10-06T05:07:40.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.587+0000 7f9150ff9700 1 -- 172.21.15.98:0/3214300296 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f914400cb30 con 0x7f91540ce7e0 2023-10-06T05:07:40.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.587+0000 7f915ac7f700 1 -- 172.21.15.98:0/3214300296 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9154158210 con 0x7f91540ce7e0 2023-10-06T05:07:40.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.587+0000 7f9150ff9700 1 -- 172.21.15.98:0/3214300296 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9144013470 con 0x7f91540ce7e0 2023-10-06T05:07:40.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.587+0000 7f915ac7f700 1 -- 172.21.15.98:0/3214300296 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91541587d0 con 0x7f91540ce7e0 2023-10-06T05:07:40.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.588+0000 7f915ac7f700 1 -- 172.21.15.98:0/3214300296 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9154064a90 con 0x7f91540ce7e0 2023-10-06T05:07:40.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.588+0000 7f9150ff9700 1 -- 172.21.15.98:0/3214300296 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f9144013820 con 0x7f91540ce7e0 2023-10-06T05:07:40.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.589+0000 7f9150ff9700 1 --2- 172.21.15.98:0/3214300296 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f9140066f30 0x7f91400693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:40.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.589+0000 7f9150ff9700 1 -- 172.21.15.98:0/3214300296 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6867+0+0 (secure 0 0 0) 0x7f9144084d50 con 0x7f91540ce7e0 2023-10-06T05:07:40.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.589+0000 7f9153fff700 1 --2- 172.21.15.98:0/3214300296 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f9140066f30 0x7f91400693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:40.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.589+0000 7f9153fff700 1 --2- 172.21.15.98:0/3214300296 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f9140066f30 0x7f91400693e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f915408a830 tx=0x7f9148017000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:40.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.593+0000 7f9150ff9700 1 -- 172.21.15.98:0/3214300296 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9144088030 con 0x7f91540ce7e0 2023-10-06T05:07:40.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.776+0000 7f915ac7f700 1 -- 172.21.15.98:0/3214300296 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f915414e310 con 0x7f91540ce7e0 2023-10-06T05:07:40.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.777+0000 7f9150ff9700 1 -- 172.21.15.98:0/3214300296 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v31) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9144051b30 con 0x7f91540ce7e0 2023-10-06T05:07:40.778 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2023-10-06T05:07:40.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.779+0000 7f915ac7f700 1 -- 172.21.15.98:0/3214300296 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f9140066f30 msgr2=0x7f91400693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:40.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.779+0000 7f915ac7f700 1 --2- 172.21.15.98:0/3214300296 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f9140066f30 0x7f91400693e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f915408a830 tx=0x7f9148017000 comp rx=0 tx=0).stop 2023-10-06T05:07:40.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.779+0000 7f915ac7f700 1 -- 172.21.15.98:0/3214300296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91540ce7e0 msgr2=0x7f915416a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:40.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.779+0000 7f915ac7f700 1 --2- 172.21.15.98:0/3214300296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91540ce7e0 0x7f915416a6b0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f914400ec80 tx=0x7f91440091a0 comp rx=0 tx=0).stop 2023-10-06T05:07:40.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.779+0000 7f915ac7f700 1 -- 172.21.15.98:0/3214300296 shutdown_connections 2023-10-06T05:07:40.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.779+0000 7f915ac7f700 1 --2- 172.21.15.98:0/3214300296 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f91541598a0 0x7f91540c66c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:40.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.779+0000 7f915ac7f700 1 --2- 172.21.15.98:0/3214300296 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f9140066f30 0x7f91400693e0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:40.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.779+0000 7f915ac7f700 1 --2- 172.21.15.98:0/3214300296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91540ce7e0 0x7f915416a6b0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:40.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.779+0000 7f915ac7f700 1 --2- 172.21.15.98:0/3214300296 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f91541594d0 0x7f91540c6180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:40.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.779+0000 7f915ac7f700 1 -- 172.21.15.98:0/3214300296 >> 172.21.15.98:0/3214300296 conn(0x7f9154089e50 msgr2=0x7f91540cad00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:40.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.779+0000 7f915ac7f700 1 -- 172.21.15.98:0/3214300296 shutdown_connections 2023-10-06T05:07:40.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.779+0000 7f915ac7f700 1 -- 172.21.15.98:0/3214300296 wait complete. 2023-10-06T05:07:40.793 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-10-06T05:07:40.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.909+0000 7f4caec90700 1 Processor -- start 2023-10-06T05:07:40.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.909+0000 7f4caec90700 1 -- start start 2023-10-06T05:07:40.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.910+0000 7f4caec90700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4ca81598a0 0x7f4ca80ce7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:40.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.910+0000 7f4caec90700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4ca80cbc70 0x7f4ca80ced20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:40.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.910+0000 7f4caec90700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ca80cf5d0 con 0x7f4ca81594d0 2023-10-06T05:07:40.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.910+0000 7f4caec90700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ca80cf8b0 con 0x7f4ca80cbc70 2023-10-06T05:07:40.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.910+0000 7f4caec90700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca80cfb90 con 0x7f4ca81598a0 2023-10-06T05:07:40.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.910+0000 7f4cad22d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4ca80cbc70 0x7f4ca80ced20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:40.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.910+0000 7f4cad22d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4ca80cbc70 0x7f4ca80ced20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60290/0 (socket says 172.21.15.98:60290) 2023-10-06T05:07:40.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.910+0000 7f4cad22d700 1 -- 172.21.15.98:0/3012494555 learned_addr learned my addr 172.21.15.98:0/3012494555 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:40.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.911+0000 7f4c9ffff700 1 --2- 172.21.15.98:0/3012494555 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4ca81598a0 0x7f4ca80ce7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:40.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.911+0000 7f4cad22d700 1 -- 172.21.15.98:0/3012494555 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4ca81598a0 msgr2=0x7f4ca80ce7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:40.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.911+0000 7f4cad22d700 1 --2- 172.21.15.98:0/3012494555 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4ca81598a0 0x7f4ca80ce7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:40.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.911+0000 7f4cad22d700 1 -- 172.21.15.98:0/3012494555 >> v1:172.21.15.174:6789/0 conn(0x7f4ca81594d0 legacy=0x7f4ca80cb560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:40.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.911+0000 7f4cad22d700 1 -- 172.21.15.98:0/3012494555 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ca80cfe70 con 0x7f4ca80cbc70 2023-10-06T05:07:40.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.911+0000 7f4cad22d700 1 --2- 172.21.15.98:0/3012494555 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ca80cbc70 0x7f4ca80ced20 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f4ca00099b0 tx=0x7f4ca0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c41d91e5d25ca24 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:40.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.911+0000 7f4c9f7fe700 1 -- 172.21.15.98:0/3012494555 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ca000b820 con 0x7f4ca80cbc70 2023-10-06T05:07:40.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.911+0000 7f4c9f7fe700 1 -- 172.21.15.98:0/3012494555 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ca000b980 con 0x7f4ca80cbc70 2023-10-06T05:07:40.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.911+0000 7f4c9f7fe700 1 -- 172.21.15.98:0/3012494555 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ca000bd20 con 0x7f4ca80cbc70 2023-10-06T05:07:40.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.911+0000 7f4caec90700 1 -- 172.21.15.98:0/3012494555 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ca80cbc70 msgr2=0x7f4ca80ced20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:40.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.911+0000 7f4caec90700 1 --2- 172.21.15.98:0/3012494555 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ca80cbc70 0x7f4ca80ced20 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f4ca00099b0 tx=0x7f4ca0009250 comp rx=0 tx=0).stop 2023-10-06T05:07:40.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.911+0000 7f4caec90700 1 -- 172.21.15.98:0/3012494555 shutdown_connections 2023-10-06T05:07:40.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.911+0000 7f4caec90700 1 --2- 172.21.15.98:0/3012494555 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ca80cbc70 0x7f4ca80ced20 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:40.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.911+0000 7f4caec90700 1 --2- 172.21.15.98:0/3012494555 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4ca81598a0 0x7f4ca80ce7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:40.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.911+0000 7f4caec90700 1 -- 172.21.15.98:0/3012494555 >> 172.21.15.98:0/3012494555 conn(0x7f4ca8089e50 msgr2=0x7f4ca8099770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:40.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.911+0000 7f4caec90700 1 -- 172.21.15.98:0/3012494555 shutdown_connections 2023-10-06T05:07:40.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.912+0000 7f4caec90700 1 -- 172.21.15.98:0/3012494555 wait complete. 2023-10-06T05:07:40.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.913+0000 7f4caec90700 1 Processor -- start 2023-10-06T05:07:40.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.913+0000 7f4caec90700 1 -- start start 2023-10-06T05:07:40.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.913+0000 7f4caec90700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4ca80cbc70 0x7f4ca8155640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:40.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.913+0000 7f4caec90700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4ca81594d0 0x7f4ca815c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:40.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.913+0000 7f4caec90700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ca81598a0 0x7f4ca815a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:40.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.913+0000 7f4caec90700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca8155c50 con 0x7f4ca81598a0 2023-10-06T05:07:40.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.913+0000 7f4caec90700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca8155f30 con 0x7f4ca80cbc70 2023-10-06T05:07:40.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.913+0000 7f4caec90700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca8157aa0 con 0x7f4ca81594d0 2023-10-06T05:07:40.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.914+0000 7f4cad22d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ca81598a0 0x7f4ca815a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:40.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.914+0000 7f4cad22d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ca81598a0 0x7f4ca815a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60296/0 (socket says 172.21.15.98:60296) 2023-10-06T05:07:40.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.914+0000 7f4cad22d700 1 -- 172.21.15.98:0/3762118828 learned_addr learned my addr 172.21.15.98:0/3762118828 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:40.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.914+0000 7f4c9ffff700 1 --2- 172.21.15.98:0/3762118828 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4ca81594d0 0x7f4ca815c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:40.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.914+0000 7f4caca2c700 1 --2- 172.21.15.98:0/3762118828 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4ca80cbc70 0x7f4ca8155640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:40.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.914+0000 7f4cad22d700 1 -- 172.21.15.98:0/3762118828 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4ca81594d0 msgr2=0x7f4ca815c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:40.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.914+0000 7f4cad22d700 1 --2- 172.21.15.98:0/3762118828 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4ca81594d0 0x7f4ca815c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:40.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.914+0000 7f4cad22d700 1 -- 172.21.15.98:0/3762118828 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4ca80cbc70 msgr2=0x7f4ca8155640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:40.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.914+0000 7f4cad22d700 1 --2- 172.21.15.98:0/3762118828 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4ca80cbc70 0x7f4ca8155640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:40.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.914+0000 7f4cad22d700 1 -- 172.21.15.98:0/3762118828 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ca00094f0 con 0x7f4ca81598a0 2023-10-06T05:07:40.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.914+0000 7f4cad22d700 1 --2- 172.21.15.98:0/3762118828 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ca81598a0 0x7f4ca815a720 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f4ca00094c0 tx=0x7f4ca0016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:40.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.914+0000 7f4c9cff9700 1 -- 172.21.15.98:0/3762118828 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ca001b450 con 0x7f4ca81598a0 2023-10-06T05:07:40.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.914+0000 7f4c9cff9700 1 -- 172.21.15.98:0/3762118828 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ca0020070 con 0x7f4ca81598a0 2023-10-06T05:07:40.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.914+0000 7f4caec90700 1 -- 172.21.15.98:0/3762118828 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ca8157d80 con 0x7f4ca81598a0 2023-10-06T05:07:40.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.914+0000 7f4c9cff9700 1 -- 172.21.15.98:0/3762118828 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ca001b800 con 0x7f4ca81598a0 2023-10-06T05:07:40.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.914+0000 7f4caec90700 1 -- 172.21.15.98:0/3762118828 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ca8158580 con 0x7f4ca81598a0 2023-10-06T05:07:40.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.915+0000 7f4caec90700 1 -- 172.21.15.98:0/3762118828 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ca8155d90 con 0x7f4ca81598a0 2023-10-06T05:07:40.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.916+0000 7f4c9cff9700 1 -- 172.21.15.98:0/3762118828 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f4ca0022030 con 0x7f4ca81598a0 2023-10-06T05:07:40.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.916+0000 7f4c9cff9700 1 --2- 172.21.15.98:0/3762118828 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f4c94066f30 0x7f4c940693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:40.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.916+0000 7f4c9cff9700 1 -- 172.21.15.98:0/3762118828 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6867+0+0 (secure 0 0 0) 0x7f4ca0086ad0 con 0x7f4ca81598a0 2023-10-06T05:07:40.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.916+0000 7f4caca2c700 1 --2- 172.21.15.98:0/3762118828 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f4c94066f30 0x7f4c940693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:40.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.916+0000 7f4caca2c700 1 --2- 172.21.15.98:0/3762118828 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f4c94066f30 0x7f4c940693e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4ca4007fd0 tx=0x7f4ca4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:40.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:40.920+0000 7f4c9cff9700 1 -- 172.21.15.98:0/3762118828 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4ca8155d90 con 0x7f4ca81598a0 2023-10-06T05:07:41.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.105+0000 7f4caec90700 1 -- 172.21.15.98:0/3762118828 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f4ca8155d90 con 0x7f4ca81598a0 2023-10-06T05:07:41.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.105+0000 7f4c9cff9700 1 -- 172.21.15.98:0/3762118828 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v31) v1 ==== 92+0+10 (secure 0 0 0) 0x7f4ca8155d90 con 0x7f4ca81598a0 2023-10-06T05:07:41.107 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 8 2023-10-06T05:07:41.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.107+0000 7f4caec90700 1 -- 172.21.15.98:0/3762118828 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f4c94066f30 msgr2=0x7f4c940693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:41.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.107+0000 7f4caec90700 1 --2- 172.21.15.98:0/3762118828 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f4c94066f30 0x7f4c940693e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4ca4007fd0 tx=0x7f4ca4009040 comp rx=0 tx=0).stop 2023-10-06T05:07:41.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.107+0000 7f4caec90700 1 -- 172.21.15.98:0/3762118828 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ca81598a0 msgr2=0x7f4ca815a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:41.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.107+0000 7f4caec90700 1 --2- 172.21.15.98:0/3762118828 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ca81598a0 0x7f4ca815a720 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f4ca00094c0 tx=0x7f4ca0016000 comp rx=0 tx=0).stop 2023-10-06T05:07:41.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.107+0000 7f4caec90700 1 -- 172.21.15.98:0/3762118828 shutdown_connections 2023-10-06T05:07:41.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.107+0000 7f4caec90700 1 --2- 172.21.15.98:0/3762118828 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f4c94066f30 0x7f4c940693e0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:41.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.107+0000 7f4caec90700 1 --2- 172.21.15.98:0/3762118828 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ca81598a0 0x7f4ca815a720 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:41.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.107+0000 7f4caec90700 1 --2- 172.21.15.98:0/3762118828 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4ca80cbc70 0x7f4ca8155640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:41.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.107+0000 7f4caec90700 1 --2- 172.21.15.98:0/3762118828 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4ca81594d0 0x7f4ca815c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:41.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.107+0000 7f4caec90700 1 -- 172.21.15.98:0/3762118828 >> 172.21.15.98:0/3762118828 conn(0x7f4ca8089e50 msgr2=0x7f4ca8149e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:41.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.107+0000 7f4caec90700 1 -- 172.21.15.98:0/3762118828 shutdown_connections 2023-10-06T05:07:41.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.107+0000 7f4caec90700 1 -- 172.21.15.98:0/3762118828 wait complete. 2023-10-06T05:07:41.120 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-06T05:07:41.237 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.236+0000 7fed00aa5700 1 Processor -- start 2023-10-06T05:07:41.238 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.236+0000 7fed00aa5700 1 -- start start 2023-10-06T05:07:41.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.237+0000 7fed00aa5700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fecfc0b0a10 0x7fecfc0cbee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:41.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.237+0000 7fed00aa5700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fecfc0cc420 0x7fecfc0c9050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:41.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.237+0000 7fed00aa5700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecfc0c53c0 con 0x7fecfc0b0640 2023-10-06T05:07:41.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.237+0000 7fed00aa5700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecfc0ca790 con 0x7fecfc0b0a10 2023-10-06T05:07:41.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.237+0000 7fed00aa5700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecfc0ccd00 con 0x7fecfc0cc420 2023-10-06T05:07:41.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.237+0000 7fecf1d9b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fecfc0b0a10 0x7fecfc0cbee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:41.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.237+0000 7fecf1d9b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fecfc0b0a10 0x7fecfc0cbee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60302/0 (socket says 172.21.15.98:60302) 2023-10-06T05:07:41.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.237+0000 7fecf1d9b700 1 -- 172.21.15.98:0/2844333549 learned_addr learned my addr 172.21.15.98:0/2844333549 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:41.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.237+0000 7fecfad9d700 1 --2- 172.21.15.98:0/2844333549 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fecfc0cc420 0x7fecfc0c9050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:41.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.237+0000 7fecf1d9b700 1 -- 172.21.15.98:0/2844333549 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fecfc0cc420 msgr2=0x7fecfc0c9050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:41.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.237+0000 7fecf1d9b700 1 --2- 172.21.15.98:0/2844333549 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fecfc0cc420 0x7fecfc0c9050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:41.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.237+0000 7fecf1d9b700 1 -- 172.21.15.98:0/2844333549 >> v1:172.21.15.174:6789/0 conn(0x7fecfc0b0640 legacy=0x7fecfc0cecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:41.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.237+0000 7fecf1d9b700 1 -- 172.21.15.98:0/2844333549 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecfc0ccfe0 con 0x7fecfc0b0a10 2023-10-06T05:07:41.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.237+0000 7fecfad9d700 1 --2- 172.21.15.98:0/2844333549 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fecfc0cc420 0x7fecfc0c9050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:07:41.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.238+0000 7fecf1d9b700 1 --2- 172.21.15.98:0/2844333549 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fecfc0b0a10 0x7fecfc0cbee0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fece40099b0 tx=0x7fece4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9926c5df1c78f9bf server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:41.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.238+0000 7fecf9d9b700 1 -- 172.21.15.98:0/2844333549 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece400b820 con 0x7fecfc0b0a10 2023-10-06T05:07:41.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.238+0000 7fecf9d9b700 1 -- 172.21.15.98:0/2844333549 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fece400b980 con 0x7fecfc0b0a10 2023-10-06T05:07:41.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.238+0000 7fecf9d9b700 1 -- 172.21.15.98:0/2844333549 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece400bd20 con 0x7fecfc0b0a10 2023-10-06T05:07:41.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.238+0000 7fed00aa5700 1 -- 172.21.15.98:0/2844333549 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fecfc0b0a10 msgr2=0x7fecfc0cbee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:41.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.238+0000 7fed00aa5700 1 --2- 172.21.15.98:0/2844333549 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fecfc0b0a10 0x7fecfc0cbee0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fece40099b0 tx=0x7fece4009250 comp rx=0 tx=0).stop 2023-10-06T05:07:41.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.238+0000 7fed00aa5700 1 -- 172.21.15.98:0/2844333549 shutdown_connections 2023-10-06T05:07:41.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.238+0000 7fed00aa5700 1 --2- 172.21.15.98:0/2844333549 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fecfc0cc420 0x7fecfc0c9050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:41.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.238+0000 7fed00aa5700 1 --2- 172.21.15.98:0/2844333549 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fecfc0b0a10 0x7fecfc0cbee0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:41.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.238+0000 7fed00aa5700 1 -- 172.21.15.98:0/2844333549 >> 172.21.15.98:0/2844333549 conn(0x7fecfc087fc0 msgr2=0x7fecfc0ce310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:41.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.238+0000 7fed00aa5700 1 -- 172.21.15.98:0/2844333549 shutdown_connections 2023-10-06T05:07:41.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.238+0000 7fed00aa5700 1 -- 172.21.15.98:0/2844333549 wait complete. 2023-10-06T05:07:41.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.239+0000 7fed00aa5700 1 Processor -- start 2023-10-06T05:07:41.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.239+0000 7fed00aa5700 1 -- start start 2023-10-06T05:07:41.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.240+0000 7fed00aa5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fecfc0b0640 0x7fecfc15a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:41.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.240+0000 7fed00aa5700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fecfc0b0a10 0x7fecfc0b2a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:41.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.240+0000 7fed00aa5700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fecfc0cc420 0x7fecfc154be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:41.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.240+0000 7fed00aa5700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecfc159920 con 0x7fecfc0b0640 2023-10-06T05:07:41.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.240+0000 7fed00aa5700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecfc15ae10 con 0x7fecfc0b0a10 2023-10-06T05:07:41.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.240+0000 7fed00aa5700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecfc15b060 con 0x7fecfc0cc420 2023-10-06T05:07:41.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.240+0000 7fecfa59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fecfc0b0640 0x7fecfc15a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:41.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.240+0000 7fecfa59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fecfc0b0640 0x7fecfc15a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60306/0 (socket says 172.21.15.98:60306) 2023-10-06T05:07:41.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.240+0000 7fecfa59c700 1 -- 172.21.15.98:0/3835994232 learned_addr learned my addr 172.21.15.98:0/3835994232 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:41.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.240+0000 7fecf1d9b700 1 --2- 172.21.15.98:0/3835994232 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fecfc0b0a10 0x7fecfc0b2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:41.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.240+0000 7fecfad9d700 1 --2- 172.21.15.98:0/3835994232 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fecfc0cc420 0x7fecfc154be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:41.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.241+0000 7fecfa59c700 1 -- 172.21.15.98:0/3835994232 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fecfc0cc420 msgr2=0x7fecfc154be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:41.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.241+0000 7fecfa59c700 1 --2- 172.21.15.98:0/3835994232 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fecfc0cc420 0x7fecfc154be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:41.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.241+0000 7fecfa59c700 1 -- 172.21.15.98:0/3835994232 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fecfc0b0a10 msgr2=0x7fecfc0b2a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:41.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.241+0000 7fecfa59c700 1 --2- 172.21.15.98:0/3835994232 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fecfc0b0a10 0x7fecfc0b2a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:41.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.241+0000 7fecfa59c700 1 -- 172.21.15.98:0/3835994232 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fece40094f0 con 0x7fecfc0b0640 2023-10-06T05:07:41.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.241+0000 7fecfad9d700 1 --2- 172.21.15.98:0/3835994232 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fecfc0cc420 0x7fecfc154be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:07:41.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.241+0000 7fecfa59c700 1 --2- 172.21.15.98:0/3835994232 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fecfc0b0640 0x7fecfc15a8d0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fecec00b5e0 tx=0x7fecec0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:41.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.241+0000 7fecf2ffd700 1 -- 172.21.15.98:0/3835994232 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecec00f510 con 0x7fecfc0b0640 2023-10-06T05:07:41.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.241+0000 7fecf2ffd700 1 -- 172.21.15.98:0/3835994232 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecec00f690 con 0x7fecfc0b0640 2023-10-06T05:07:41.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.241+0000 7fed00aa5700 1 -- 172.21.15.98:0/3835994232 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecfc15b340 con 0x7fecfc0b0640 2023-10-06T05:07:41.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.241+0000 7fecf2ffd700 1 -- 172.21.15.98:0/3835994232 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecec00fa50 con 0x7fecfc0b0640 2023-10-06T05:07:41.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.241+0000 7fed00aa5700 1 -- 172.21.15.98:0/3835994232 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecfc15bbe0 con 0x7fecfc0b0640 2023-10-06T05:07:41.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.242+0000 7fed00aa5700 1 -- 172.21.15.98:0/3835994232 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecfc064a90 con 0x7fecfc0b0640 2023-10-06T05:07:41.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.242+0000 7fecf2ffd700 1 -- 172.21.15.98:0/3835994232 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fecec01a400 con 0x7fecfc0b0640 2023-10-06T05:07:41.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.243+0000 7fecf2ffd700 1 --2- 172.21.15.98:0/3835994232 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fecdc066f30 0x7fecdc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:41.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.243+0000 7fecf2ffd700 1 -- 172.21.15.98:0/3835994232 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6867+0+0 (secure 0 0 0) 0x7fecec084460 con 0x7fecfc0b0640 2023-10-06T05:07:41.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.243+0000 7fecf1d9b700 1 --2- 172.21.15.98:0/3835994232 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fecdc066f30 0x7fecdc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:41.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.243+0000 7fecf1d9b700 1 --2- 172.21.15.98:0/3835994232 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fecdc066f30 0x7fecdc0693e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fece40099b0 tx=0x7fece4016000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:41.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.247+0000 7fecf2ffd700 1 -- 172.21.15.98:0/3835994232 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fecec0512c0 con 0x7fecfc0b0640 2023-10-06T05:07:41.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.429+0000 7fed00aa5700 1 -- 172.21.15.98:0/3835994232 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fecfc15b500 con 0x7fecfc0b0640 2023-10-06T05:07:41.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.429+0000 7fecf2ffd700 1 -- 172.21.15.98:0/3835994232 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v31) v1 ==== 104+0+11 (secure 0 0 0) 0x7fecec0548e0 con 0x7fecfc0b0640 2023-10-06T05:07:41.431 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2023-10-06T05:07:41.432 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.431+0000 7fed00aa5700 1 -- 172.21.15.98:0/3835994232 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fecdc066f30 msgr2=0x7fecdc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:41.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.431+0000 7fed00aa5700 1 --2- 172.21.15.98:0/3835994232 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fecdc066f30 0x7fecdc0693e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fece40099b0 tx=0x7fece4016000 comp rx=0 tx=0).stop 2023-10-06T05:07:41.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.431+0000 7fed00aa5700 1 -- 172.21.15.98:0/3835994232 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fecfc0b0640 msgr2=0x7fecfc15a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:41.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.431+0000 7fed00aa5700 1 --2- 172.21.15.98:0/3835994232 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fecfc0b0640 0x7fecfc15a8d0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fecec00b5e0 tx=0x7fecec0090f0 comp rx=0 tx=0).stop 2023-10-06T05:07:41.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.431+0000 7fed00aa5700 1 -- 172.21.15.98:0/3835994232 shutdown_connections 2023-10-06T05:07:41.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.431+0000 7fed00aa5700 1 --2- 172.21.15.98:0/3835994232 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fecfc0cc420 0x7fecfc154be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:41.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.431+0000 7fed00aa5700 1 --2- 172.21.15.98:0/3835994232 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fecdc066f30 0x7fecdc0693e0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:41.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.431+0000 7fed00aa5700 1 --2- 172.21.15.98:0/3835994232 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fecfc0b0640 0x7fecfc15a8d0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:41.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.431+0000 7fed00aa5700 1 --2- 172.21.15.98:0/3835994232 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fecfc0b0a10 0x7fecfc0b2a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:41.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.431+0000 7fed00aa5700 1 -- 172.21.15.98:0/3835994232 >> 172.21.15.98:0/3835994232 conn(0x7fecfc087fc0 msgr2=0x7fecfc156cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:41.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.431+0000 7fed00aa5700 1 -- 172.21.15.98:0/3835994232 shutdown_connections 2023-10-06T05:07:41.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.431+0000 7fed00aa5700 1 -- 172.21.15.98:0/3835994232 wait complete. 2023-10-06T05:07:41.444 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-06T05:07:41.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.560+0000 7fc432fb4700 1 Processor -- start 2023-10-06T05:07:41.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.560+0000 7fc432fb4700 1 -- start start 2023-10-06T05:07:41.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.561+0000 7fc432fb4700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fc42c1594d0 0x7fc42c0cb560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:41.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.561+0000 7fc432fb4700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc42c0cbaa0 0x7fc42c0ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:41.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.561+0000 7fc432fb4700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc42c0cbeb0 con 0x7fc42c1598a0 2023-10-06T05:07:41.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.561+0000 7fc432fb4700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc42c0cf5d0 con 0x7fc42c1594d0 2023-10-06T05:07:41.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.561+0000 7fc432fb4700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc42c0cf8b0 con 0x7fc42c0cbaa0 2023-10-06T05:07:41.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.561+0000 7fc42bfff700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fc42c1598a0 0x7fc42c0ce7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:40408/0 (socket says 172.21.15.98:40408) 2023-10-06T05:07:41.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.561+0000 7fc42bfff700 1 -- 172.21.15.98:0/3576337423 learned_addr learned my addr 172.21.15.98:0/3576337423 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:07:41.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.561+0000 7fc430d50700 1 --2- 172.21.15.98:0/3576337423 >> v2:172.21.15.174:3300/0 conn(0x7fc42c1594d0 0x7fc42c0cb560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:41.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.561+0000 7fc431551700 1 --2- 172.21.15.98:0/3576337423 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc42c0cbaa0 0x7fc42c0ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:41.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.561+0000 7fc42b7fe700 1 -- 172.21.15.98:0/3576337423 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2103198137 0 0) 0x7fc42c0cbeb0 con 0x7fc42c1598a0 2023-10-06T05:07:41.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.562+0000 7fc42b7fe700 1 -- 172.21.15.98:0/3576337423 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc414003570 con 0x7fc42c1598a0 2023-10-06T05:07:41.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.562+0000 7fc42b7fe700 1 -- 172.21.15.98:0/3576337423 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1783524846 0 0) 0x7fc414003570 con 0x7fc42c1598a0 2023-10-06T05:07:41.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.562+0000 7fc42b7fe700 1 -- 172.21.15.98:0/3576337423 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc42c0cbaa0 msgr2=0x7fc42c0ceef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:41.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.562+0000 7fc42b7fe700 1 --2- 172.21.15.98:0/3576337423 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc42c0cbaa0 0x7fc42c0ceef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:41.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.562+0000 7fc42b7fe700 1 -- 172.21.15.98:0/3576337423 >> v2:172.21.15.174:3300/0 conn(0x7fc42c1594d0 msgr2=0x7fc42c0cb560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:41.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.562+0000 7fc42b7fe700 1 --2- 172.21.15.98:0/3576337423 >> v2:172.21.15.174:3300/0 conn(0x7fc42c1594d0 0x7fc42c0cb560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:41.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.562+0000 7fc42b7fe700 1 -- 172.21.15.98:0/3576337423 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc42c0cfb90 con 0x7fc42c1598a0 2023-10-06T05:07:41.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.562+0000 7fc42b7fe700 1 -- 172.21.15.98:0/3576337423 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1543991876 0 0) 0x7fc420001470 con 0x7fc42c1598a0 2023-10-06T05:07:41.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.562+0000 7fc42b7fe700 1 -- 172.21.15.98:0/3576337423 >> v1:172.21.15.98:6789/0 conn(0x7fc42c1598a0 legacy=0x7fc42c0ce7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:41.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.562+0000 7fc42b7fe700 1 --2- 172.21.15.98:0/3576337423 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc414003a60 0x7fc414003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:41.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.562+0000 7fc42b7fe700 1 --2- 172.21.15.98:0/3576337423 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc414004810 0x7fc414004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:41.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.562+0000 7fc42b7fe700 1 --2- 172.21.15.98:0/3576337423 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc42c0cbaa0 0x7fc4140051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:41.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.562+0000 7fc42b7fe700 1 -- 172.21.15.98:0/3576337423 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc42c0cf5d0 con 0x7fc414004810 2023-10-06T05:07:41.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.562+0000 7fc42b7fe700 1 -- 172.21.15.98:0/3576337423 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc42c0cf8b0 con 0x7fc414003a60 2023-10-06T05:07:41.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.562+0000 7fc42b7fe700 1 -- 172.21.15.98:0/3576337423 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc41400ba60 con 0x7fc42c0cbaa0 2023-10-06T05:07:41.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.562+0000 7fc430d50700 1 --2- 172.21.15.98:0/3576337423 >> v2:172.21.15.174:3300/0 conn(0x7fc42c1594d0 0x7fc42c0cb560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T05:07:41.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.562+0000 7fc42bfff700 1 --2- 172.21.15.98:0/3576337423 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc414004810 0x7fc414004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:41.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.562+0000 7fc431551700 1 --2- 172.21.15.98:0/3576337423 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc42c0cbaa0 0x7fc4140051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:41.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.562+0000 7fc430d50700 1 --2- 172.21.15.98:0/3576337423 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc414003a60 0x7fc414003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:41.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.562+0000 7fc42bfff700 1 -- 172.21.15.98:0/3576337423 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc42c0cbaa0 msgr2=0x7fc4140051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:41.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.562+0000 7fc42bfff700 1 --2- 172.21.15.98:0/3576337423 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc42c0cbaa0 0x7fc4140051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:41.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.562+0000 7fc42bfff700 1 -- 172.21.15.98:0/3576337423 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc414003a60 msgr2=0x7fc414003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:41.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.562+0000 7fc42bfff700 1 --2- 172.21.15.98:0/3576337423 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc414003a60 0x7fc414003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:41.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.562+0000 7fc42bfff700 1 -- 172.21.15.98:0/3576337423 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc414003570 con 0x7fc414004810 2023-10-06T05:07:41.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.562+0000 7fc430d50700 1 --2- 172.21.15.98:0/3576337423 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc414003a60 0x7fc414003e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:07:41.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.562+0000 7fc42bfff700 1 --2- 172.21.15.98:0/3576337423 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc414004810 0x7fc414004c80 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fc42c067cf0 tx=0x7fc420007940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7e60b20a03446d7 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:41.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.562+0000 7fc42b7fe700 1 -- 172.21.15.98:0/3576337423 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc42000cbc0 con 0x7fc414004810 2023-10-06T05:07:41.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.562+0000 7fc42b7fe700 1 -- 172.21.15.98:0/3576337423 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc42000cd20 con 0x7fc414004810 2023-10-06T05:07:41.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.562+0000 7fc42b7fe700 1 -- 172.21.15.98:0/3576337423 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc420005b50 con 0x7fc414004810 2023-10-06T05:07:41.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.563+0000 7fc432fb4700 1 -- 172.21.15.98:0/3576337423 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc414004810 msgr2=0x7fc414004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:41.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.563+0000 7fc432fb4700 1 --2- 172.21.15.98:0/3576337423 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc414004810 0x7fc414004c80 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fc42c067cf0 tx=0x7fc420007940 comp rx=0 tx=0).stop 2023-10-06T05:07:41.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.563+0000 7fc431551700 1 -- 172.21.15.98:0/3576337423 reap_dead start 2023-10-06T05:07:41.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.563+0000 7fc432fb4700 1 -- 172.21.15.98:0/3576337423 shutdown_connections 2023-10-06T05:07:41.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.563+0000 7fc432fb4700 1 -- 172.21.15.98:0/3576337423 >> 172.21.15.98:0/3576337423 conn(0x7fc42c089e50 msgr2=0x7fc42c099770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:41.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.563+0000 7fc432fb4700 1 -- 172.21.15.98:0/3576337423 shutdown_connections 2023-10-06T05:07:41.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.563+0000 7fc432fb4700 1 -- 172.21.15.98:0/3576337423 wait complete. 2023-10-06T05:07:41.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.564+0000 7fc432fb4700 1 Processor -- start 2023-10-06T05:07:41.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.564+0000 7fc432fb4700 1 -- start start 2023-10-06T05:07:41.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.565+0000 7fc432fb4700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc414004810 0x7fc42c1699c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:41.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.565+0000 7fc432fb4700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc42c0695b0 0x7fc42c069a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:41.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.565+0000 7fc432fb4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc42c15af80 0x7fc42c15b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:41.573 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.565+0000 7fc432fb4700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc42c069f60 con 0x7fc42c15af80 2023-10-06T05:07:41.573 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.565+0000 7fc432fb4700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc42c0c8eb0 con 0x7fc414004810 2023-10-06T05:07:41.573 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.565+0000 7fc432fb4700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc42c13ca70 con 0x7fc42c0695b0 2023-10-06T05:07:41.573 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.565+0000 7fc431551700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc42c15af80 0x7fc42c15b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:41.573 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.565+0000 7fc431551700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc42c15af80 0x7fc42c15b420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60332/0 (socket says 172.21.15.98:60332) 2023-10-06T05:07:41.573 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.565+0000 7fc431551700 1 -- 172.21.15.98:0/441512604 learned_addr learned my addr 172.21.15.98:0/441512604 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:41.573 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.565+0000 7fc42bfff700 1 --2- 172.21.15.98:0/441512604 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc42c0695b0 0x7fc42c069a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:41.573 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.565+0000 7fc430d50700 1 --2- 172.21.15.98:0/441512604 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc414004810 0x7fc42c1699c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:41.573 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.565+0000 7fc431551700 1 -- 172.21.15.98:0/441512604 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc42c0695b0 msgr2=0x7fc42c069a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:41.573 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.565+0000 7fc431551700 1 --2- 172.21.15.98:0/441512604 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc42c0695b0 0x7fc42c069a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:41.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.565+0000 7fc431551700 1 -- 172.21.15.98:0/441512604 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc414004810 msgr2=0x7fc42c1699c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:41.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.565+0000 7fc431551700 1 --2- 172.21.15.98:0/441512604 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc414004810 0x7fc42c1699c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:41.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.565+0000 7fc431551700 1 -- 172.21.15.98:0/441512604 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc42c15bcb0 con 0x7fc42c15af80 2023-10-06T05:07:41.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.565+0000 7fc42bfff700 1 --2- 172.21.15.98:0/441512604 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc42c0695b0 0x7fc42c069a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:07:41.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.565+0000 7fc430d50700 1 --2- 172.21.15.98:0/441512604 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc414004810 0x7fc42c1699c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:07:41.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.566+0000 7fc431551700 1 --2- 172.21.15.98:0/441512604 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc42c15af80 0x7fc42c15b420 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fc42c088cb0 tx=0x7fc4240091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:41.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.566+0000 7fc428ff9700 1 -- 172.21.15.98:0/441512604 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc42400f910 con 0x7fc42c15af80 2023-10-06T05:07:41.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.566+0000 7fc428ff9700 1 -- 172.21.15.98:0/441512604 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc42400fa70 con 0x7fc42c15af80 2023-10-06T05:07:41.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.566+0000 7fc432fb4700 1 -- 172.21.15.98:0/441512604 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc42c0c8580 con 0x7fc42c15af80 2023-10-06T05:07:41.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.566+0000 7fc428ff9700 1 -- 172.21.15.98:0/441512604 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc42400fe30 con 0x7fc42c15af80 2023-10-06T05:07:41.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.566+0000 7fc432fb4700 1 -- 172.21.15.98:0/441512604 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc42c0c8b40 con 0x7fc42c15af80 2023-10-06T05:07:41.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.567+0000 7fc432fb4700 1 -- 172.21.15.98:0/441512604 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc42c0c9020 con 0x7fc42c15af80 2023-10-06T05:07:41.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.567+0000 7fc428ff9700 1 -- 172.21.15.98:0/441512604 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fc424014650 con 0x7fc42c15af80 2023-10-06T05:07:41.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.567+0000 7fc428ff9700 1 --2- 172.21.15.98:0/441512604 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fc418066f30 0x7fc4180693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:41.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.568+0000 7fc428ff9700 1 -- 172.21.15.98:0/441512604 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6867+0+0 (secure 0 0 0) 0x7fc424084fb0 con 0x7fc42c15af80 2023-10-06T05:07:41.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.568+0000 7fc430d50700 1 --2- 172.21.15.98:0/441512604 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fc418066f30 0x7fc4180693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:41.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.568+0000 7fc430d50700 1 --2- 172.21.15.98:0/441512604 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fc418066f30 0x7fc4180693e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fc41c0099f0 tx=0x7fc41c00b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:41.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.572+0000 7fc428ff9700 1 -- 172.21.15.98:0/441512604 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc424051e10 con 0x7fc42c15af80 2023-10-06T05:07:41.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.758+0000 7fc432fb4700 1 -- 172.21.15.98:0/441512604 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc42c064a90 con 0x7fc42c15af80 2023-10-06T05:07:41.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.759+0000 7fc428ff9700 1 -- 172.21.15.98:0/441512604 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v31) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc424000e60 con 0x7fc42c15af80 2023-10-06T05:07:41.760 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2023-10-06T05:07:41.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.761+0000 7fc432fb4700 1 -- 172.21.15.98:0/441512604 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fc418066f30 msgr2=0x7fc4180693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:41.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.761+0000 7fc432fb4700 1 --2- 172.21.15.98:0/441512604 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fc418066f30 0x7fc4180693e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fc41c0099f0 tx=0x7fc41c00b040 comp rx=0 tx=0).stop 2023-10-06T05:07:41.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.761+0000 7fc432fb4700 1 -- 172.21.15.98:0/441512604 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc42c15af80 msgr2=0x7fc42c15b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:41.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.761+0000 7fc432fb4700 1 --2- 172.21.15.98:0/441512604 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc42c15af80 0x7fc42c15b420 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fc42c088cb0 tx=0x7fc4240091a0 comp rx=0 tx=0).stop 2023-10-06T05:07:41.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.761+0000 7fc432fb4700 1 -- 172.21.15.98:0/441512604 shutdown_connections 2023-10-06T05:07:41.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.761+0000 7fc432fb4700 1 --2- 172.21.15.98:0/441512604 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fc418066f30 0x7fc4180693e0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:41.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.761+0000 7fc432fb4700 1 --2- 172.21.15.98:0/441512604 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc42c15af80 0x7fc42c15b420 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:41.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.761+0000 7fc432fb4700 1 --2- 172.21.15.98:0/441512604 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc414004810 0x7fc42c1699c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:41.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.761+0000 7fc432fb4700 1 --2- 172.21.15.98:0/441512604 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc42c0695b0 0x7fc42c069a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:41.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.761+0000 7fc432fb4700 1 -- 172.21.15.98:0/441512604 >> 172.21.15.98:0/441512604 conn(0x7fc42c089e50 msgr2=0x7fc42c158810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:41.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.762+0000 7fc432fb4700 1 -- 172.21.15.98:0/441512604 shutdown_connections 2023-10-06T05:07:41.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.762+0000 7fc432fb4700 1 -- 172.21.15.98:0/441512604 wait complete. 2023-10-06T05:07:41.776 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-06T05:07:41.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.892+0000 7f63bf245700 1 Processor -- start 2023-10-06T05:07:41.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.892+0000 7f63bf245700 1 -- start start 2023-10-06T05:07:41.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.893+0000 7f63bf245700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f63b81598a0 0x7f63b80ce7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:41.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.893+0000 7f63bf245700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f63b80cbc70 0x7f63b80ced20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:41.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.893+0000 7f63bf245700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63b80cf5d0 con 0x7f63b81594d0 2023-10-06T05:07:41.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.893+0000 7f63bf245700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63b80cf8b0 con 0x7f63b80cbc70 2023-10-06T05:07:41.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.893+0000 7f63bf245700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63b80cfb90 con 0x7f63b81598a0 2023-10-06T05:07:41.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.893+0000 7f63bd7e2700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f63b80cbc70 0x7f63b80ced20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:41.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.893+0000 7f63bd7e2700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f63b80cbc70 0x7f63b80ced20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60344/0 (socket says 172.21.15.98:60344) 2023-10-06T05:07:41.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.893+0000 7f63bd7e2700 1 -- 172.21.15.98:0/829308766 learned_addr learned my addr 172.21.15.98:0/829308766 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:41.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.894+0000 7f63b7fff700 1 --2- 172.21.15.98:0/829308766 >> v2:172.21.15.174:3300/0 conn(0x7f63b81598a0 0x7f63b80ce7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:41.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.894+0000 7f63bd7e2700 1 -- 172.21.15.98:0/829308766 >> v2:172.21.15.174:3300/0 conn(0x7f63b81598a0 msgr2=0x7f63b80ce7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:41.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.894+0000 7f63bd7e2700 1 --2- 172.21.15.98:0/829308766 >> v2:172.21.15.174:3300/0 conn(0x7f63b81598a0 0x7f63b80ce7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:41.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.894+0000 7f63bd7e2700 1 -- 172.21.15.98:0/829308766 >> v1:172.21.15.174:6789/0 conn(0x7f63b81594d0 legacy=0x7f63b80cb560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:41.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.894+0000 7f63bd7e2700 1 -- 172.21.15.98:0/829308766 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63b80cfe70 con 0x7f63b80cbc70 2023-10-06T05:07:41.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.894+0000 7f63b7fff700 1 --2- 172.21.15.98:0/829308766 >> v2:172.21.15.174:3300/0 conn(0x7f63b81598a0 0x7f63b80ce7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:07:41.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.894+0000 7f63bd7e2700 1 --2- 172.21.15.98:0/829308766 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63b80cbc70 0x7f63b80ced20 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f63b80c7370 tx=0x7f63b0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a41f70e187dcbb39 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:41.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.894+0000 7f63b77fe700 1 -- 172.21.15.98:0/829308766 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b000b820 con 0x7f63b80cbc70 2023-10-06T05:07:41.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.894+0000 7f63b77fe700 1 -- 172.21.15.98:0/829308766 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63b000b980 con 0x7f63b80cbc70 2023-10-06T05:07:41.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.894+0000 7f63b77fe700 1 -- 172.21.15.98:0/829308766 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b000bd20 con 0x7f63b80cbc70 2023-10-06T05:07:41.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.895+0000 7f63bf245700 1 -- 172.21.15.98:0/829308766 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63b80cbc70 msgr2=0x7f63b80ced20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:41.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.895+0000 7f63bf245700 1 --2- 172.21.15.98:0/829308766 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63b80cbc70 0x7f63b80ced20 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f63b80c7370 tx=0x7f63b0009250 comp rx=0 tx=0).stop 2023-10-06T05:07:41.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.895+0000 7f63bf245700 1 -- 172.21.15.98:0/829308766 shutdown_connections 2023-10-06T05:07:41.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.895+0000 7f63bf245700 1 --2- 172.21.15.98:0/829308766 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63b80cbc70 0x7f63b80ced20 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:41.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.895+0000 7f63bf245700 1 --2- 172.21.15.98:0/829308766 >> v2:172.21.15.174:3300/0 conn(0x7f63b81598a0 0x7f63b80ce7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:41.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.895+0000 7f63bf245700 1 -- 172.21.15.98:0/829308766 >> 172.21.15.98:0/829308766 conn(0x7f63b8089e50 msgr2=0x7f63b8099770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:41.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.895+0000 7f63bf245700 1 -- 172.21.15.98:0/829308766 shutdown_connections 2023-10-06T05:07:41.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.895+0000 7f63bf245700 1 -- 172.21.15.98:0/829308766 wait complete. 2023-10-06T05:07:41.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.896+0000 7f63bf245700 1 Processor -- start 2023-10-06T05:07:41.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.896+0000 7f63bf245700 1 -- start start 2023-10-06T05:07:41.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.897+0000 7f63bf245700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63b80cbc70 0x7f63b8169820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:41.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.897+0000 7f63bf245700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f63b81594d0 0x7f63b8157f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:41.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.897+0000 7f63bf245700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63b81598a0 0x7f63b816d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:41.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.897+0000 7f63bf245700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63b80ca450 con 0x7f63b80cbc70 2023-10-06T05:07:41.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.897+0000 7f63bf245700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63b8157ac0 con 0x7f63b81598a0 2023-10-06T05:07:41.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.897+0000 7f63bf245700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63b8169d60 con 0x7f63b81594d0 2023-10-06T05:07:41.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.897+0000 7f63bcfe1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63b80cbc70 0x7f63b8169820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:41.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.897+0000 7f63bcfe1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63b80cbc70 0x7f63b8169820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60350/0 (socket says 172.21.15.98:60350) 2023-10-06T05:07:41.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.897+0000 7f63bcfe1700 1 -- 172.21.15.98:0/141260295 learned_addr learned my addr 172.21.15.98:0/141260295 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:41.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.897+0000 7f63bd7e2700 1 --2- 172.21.15.98:0/141260295 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63b81598a0 0x7f63b816d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:41.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.897+0000 7f63b7fff700 1 --2- 172.21.15.98:0/141260295 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f63b81594d0 0x7f63b8157f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:41.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.897+0000 7f63bcfe1700 1 -- 172.21.15.98:0/141260295 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f63b81594d0 msgr2=0x7f63b8157f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:41.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.897+0000 7f63bcfe1700 1 --2- 172.21.15.98:0/141260295 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f63b81594d0 0x7f63b8157f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:41.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.897+0000 7f63bcfe1700 1 -- 172.21.15.98:0/141260295 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63b81598a0 msgr2=0x7f63b816d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:41.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.897+0000 7f63bcfe1700 1 --2- 172.21.15.98:0/141260295 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63b81598a0 0x7f63b816d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:41.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.897+0000 7f63bcfe1700 1 -- 172.21.15.98:0/141260295 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63b00094f0 con 0x7f63b80cbc70 2023-10-06T05:07:41.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.897+0000 7f63b7fff700 1 --2- 172.21.15.98:0/141260295 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f63b81594d0 0x7f63b8157f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:07:41.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.897+0000 7f63bcfe1700 1 --2- 172.21.15.98:0/141260295 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63b80cbc70 0x7f63b8169820 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f63a800c980 tx=0x7f63a8002760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:41.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.897+0000 7f63b4ff9700 1 -- 172.21.15.98:0/141260295 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63a800ce70 con 0x7f63b80cbc70 2023-10-06T05:07:41.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.897+0000 7f63b4ff9700 1 -- 172.21.15.98:0/141260295 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63a800a7a0 con 0x7f63b80cbc70 2023-10-06T05:07:41.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.897+0000 7f63bf245700 1 -- 172.21.15.98:0/141260295 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63b8156eb0 con 0x7f63b80cbc70 2023-10-06T05:07:41.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.897+0000 7f63b4ff9700 1 -- 172.21.15.98:0/141260295 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63a800ab60 con 0x7f63b80cbc70 2023-10-06T05:07:41.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.897+0000 7f63bf245700 1 -- 172.21.15.98:0/141260295 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63b8157470 con 0x7f63b80cbc70 2023-10-06T05:07:41.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.898+0000 7f63bf245700 1 -- 172.21.15.98:0/141260295 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63b8064a90 con 0x7f63b80cbc70 2023-10-06T05:07:41.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.899+0000 7f63b4ff9700 1 -- 172.21.15.98:0/141260295 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f63a8012420 con 0x7f63b80cbc70 2023-10-06T05:07:41.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.899+0000 7f63b4ff9700 1 --2- 172.21.15.98:0/141260295 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f63a4066f30 0x7f63a40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:41.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.899+0000 7f63b4ff9700 1 -- 172.21.15.98:0/141260295 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6867+0+0 (secure 0 0 0) 0x7f63a8084d70 con 0x7f63b80cbc70 2023-10-06T05:07:41.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.899+0000 7f63b7fff700 1 --2- 172.21.15.98:0/141260295 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f63a4066f30 0x7f63a40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:41.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.899+0000 7f63b7fff700 1 --2- 172.21.15.98:0/141260295 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f63a4066f30 0x7f63a40693e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f63b8069450 tx=0x7f63ac008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:41.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:41.903+0000 7f63b4ff9700 1 -- 172.21.15.98:0/141260295 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f63a8055690 con 0x7f63b80cbc70 2023-10-06T05:07:42.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.088+0000 7f63bf245700 1 -- 172.21.15.98:0/141260295 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f63b80cf260 con 0x7f63b80cbc70 2023-10-06T05:07:42.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.088+0000 7f63b4ff9700 1 -- 172.21.15.98:0/141260295 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v31) v1 ==== 103+0+11 (secure 0 0 0) 0x7f63a8055220 con 0x7f63b80cbc70 2023-10-06T05:07:42.089 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2023-10-06T05:07:42.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.090+0000 7f63bf245700 1 -- 172.21.15.98:0/141260295 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f63a4066f30 msgr2=0x7f63a40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:42.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.090+0000 7f63bf245700 1 --2- 172.21.15.98:0/141260295 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f63a4066f30 0x7f63a40693e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f63b8069450 tx=0x7f63ac008040 comp rx=0 tx=0).stop 2023-10-06T05:07:42.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.090+0000 7f63bf245700 1 -- 172.21.15.98:0/141260295 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63b80cbc70 msgr2=0x7f63b8169820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:42.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.090+0000 7f63bf245700 1 --2- 172.21.15.98:0/141260295 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63b80cbc70 0x7f63b8169820 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f63a800c980 tx=0x7f63a8002760 comp rx=0 tx=0).stop 2023-10-06T05:07:42.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.090+0000 7f63bf245700 1 -- 172.21.15.98:0/141260295 shutdown_connections 2023-10-06T05:07:42.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.090+0000 7f63bf245700 1 --2- 172.21.15.98:0/141260295 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63b81598a0 0x7f63b816d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:42.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.090+0000 7f63bf245700 1 --2- 172.21.15.98:0/141260295 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f63a4066f30 0x7f63a40693e0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:42.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.090+0000 7f63bf245700 1 --2- 172.21.15.98:0/141260295 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63b80cbc70 0x7f63b8169820 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:42.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.090+0000 7f63bf245700 1 --2- 172.21.15.98:0/141260295 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f63b81594d0 0x7f63b8157f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:42.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.090+0000 7f63bf245700 1 -- 172.21.15.98:0/141260295 >> 172.21.15.98:0/141260295 conn(0x7f63b8089e50 msgr2=0x7f63b8099770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:42.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.091+0000 7f63bf245700 1 -- 172.21.15.98:0/141260295 shutdown_connections 2023-10-06T05:07:42.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.091+0000 7f63bf245700 1 -- 172.21.15.98:0/141260295 wait complete. 2023-10-06T05:07:42.106 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-06T05:07:42.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.219+0000 7fb90cd4f700 1 Processor -- start 2023-10-06T05:07:42.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.219+0000 7fb90cd4f700 1 -- start start 2023-10-06T05:07:42.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.220+0000 7fb90cd4f700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fb9080b2810 0x7fb9080cc080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:42.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.220+0000 7fb90cd4f700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9080cb7a0 con 0x7fb9080cbc30 2023-10-06T05:07:42.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.220+0000 7fb90cd4f700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9080b0860 con 0x7fb908159440 2023-10-06T05:07:42.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.220+0000 7fb90cd4f700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9080cb150 con 0x7fb9080b2810 2023-10-06T05:07:42.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.220+0000 7fb906ffd700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fb9080cbc30 0x7fb9080b2100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:40416/0 (socket says 172.21.15.98:40416) 2023-10-06T05:07:42.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.220+0000 7fb906ffd700 1 -- 172.21.15.98:0/1930438210 learned_addr learned my addr 172.21.15.98:0/1930438210 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:07:42.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.220+0000 7fb907fff700 1 --2- 172.21.15.98:0/1930438210 >> v2:172.21.15.174:3300/0 conn(0x7fb9080b2810 0x7fb9080cc080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:42.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.220+0000 7fb9067fc700 1 -- 172.21.15.98:0/1930438210 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3597943240 0 0) 0x7fb9080cb7a0 con 0x7fb9080cbc30 2023-10-06T05:07:42.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.220+0000 7fb9067fc700 1 -- 172.21.15.98:0/1930438210 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8f0003570 con 0x7fb9080cbc30 2023-10-06T05:07:42.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.221+0000 7fb907fff700 1 -- 172.21.15.98:0/1930438210 >> v1:172.21.15.174:6789/0 conn(0x7fb908159440 legacy=0x7fb9080b3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:42.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.221+0000 7fb907fff700 1 -- 172.21.15.98:0/1930438210 >> v1:172.21.15.98:6789/0 conn(0x7fb9080cbc30 legacy=0x7fb9080b2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:42.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.221+0000 7fb907fff700 1 -- 172.21.15.98:0/1930438210 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9080cb430 con 0x7fb9080b2810 2023-10-06T05:07:42.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.221+0000 7fb907fff700 1 --2- 172.21.15.98:0/1930438210 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9080b2810 0x7fb9080cc080 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb900009e60 tx=0x7fb900009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e029360f62970567 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:42.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.221+0000 7fb9067fc700 1 -- 172.21.15.98:0/1930438210 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb90000e070 con 0x7fb9080b2810 2023-10-06T05:07:42.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.221+0000 7fb9067fc700 1 -- 172.21.15.98:0/1930438210 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb90000bb60 con 0x7fb9080b2810 2023-10-06T05:07:42.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.221+0000 7fb9067fc700 1 -- 172.21.15.98:0/1930438210 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb900012430 con 0x7fb9080b2810 2023-10-06T05:07:42.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.221+0000 7fb90cd4f700 1 -- 172.21.15.98:0/1930438210 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9080b2810 msgr2=0x7fb9080cc080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:42.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.221+0000 7fb90cd4f700 1 --2- 172.21.15.98:0/1930438210 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9080b2810 0x7fb9080cc080 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb900009e60 tx=0x7fb900009780 comp rx=0 tx=0).stop 2023-10-06T05:07:42.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.221+0000 7fb90cd4f700 1 -- 172.21.15.98:0/1930438210 shutdown_connections 2023-10-06T05:07:42.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.221+0000 7fb90cd4f700 1 --2- 172.21.15.98:0/1930438210 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9080b2810 0x7fb9080cc080 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:42.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.221+0000 7fb90cd4f700 1 -- 172.21.15.98:0/1930438210 >> 172.21.15.98:0/1930438210 conn(0x7fb90806ece0 msgr2=0x7fb90806f0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:42.225 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.221+0000 7fb90cd4f700 1 -- 172.21.15.98:0/1930438210 shutdown_connections 2023-10-06T05:07:42.225 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.221+0000 7fb90cd4f700 1 -- 172.21.15.98:0/1930438210 wait complete. 2023-10-06T05:07:42.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.223+0000 7fb90cd4f700 1 Processor -- start 2023-10-06T05:07:42.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.223+0000 7fb90cd4f700 1 -- start start 2023-10-06T05:07:42.227 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.223+0000 7fb90cd4f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9080b2810 0x7fb9080c7c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:42.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.223+0000 7fb90cd4f700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9080cbc30 0x7fb908155be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:42.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.223+0000 7fb90cd4f700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb908159440 0x7fb90815c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:42.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.223+0000 7fb90cd4f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9080c8ed0 con 0x7fb9080b2810 2023-10-06T05:07:42.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.223+0000 7fb90cd4f700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb908156120 con 0x7fb9080cbc30 2023-10-06T05:07:42.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.223+0000 7fb90cd4f700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9080c82c0 con 0x7fb908159440 2023-10-06T05:07:42.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.224+0000 7fb9077fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9080b2810 0x7fb9080c7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:42.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.224+0000 7fb9077fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9080b2810 0x7fb9080c7c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60354/0 (socket says 172.21.15.98:60354) 2023-10-06T05:07:42.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.224+0000 7fb9077fe700 1 -- 172.21.15.98:0/238416399 learned_addr learned my addr 172.21.15.98:0/238416399 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:42.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.224+0000 7fb907fff700 1 --2- 172.21.15.98:0/238416399 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb908159440 0x7fb90815c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:42.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.224+0000 7fb906ffd700 1 --2- 172.21.15.98:0/238416399 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9080cbc30 0x7fb908155be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:42.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.224+0000 7fb9077fe700 1 -- 172.21.15.98:0/238416399 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb908159440 msgr2=0x7fb90815c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:42.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.224+0000 7fb9077fe700 1 --2- 172.21.15.98:0/238416399 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb908159440 0x7fb90815c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:42.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.224+0000 7fb9077fe700 1 -- 172.21.15.98:0/238416399 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9080cbc30 msgr2=0x7fb908155be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:42.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.224+0000 7fb9077fe700 1 --2- 172.21.15.98:0/238416399 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9080cbc30 0x7fb908155be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:42.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.224+0000 7fb9077fe700 1 -- 172.21.15.98:0/238416399 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb900009a20 con 0x7fb9080b2810 2023-10-06T05:07:42.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.224+0000 7fb9077fe700 1 --2- 172.21.15.98:0/238416399 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9080b2810 0x7fb9080c7c90 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fb8f800c510 tx=0x7fb8f800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:42.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.224+0000 7fb8ef7fe700 1 -- 172.21.15.98:0/238416399 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8f8011590 con 0x7fb9080b2810 2023-10-06T05:07:42.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.224+0000 7fb8ef7fe700 1 -- 172.21.15.98:0/238416399 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8f8011710 con 0x7fb9080b2810 2023-10-06T05:07:42.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.224+0000 7fb90cd4f700 1 -- 172.21.15.98:0/238416399 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9080c85a0 con 0x7fb9080b2810 2023-10-06T05:07:42.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.224+0000 7fb8ef7fe700 1 -- 172.21.15.98:0/238416399 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8f8011ad0 con 0x7fb9080b2810 2023-10-06T05:07:42.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.224+0000 7fb90cd4f700 1 -- 172.21.15.98:0/238416399 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9080c8b60 con 0x7fb9080b2810 2023-10-06T05:07:42.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.225+0000 7fb90cd4f700 1 -- 172.21.15.98:0/238416399 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9080b1770 con 0x7fb9080b2810 2023-10-06T05:07:42.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.226+0000 7fb8ef7fe700 1 -- 172.21.15.98:0/238416399 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fb8f801b3b0 con 0x7fb9080b2810 2023-10-06T05:07:42.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.226+0000 7fb8ef7fe700 1 --2- 172.21.15.98:0/238416399 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fb8f4066f30 0x7fb8f40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:42.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.226+0000 7fb906ffd700 1 --2- 172.21.15.98:0/238416399 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fb8f4066f30 0x7fb8f40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:42.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.226+0000 7fb8ef7fe700 1 -- 172.21.15.98:0/238416399 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6867+0+0 (secure 0 0 0) 0x7fb8f8083f40 con 0x7fb9080b2810 2023-10-06T05:07:42.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.226+0000 7fb906ffd700 1 --2- 172.21.15.98:0/238416399 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fb8f4066f30 0x7fb8f40693e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fb9081576e0 tx=0x7fb8fc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:42.231 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.230+0000 7fb8ef7fe700 1 -- 172.21.15.98:0/238416399 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb8f8050d20 con 0x7fb9080b2810 2023-10-06T05:07:42.455 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.454+0000 7fb90cd4f700 1 -- 172.21.15.98:0/238416399 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb90805f230 con 0x7fb9080b2810 2023-10-06T05:07:42.456 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.455+0000 7fb8ef7fe700 1 -- 172.21.15.98:0/238416399 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4596 (secure 0 0 0) 0x7fb8f8017070 con 0x7fb9080b2810 2023-10-06T05:07:42.456 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:07:42.456 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4285,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6834/925250632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":925250632},{"type":"v1","addr":"172.21.15.174:6834","nonce":925250632}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4288,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/1629449810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6833","nonce":1629449810},{"type":"v1","addr":"172.21.15.174:6835","nonce":1629449810}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4299,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/3961327990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3961327990},{"type":"v1","addr":"172.21.15.98:6836","nonce":3961327990}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-06T05:06:57.086683+0000","modified":"2023-10-06T05:07:09.773654+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.98:6837/2683239094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":2683239094},{"type":"v1","addr":"172.21.15.98:6837","nonce":2683239094}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-06T05:07:42.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.457+0000 7fb90cd4f700 1 -- 172.21.15.98:0/238416399 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fb8f4066f30 msgr2=0x7fb8f40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:42.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.457+0000 7fb90cd4f700 1 --2- 172.21.15.98:0/238416399 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fb8f4066f30 0x7fb8f40693e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fb9081576e0 tx=0x7fb8fc009040 comp rx=0 tx=0).stop 2023-10-06T05:07:42.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.457+0000 7fb90cd4f700 1 -- 172.21.15.98:0/238416399 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9080b2810 msgr2=0x7fb9080c7c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:42.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.457+0000 7fb90cd4f700 1 --2- 172.21.15.98:0/238416399 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9080b2810 0x7fb9080c7c90 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fb8f800c510 tx=0x7fb8f800c8d0 comp rx=0 tx=0).stop 2023-10-06T05:07:42.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.457+0000 7fb90cd4f700 1 -- 172.21.15.98:0/238416399 shutdown_connections 2023-10-06T05:07:42.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.457+0000 7fb90cd4f700 1 --2- 172.21.15.98:0/238416399 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb908159440 0x7fb90815c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:42.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.457+0000 7fb90cd4f700 1 --2- 172.21.15.98:0/238416399 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fb8f4066f30 0x7fb8f40693e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:42.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.457+0000 7fb90cd4f700 1 --2- 172.21.15.98:0/238416399 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9080b2810 0x7fb9080c7c90 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:42.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.457+0000 7fb90cd4f700 1 --2- 172.21.15.98:0/238416399 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9080cbc30 0x7fb908155be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:42.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.457+0000 7fb90cd4f700 1 -- 172.21.15.98:0/238416399 >> 172.21.15.98:0/238416399 conn(0x7fb90806ece0 msgr2=0x7fb9080ce000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:42.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.457+0000 7fb90cd4f700 1 -- 172.21.15.98:0/238416399 shutdown_connections 2023-10-06T05:07:42.459 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.457+0000 7fb90cd4f700 1 -- 172.21.15.98:0/238416399 wait complete. 2023-10-06T05:07:42.459 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 8 2023-10-06T05:07:42.470 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T05:07:42.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.583+0000 7fe486581700 1 Processor -- start 2023-10-06T05:07:42.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.583+0000 7fe486581700 1 -- start start 2023-10-06T05:07:42.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.584+0000 7fe486581700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe480159ad0 0x7fe480159f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:42.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.584+0000 7fe486581700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe480068d10 con 0x7fe4800ca670 2023-10-06T05:07:42.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.584+0000 7fe486581700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe48015a670 con 0x7fe4800cb5f0 2023-10-06T05:07:42.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.584+0000 7fe486581700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe480154020 con 0x7fe480159ad0 2023-10-06T05:07:42.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.584+0000 7fe484b1e700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fe4800ca670 0x7fe4800ce190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:40430/0 (socket says 172.21.15.98:40430) 2023-10-06T05:07:42.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.584+0000 7fe484b1e700 1 -- 172.21.15.98:0/838059296 learned_addr learned my addr 172.21.15.98:0/838059296 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:07:42.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.585+0000 7fe47f7fe700 1 --2- 172.21.15.98:0/838059296 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe480159ad0 0x7fe480159f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:42.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.585+0000 7fe47effd700 1 -- 172.21.15.98:0/838059296 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 97520534 0 0) 0x7fe480068d10 con 0x7fe4800ca670 2023-10-06T05:07:42.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.585+0000 7fe47effd700 1 -- 172.21.15.98:0/838059296 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe468003570 con 0x7fe4800ca670 2023-10-06T05:07:42.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.585+0000 7fe47effd700 1 -- 172.21.15.98:0/838059296 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2704002128 0 0) 0x7fe48015a670 con 0x7fe4800cb5f0 2023-10-06T05:07:42.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.585+0000 7fe47effd700 1 -- 172.21.15.98:0/838059296 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe480068d10 con 0x7fe4800cb5f0 2023-10-06T05:07:42.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.585+0000 7fe47effd700 1 -- 172.21.15.98:0/838059296 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 114299763 0 0) 0x7fe468003570 con 0x7fe4800ca670 2023-10-06T05:07:42.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.585+0000 7fe47f7fe700 1 -- 172.21.15.98:0/838059296 >> v1:172.21.15.174:6789/0 conn(0x7fe4800cb5f0 legacy=0x7fe4800691a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:42.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.585+0000 7fe47f7fe700 1 -- 172.21.15.98:0/838059296 >> v1:172.21.15.98:6789/0 conn(0x7fe4800ca670 legacy=0x7fe4800ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:42.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.585+0000 7fe47f7fe700 1 -- 172.21.15.98:0/838059296 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe48015a950 con 0x7fe480159ad0 2023-10-06T05:07:42.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.585+0000 7fe47f7fe700 1 --2- 172.21.15.98:0/838059296 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe480159ad0 0x7fe480159f20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe474009910 tx=0x7fe4740091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=903603c7d38ecf48 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:42.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.585+0000 7fe47effd700 1 -- 172.21.15.98:0/838059296 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4780017f0 con 0x7fe480159ad0 2023-10-06T05:07:42.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.585+0000 7fe47effd700 1 -- 172.21.15.98:0/838059296 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe474011420 con 0x7fe480159ad0 2023-10-06T05:07:42.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.585+0000 7fe47effd700 1 -- 172.21.15.98:0/838059296 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4740117c0 con 0x7fe480159ad0 2023-10-06T05:07:42.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.586+0000 7fe486581700 1 -- 172.21.15.98:0/838059296 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe480159ad0 msgr2=0x7fe480159f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:42.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.586+0000 7fe486581700 1 --2- 172.21.15.98:0/838059296 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe480159ad0 0x7fe480159f20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe474009910 tx=0x7fe4740091a0 comp rx=0 tx=0).stop 2023-10-06T05:07:42.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.586+0000 7fe486581700 1 -- 172.21.15.98:0/838059296 shutdown_connections 2023-10-06T05:07:42.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.586+0000 7fe486581700 1 --2- 172.21.15.98:0/838059296 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe480159ad0 0x7fe480159f20 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:42.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.586+0000 7fe486581700 1 -- 172.21.15.98:0/838059296 >> 172.21.15.98:0/838059296 conn(0x7fe4800707d0 msgr2=0x7fe480070bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:42.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.586+0000 7fe486581700 1 -- 172.21.15.98:0/838059296 shutdown_connections 2023-10-06T05:07:42.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.586+0000 7fe486581700 1 -- 172.21.15.98:0/838059296 wait complete. 2023-10-06T05:07:42.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.587+0000 7fe486581700 1 Processor -- start 2023-10-06T05:07:42.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.587+0000 7fe486581700 1 -- start start 2023-10-06T05:07:42.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.588+0000 7fe486581700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe4800ca670 0x7fe4801552a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:42.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.588+0000 7fe486581700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4800cb5f0 0x7fe4800b34d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:42.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.588+0000 7fe486581700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe480159ad0 0x7fe4800b1550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:42.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.588+0000 7fe486581700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4800c2920 con 0x7fe4800cb5f0 2023-10-06T05:07:42.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.588+0000 7fe486581700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4801557e0 con 0x7fe4800ca670 2023-10-06T05:07:42.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.588+0000 7fe486581700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4800c1d30 con 0x7fe480159ad0 2023-10-06T05:07:42.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.588+0000 7fe47f7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4800cb5f0 0x7fe4800b34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:42.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.588+0000 7fe47f7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4800cb5f0 0x7fe4800b34d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60364/0 (socket says 172.21.15.98:60364) 2023-10-06T05:07:42.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.588+0000 7fe47f7fe700 1 -- 172.21.15.98:0/3424369078 learned_addr learned my addr 172.21.15.98:0/3424369078 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:42.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.588+0000 7fe484b1e700 1 --2- 172.21.15.98:0/3424369078 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe480159ad0 0x7fe4800b1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:42.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.588+0000 7fe47ffff700 1 --2- 172.21.15.98:0/3424369078 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe4800ca670 0x7fe4801552a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:42.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.588+0000 7fe47f7fe700 1 -- 172.21.15.98:0/3424369078 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe480159ad0 msgr2=0x7fe4800b1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:42.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.588+0000 7fe47f7fe700 1 --2- 172.21.15.98:0/3424369078 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe480159ad0 0x7fe4800b1550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:42.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.588+0000 7fe47f7fe700 1 -- 172.21.15.98:0/3424369078 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe4800ca670 msgr2=0x7fe4801552a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:42.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.588+0000 7fe47f7fe700 1 --2- 172.21.15.98:0/3424369078 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe4800ca670 0x7fe4801552a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:42.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.588+0000 7fe47f7fe700 1 -- 172.21.15.98:0/3424369078 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe474009420 con 0x7fe4800cb5f0 2023-10-06T05:07:42.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.588+0000 7fe47f7fe700 1 --2- 172.21.15.98:0/3424369078 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4800cb5f0 0x7fe4800b34d0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fe4740030c0 tx=0x7fe474016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:42.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.589+0000 7fe467fff700 1 -- 172.21.15.98:0/3424369078 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe47401cc50 con 0x7fe4800cb5f0 2023-10-06T05:07:42.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.589+0000 7fe467fff700 1 -- 172.21.15.98:0/3424369078 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe47401cdb0 con 0x7fe4800cb5f0 2023-10-06T05:07:42.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.589+0000 7fe486581700 1 -- 172.21.15.98:0/3424369078 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4800c1ff0 con 0x7fe4800cb5f0 2023-10-06T05:07:42.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.589+0000 7fe467fff700 1 -- 172.21.15.98:0/3424369078 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe47401a6a0 con 0x7fe4800cb5f0 2023-10-06T05:07:42.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.589+0000 7fe486581700 1 -- 172.21.15.98:0/3424369078 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4800c25b0 con 0x7fe4800cb5f0 2023-10-06T05:07:42.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.590+0000 7fe486581700 1 -- 172.21.15.98:0/3424369078 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe480064a90 con 0x7fe4800cb5f0 2023-10-06T05:07:42.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.590+0000 7fe467fff700 1 -- 172.21.15.98:0/3424369078 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fe47401aad0 con 0x7fe4800cb5f0 2023-10-06T05:07:42.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.590+0000 7fe467fff700 1 --2- 172.21.15.98:0/3424369078 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fe46c066f30 0x7fe46c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:42.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.590+0000 7fe467fff700 1 -- 172.21.15.98:0/3424369078 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6867+0+0 (secure 0 0 0) 0x7fe4740868d0 con 0x7fe4800cb5f0 2023-10-06T05:07:42.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.590+0000 7fe47ffff700 1 --2- 172.21.15.98:0/3424369078 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fe46c066f30 0x7fe46c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:42.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.591+0000 7fe47ffff700 1 --2- 172.21.15.98:0/3424369078 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fe46c066f30 0x7fe46c0693e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fe470003870 tx=0x7fe470009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:42.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.594+0000 7fe467fff700 1 -- 172.21.15.98:0/3424369078 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe4740536b0 con 0x7fe4800cb5f0 2023-10-06T05:07:42.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.780+0000 7fe486581700 1 -- 172.21.15.98:0/3424369078 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe4800b1330 con 0x7fe4800cb5f0 2023-10-06T05:07:42.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.781+0000 7fe467fff700 1 -- 172.21.15.98:0/3424369078 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v31) v1 ==== 74+0+18801 (secure 0 0 0) 0x7fe474056cd0 con 0x7fe4800cb5f0 2023-10-06T05:07:42.782 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:07:42.782 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":31,"fsid":"f50c5df5-cad5-47a4-9bb1-8c791d4311ec","created":"2023-10-06T05:06:32.971175+0000","modified":"2023-10-06T05:07:39.938013+0000","last_up_change":"2023-10-06T05:06:44.016489+0000","last_in_change":"2023-10-06T05:06:35.544511+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-06T05:06:44.559671+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-06T05:06:51.432161+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-10-06T05:06:55.589057+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-10-06T05:06:56.041687+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-10-06T05:06:59.754099+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d9495d95-da75-4c3b-8927-e9bae6a8b093","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":90627},{"type":"v1","addr":"172.21.15.98:6801","nonce":90627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":90627},{"type":"v1","addr":"172.21.15.98:6803","nonce":90627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":90627},{"type":"v1","addr":"172.21.15.98:6807","nonce":90627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":90627},{"type":"v1","addr":"172.21.15.98:6805","nonce":90627}]},"public_addr":"172.21.15.98:6801/90627","cluster_addr":"172.21.15.98:6803/90627","heartbeat_back_addr":"172.21.15.98:6807/90627","heartbeat_front_addr":"172.21.15.98:6805/90627","state":["exists","up"]},{"osd":1,"uuid":"f25ba5be-d65e-4707-a134-154092deb754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":90628},{"type":"v1","addr":"172.21.15.98:6809","nonce":90628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":90628},{"type":"v1","addr":"172.21.15.98:6811","nonce":90628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":90628},{"type":"v1","addr":"172.21.15.98:6815","nonce":90628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":90628},{"type":"v1","addr":"172.21.15.98:6813","nonce":90628}]},"public_addr":"172.21.15.98:6809/90628","cluster_addr":"172.21.15.98:6811/90628","heartbeat_back_addr":"172.21.15.98:6815/90628","heartbeat_front_addr":"172.21.15.98:6813/90628","state":["exists","up"]},{"osd":2,"uuid":"5ab9e028-945c-43e2-8c95-3b0b4cf38e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":90662},{"type":"v1","addr":"172.21.15.98:6818","nonce":90662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":90662},{"type":"v1","addr":"172.21.15.98:6822","nonce":90662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":90662},{"type":"v1","addr":"172.21.15.98:6830","nonce":90662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":90662},{"type":"v1","addr":"172.21.15.98:6826","nonce":90662}]},"public_addr":"172.21.15.98:6818/90662","cluster_addr":"172.21.15.98:6822/90662","heartbeat_back_addr":"172.21.15.98:6830/90662","heartbeat_front_addr":"172.21.15.98:6826/90662","state":["exists","up"]},{"osd":3,"uuid":"3bb803ff-1738-46bf-be62-2267bf378822","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6817","nonce":90663},{"type":"v1","addr":"172.21.15.98:6819","nonce":90663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6821","nonce":90663},{"type":"v1","addr":"172.21.15.98:6823","nonce":90663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6829","nonce":90663},{"type":"v1","addr":"172.21.15.98:6831","nonce":90663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6825","nonce":90663},{"type":"v1","addr":"172.21.15.98:6827","nonce":90663}]},"public_addr":"172.21.15.98:6819/90663","cluster_addr":"172.21.15.98:6823/90663","heartbeat_back_addr":"172.21.15.98:6831/90663","heartbeat_front_addr":"172.21.15.98:6827/90663","state":["exists","up"]},{"osd":4,"uuid":"7d57a4c9-6462-43f6-b64a-7f9e9b4353d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":90847},{"type":"v1","addr":"172.21.15.174:6801","nonce":90847}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":90847},{"type":"v1","addr":"172.21.15.174:6803","nonce":90847}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":90847},{"type":"v1","addr":"172.21.15.174:6807","nonce":90847}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":90847},{"type":"v1","addr":"172.21.15.174:6805","nonce":90847}]},"public_addr":"172.21.15.174:6801/90847","cluster_addr":"172.21.15.174:6803/90847","heartbeat_back_addr":"172.21.15.174:6807/90847","heartbeat_front_addr":"172.21.15.174:6805/90847","state":["exists","up"]},{"osd":5,"uuid":"83287ea4-e44d-4af9-bc77-36d47c60d4a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":90848},{"type":"v1","addr":"172.21.15.174:6809","nonce":90848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":90848},{"type":"v1","addr":"172.21.15.174:6811","nonce":90848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":90848},{"type":"v1","addr":"172.21.15.174:6815","nonce":90848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":90848},{"type":"v1","addr":"172.21.15.174:6813","nonce":90848}]},"public_addr":"172.21.15.174:6809/90848","cluster_addr":"172.21.15.174:6811/90848","heartbeat_back_addr":"172.21.15.174:6815/90848","heartbeat_front_addr":"172.21.15.174:6813/90848","state":["exists","up"]},{"osd":6,"uuid":"496a6272-892e-43c4-a145-206623c22f99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":90877},{"type":"v1","addr":"172.21.15.174:6817","nonce":90877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":90877},{"type":"v1","addr":"172.21.15.174:6819","nonce":90877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":90877},{"type":"v1","addr":"172.21.15.174:6823","nonce":90877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":90877},{"type":"v1","addr":"172.21.15.174:6821","nonce":90877}]},"public_addr":"172.21.15.174:6817/90877","cluster_addr":"172.21.15.174:6819/90877","heartbeat_back_addr":"172.21.15.174:6823/90877","heartbeat_front_addr":"172.21.15.174:6821/90877","state":["exists","up"]},{"osd":7,"uuid":"49967e34-8b07-4bab-a13e-0c16d6a0f3a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":90878},{"type":"v1","addr":"172.21.15.174:6825","nonce":90878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":90878},{"type":"v1","addr":"172.21.15.174:6827","nonce":90878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":90878},{"type":"v1","addr":"172.21.15.174:6831","nonce":90878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":90878},{"type":"v1","addr":"172.21.15.174:6829","nonce":90878}]},"public_addr":"172.21.15.174:6825/90878","cluster_addr":"172.21.15.174:6827/90878","heartbeat_back_addr":"172.21.15.174:6831/90878","heartbeat_front_addr":"172.21.15.174:6829/90878","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.446479+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.514266+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.794108+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.810894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.731919+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.717756+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.737435+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.508682+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.9","mappings":[{"from":1,"to":5}]},{"pgid":"4.c","mappings":[{"from":3,"to":2}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T05:07:42.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.783+0000 7fe486581700 1 -- 172.21.15.98:0/3424369078 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fe46c066f30 msgr2=0x7fe46c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:42.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.783+0000 7fe486581700 1 --2- 172.21.15.98:0/3424369078 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fe46c066f30 0x7fe46c0693e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fe470003870 tx=0x7fe470009040 comp rx=0 tx=0).stop 2023-10-06T05:07:42.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.783+0000 7fe486581700 1 -- 172.21.15.98:0/3424369078 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4800cb5f0 msgr2=0x7fe4800b34d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:42.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.783+0000 7fe486581700 1 --2- 172.21.15.98:0/3424369078 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4800cb5f0 0x7fe4800b34d0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fe4740030c0 tx=0x7fe474016000 comp rx=0 tx=0).stop 2023-10-06T05:07:42.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.783+0000 7fe486581700 1 -- 172.21.15.98:0/3424369078 shutdown_connections 2023-10-06T05:07:42.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.783+0000 7fe486581700 1 --2- 172.21.15.98:0/3424369078 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe480159ad0 0x7fe4800b1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:42.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.783+0000 7fe486581700 1 --2- 172.21.15.98:0/3424369078 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe4800ca670 0x7fe4801552a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:42.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.783+0000 7fe486581700 1 --2- 172.21.15.98:0/3424369078 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fe46c066f30 0x7fe46c0693e0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:42.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.783+0000 7fe486581700 1 --2- 172.21.15.98:0/3424369078 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe4800cb5f0 0x7fe4800b34d0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:42.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.783+0000 7fe486581700 1 -- 172.21.15.98:0/3424369078 >> 172.21.15.98:0/3424369078 conn(0x7fe4800707d0 msgr2=0x7fe480153440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:42.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.783+0000 7fe486581700 1 -- 172.21.15.98:0/3424369078 shutdown_connections 2023-10-06T05:07:42.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.783+0000 7fe486581700 1 -- 172.21.15.98:0/3424369078 wait complete. 2023-10-06T05:07:42.796 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-06T05:07:42.796 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-06T05:07:42.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.909+0000 7f13175b3700 1 Processor -- start 2023-10-06T05:07:42.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.909+0000 7f13175b3700 1 -- start start 2023-10-06T05:07:42.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.910+0000 7f13175b3700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1310159560 0x7f13100aa530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:42.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.910+0000 7f13175b3700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f13100b49a0 0x7f13100aaa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:42.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.910+0000 7f13175b3700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13100b0c60 con 0x7f13100b0760 2023-10-06T05:07:42.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.910+0000 7f13175b3700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13100b0f40 con 0x7f1310159560 2023-10-06T05:07:42.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.910+0000 7f13175b3700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13100b1220 con 0x7f13100b49a0 2023-10-06T05:07:42.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.910+0000 7f131534f700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1310159560 0x7f13100aa530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:42.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.910+0000 7f131534f700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1310159560 0x7f13100aa530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60370/0 (socket says 172.21.15.98:60370) 2023-10-06T05:07:42.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.910+0000 7f131534f700 1 -- 172.21.15.98:0/2231813872 learned_addr learned my addr 172.21.15.98:0/2231813872 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:42.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.910+0000 7f1314b4e700 1 --2- 172.21.15.98:0/2231813872 >> v2:172.21.15.174:3300/0 conn(0x7f13100b49a0 0x7f13100aaa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:42.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.910+0000 7f131534f700 1 -- 172.21.15.98:0/2231813872 >> v2:172.21.15.174:3300/0 conn(0x7f13100b49a0 msgr2=0x7f13100aaa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:42.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.910+0000 7f131534f700 1 --2- 172.21.15.98:0/2231813872 >> v2:172.21.15.174:3300/0 conn(0x7f13100b49a0 0x7f13100aaa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:42.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.910+0000 7f131534f700 1 -- 172.21.15.98:0/2231813872 >> v1:172.21.15.174:6789/0 conn(0x7f13100b0760 legacy=0x7f13100b3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:42.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.910+0000 7f131534f700 1 -- 172.21.15.98:0/2231813872 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13100b1500 con 0x7f1310159560 2023-10-06T05:07:42.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.911+0000 7f131534f700 1 --2- 172.21.15.98:0/2231813872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1310159560 0x7f13100aa530 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f130c009f70 tx=0x7f130c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90fd2d5ff7699e04 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:42.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.911+0000 7f1307fff700 1 -- 172.21.15.98:0/2231813872 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f130c00e070 con 0x7f1310159560 2023-10-06T05:07:42.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.911+0000 7f1307fff700 1 -- 172.21.15.98:0/2231813872 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f130c00ba50 con 0x7f1310159560 2023-10-06T05:07:42.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.911+0000 7f1307fff700 1 -- 172.21.15.98:0/2231813872 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f130c0092e0 con 0x7f1310159560 2023-10-06T05:07:42.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.911+0000 7f13175b3700 1 -- 172.21.15.98:0/2231813872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1310159560 msgr2=0x7f13100aa530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:42.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.911+0000 7f13175b3700 1 --2- 172.21.15.98:0/2231813872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1310159560 0x7f13100aa530 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f130c009f70 tx=0x7f130c009840 comp rx=0 tx=0).stop 2023-10-06T05:07:42.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.911+0000 7f13175b3700 1 -- 172.21.15.98:0/2231813872 shutdown_connections 2023-10-06T05:07:42.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.911+0000 7f13175b3700 1 --2- 172.21.15.98:0/2231813872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1310159560 0x7f13100aa530 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:42.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.911+0000 7f13175b3700 1 --2- 172.21.15.98:0/2231813872 >> v2:172.21.15.174:3300/0 conn(0x7f13100b49a0 0x7f13100aaa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:42.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.911+0000 7f13175b3700 1 -- 172.21.15.98:0/2231813872 >> 172.21.15.98:0/2231813872 conn(0x7f13100997f0 msgr2=0x7f131006f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:42.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.911+0000 7f13175b3700 1 -- 172.21.15.98:0/2231813872 shutdown_connections 2023-10-06T05:07:42.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.911+0000 7f13175b3700 1 -- 172.21.15.98:0/2231813872 wait complete. 2023-10-06T05:07:42.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.913+0000 7f13175b3700 1 Processor -- start 2023-10-06T05:07:42.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.913+0000 7f13175b3700 1 -- start start 2023-10-06T05:07:42.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.914+0000 7f13175b3700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f13100b0760 0x7f13100cbad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:42.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.914+0000 7f13175b3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13100b49a0 0x7f131016cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:42.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.914+0000 7f13175b3700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1310159560 0x7f131016d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:42.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.914+0000 7f13175b3700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1310169850 con 0x7f13100b49a0 2023-10-06T05:07:42.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.914+0000 7f13175b3700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1310157260 con 0x7f13100b0760 2023-10-06T05:07:42.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.914+0000 7f13175b3700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f131016a5c0 con 0x7f1310159560 2023-10-06T05:07:42.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.914+0000 7f1314b4e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13100b49a0 0x7f131016cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:42.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.914+0000 7f1314b4e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13100b49a0 0x7f131016cae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60386/0 (socket says 172.21.15.98:60386) 2023-10-06T05:07:42.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.914+0000 7f1314b4e700 1 -- 172.21.15.98:0/2691720733 learned_addr learned my addr 172.21.15.98:0/2691720733 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:42.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.914+0000 7f1314b4e700 1 -- 172.21.15.98:0/2691720733 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1310159560 msgr2=0x7f131016d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:42.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.914+0000 7f1315b50700 1 --2- 172.21.15.98:0/2691720733 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1310159560 0x7f131016d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:42.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.914+0000 7f1314b4e700 1 --2- 172.21.15.98:0/2691720733 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1310159560 0x7f131016d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:42.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.914+0000 7f1314b4e700 1 -- 172.21.15.98:0/2691720733 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f13100b0760 msgr2=0x7f13100cbad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:42.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.914+0000 7f131534f700 1 --2- 172.21.15.98:0/2691720733 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f13100b0760 0x7f13100cbad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:42.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.914+0000 7f1314b4e700 1 --2- 172.21.15.98:0/2691720733 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f13100b0760 0x7f13100cbad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:42.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.914+0000 7f1314b4e700 1 -- 172.21.15.98:0/2691720733 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f130c009ae0 con 0x7f13100b49a0 2023-10-06T05:07:42.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.914+0000 7f1314b4e700 1 --2- 172.21.15.98:0/2691720733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13100b49a0 0x7f131016cae0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f130000c880 tx=0x7f1300008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:42.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.914+0000 7f13057fa700 1 -- 172.21.15.98:0/2691720733 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1300011600 con 0x7f13100b49a0 2023-10-06T05:07:42.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.914+0000 7f13057fa700 1 -- 172.21.15.98:0/2691720733 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1300011760 con 0x7f13100b49a0 2023-10-06T05:07:42.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.914+0000 7f13175b3700 1 -- 172.21.15.98:0/2691720733 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f131016d6b0 con 0x7f13100b49a0 2023-10-06T05:07:42.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.914+0000 7f13057fa700 1 -- 172.21.15.98:0/2691720733 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1300011b20 con 0x7f13100b49a0 2023-10-06T05:07:42.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.914+0000 7f13175b3700 1 -- 172.21.15.98:0/2691720733 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f131009a6b0 con 0x7f13100b49a0 2023-10-06T05:07:42.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.915+0000 7f13175b3700 1 -- 172.21.15.98:0/2691720733 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1310064a90 con 0x7f13100b49a0 2023-10-06T05:07:42.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.916+0000 7f13057fa700 1 -- 172.21.15.98:0/2691720733 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f13000193f0 con 0x7f13100b49a0 2023-10-06T05:07:42.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.916+0000 7f13057fa700 1 --2- 172.21.15.98:0/2691720733 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f12f4066f30 0x7f12f40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:42.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.916+0000 7f13057fa700 1 -- 172.21.15.98:0/2691720733 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6867+0+0 (secure 0 0 0) 0x7f1300084d80 con 0x7f13100b49a0 2023-10-06T05:07:42.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.916+0000 7f131534f700 1 --2- 172.21.15.98:0/2691720733 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f12f4066f30 0x7f12f40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:42.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.916+0000 7f131534f700 1 --2- 172.21.15.98:0/2691720733 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f12f4066f30 0x7f12f40693e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f130c009ab0 tx=0x7f130c017000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:42.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:42.920+0000 7f13057fa700 1 -- 172.21.15.98:0/2691720733 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f13000555f0 con 0x7f13100b49a0 2023-10-06T05:07:43.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.149+0000 7f13175b3700 1 -- 172.21.15.98:0/2691720733 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f13100b17e0 con 0x7f13100b49a0 2023-10-06T05:07:43.151 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.150+0000 7f13057fa700 1 -- 172.21.15.98:0/2691720733 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4596 (secure 0 0 0) 0x7f1300055180 con 0x7f13100b49a0 2023-10-06T05:07:43.151 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:07:43.151 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4285,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6834/925250632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":925250632},{"type":"v1","addr":"172.21.15.174:6834","nonce":925250632}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4288,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/1629449810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6833","nonce":1629449810},{"type":"v1","addr":"172.21.15.174:6835","nonce":1629449810}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4299,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/3961327990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3961327990},{"type":"v1","addr":"172.21.15.98:6836","nonce":3961327990}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-06T05:06:57.086683+0000","modified":"2023-10-06T05:07:09.773654+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.98:6837/2683239094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":2683239094},{"type":"v1","addr":"172.21.15.98:6837","nonce":2683239094}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-06T05:07:43.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.151+0000 7f13175b3700 1 -- 172.21.15.98:0/2691720733 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f12f4066f30 msgr2=0x7f12f40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:43.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.151+0000 7f13175b3700 1 --2- 172.21.15.98:0/2691720733 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f12f4066f30 0x7f12f40693e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f130c009ab0 tx=0x7f130c017000 comp rx=0 tx=0).stop 2023-10-06T05:07:43.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.151+0000 7f13175b3700 1 -- 172.21.15.98:0/2691720733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13100b49a0 msgr2=0x7f131016cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:43.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.151+0000 7f13175b3700 1 --2- 172.21.15.98:0/2691720733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13100b49a0 0x7f131016cae0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f130000c880 tx=0x7f1300008040 comp rx=0 tx=0).stop 2023-10-06T05:07:43.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.152+0000 7f13175b3700 1 -- 172.21.15.98:0/2691720733 shutdown_connections 2023-10-06T05:07:43.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.152+0000 7f13175b3700 1 --2- 172.21.15.98:0/2691720733 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1310159560 0x7f131016d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:43.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.152+0000 7f13175b3700 1 --2- 172.21.15.98:0/2691720733 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f13100b0760 0x7f13100cbad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:43.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.152+0000 7f13175b3700 1 --2- 172.21.15.98:0/2691720733 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f12f4066f30 0x7f12f40693e0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:43.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.152+0000 7f13175b3700 1 --2- 172.21.15.98:0/2691720733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13100b49a0 0x7f131016cae0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:43.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.152+0000 7f13175b3700 1 -- 172.21.15.98:0/2691720733 >> 172.21.15.98:0/2691720733 conn(0x7f13100997f0 msgr2=0x7f13100b3ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:43.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.152+0000 7f13175b3700 1 -- 172.21.15.98:0/2691720733 shutdown_connections 2023-10-06T05:07:43.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.152+0000 7f13175b3700 1 -- 172.21.15.98:0/2691720733 wait complete. 2023-10-06T05:07:43.155 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 8 2023-10-06T05:07:43.164 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-06T05:07:43.278 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.277+0000 7f6a2b433700 1 Processor -- start 2023-10-06T05:07:43.278 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.277+0000 7f6a2b433700 1 -- start start 2023-10-06T05:07:43.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.278+0000 7f6a2b433700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6a24159560 0x7f6a240aa530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:43.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.278+0000 7f6a2b433700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f6a240b49a0 0x7f6a240aaa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:43.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.278+0000 7f6a2b433700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a240b0c60 con 0x7f6a240b0760 2023-10-06T05:07:43.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.278+0000 7f6a2b433700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a240b0f40 con 0x7f6a240b49a0 2023-10-06T05:07:43.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.278+0000 7f6a2b433700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a240b1220 con 0x7f6a24159560 2023-10-06T05:07:43.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.279+0000 7f6a291cf700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6a24159560 0x7f6a240aa530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:43.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.279+0000 7f6a289ce700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f6a240b49a0 0x7f6a240aaa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:43.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.279+0000 7f6a291cf700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6a24159560 0x7f6a240aa530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.98:57506/0 (socket says 172.21.15.98:57506) 2023-10-06T05:07:43.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.279+0000 7f6a291cf700 1 -- 172.21.15.98:0/3851796968 learned_addr learned my addr 172.21.15.98:0/3851796968 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:43.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.279+0000 7f6a289ce700 1 -- 172.21.15.98:0/3851796968 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6a24159560 msgr2=0x7f6a240aa530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:43.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.279+0000 7f6a289ce700 1 --2- 172.21.15.98:0/3851796968 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6a24159560 0x7f6a240aa530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:43.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.279+0000 7f6a289ce700 1 -- 172.21.15.98:0/3851796968 >> v1:172.21.15.174:6789/0 conn(0x7f6a240b0760 legacy=0x7f6a240b3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:43.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.279+0000 7f6a289ce700 1 -- 172.21.15.98:0/3851796968 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a240b1500 con 0x7f6a240b49a0 2023-10-06T05:07:43.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.279+0000 7f6a23fff700 1 -- 172.21.15.98:0/3851796968 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 242661903 0 0) 0x7f6a240b0c60 con 0x7f6a240b0760 2023-10-06T05:07:43.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.279+0000 7f6a289ce700 1 --2- 172.21.15.98:0/3851796968 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a240b49a0 0x7f6a240aaa70 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f6a180098e0 tx=0x7f6a180091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=51e6668b13141fc9 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:43.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.279+0000 7f6a23fff700 1 -- 172.21.15.98:0/3851796968 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a1800b640 con 0x7f6a240b49a0 2023-10-06T05:07:43.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.279+0000 7f6a23fff700 1 -- 172.21.15.98:0/3851796968 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a1800b7a0 con 0x7f6a240b49a0 2023-10-06T05:07:43.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.279+0000 7f6a23fff700 1 -- 172.21.15.98:0/3851796968 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a1800bb40 con 0x7f6a240b49a0 2023-10-06T05:07:43.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.280+0000 7f6a2b433700 1 -- 172.21.15.98:0/3851796968 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a240b49a0 msgr2=0x7f6a240aaa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:43.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.280+0000 7f6a2b433700 1 --2- 172.21.15.98:0/3851796968 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a240b49a0 0x7f6a240aaa70 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f6a180098e0 tx=0x7f6a180091a0 comp rx=0 tx=0).stop 2023-10-06T05:07:43.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.280+0000 7f6a2b433700 1 -- 172.21.15.98:0/3851796968 shutdown_connections 2023-10-06T05:07:43.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.280+0000 7f6a2b433700 1 --2- 172.21.15.98:0/3851796968 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6a24159560 0x7f6a240aa530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:43.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.280+0000 7f6a2b433700 1 --2- 172.21.15.98:0/3851796968 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a240b49a0 0x7f6a240aaa70 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:43.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.280+0000 7f6a2b433700 1 -- 172.21.15.98:0/3851796968 >> 172.21.15.98:0/3851796968 conn(0x7f6a240997f0 msgr2=0x7f6a2406f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:43.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.280+0000 7f6a2b433700 1 -- 172.21.15.98:0/3851796968 shutdown_connections 2023-10-06T05:07:43.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.280+0000 7f6a2b433700 1 -- 172.21.15.98:0/3851796968 wait complete. 2023-10-06T05:07:43.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.281+0000 7f6a2b433700 1 Processor -- start 2023-10-06T05:07:43.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.281+0000 7f6a2b433700 1 -- start start 2023-10-06T05:07:43.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.282+0000 7f6a2b433700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6a240b0760 0x7f6a240aecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:43.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.282+0000 7f6a2b433700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a240b49a0 0x7f6a2415bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:43.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.282+0000 7f6a2b433700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a24159560 0x7f6a24169cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:43.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.282+0000 7f6a2b433700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a2416acd0 con 0x7f6a240b49a0 2023-10-06T05:07:43.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.282+0000 7f6a2b433700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a2416a710 con 0x7f6a24159560 2023-10-06T05:07:43.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.282+0000 7f6a2b433700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a2416a9f0 con 0x7f6a240b0760 2023-10-06T05:07:43.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.282+0000 7f6a289ce700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a240b49a0 0x7f6a2415bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:43.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.282+0000 7f6a289ce700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a240b49a0 0x7f6a2415bfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60392/0 (socket says 172.21.15.98:60392) 2023-10-06T05:07:43.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.282+0000 7f6a289ce700 1 -- 172.21.15.98:0/2031104930 learned_addr learned my addr 172.21.15.98:0/2031104930 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:43.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.282+0000 7f6a291cf700 1 --2- 172.21.15.98:0/2031104930 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6a240b0760 0x7f6a240aecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:43.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.282+0000 7f6a299d0700 1 --2- 172.21.15.98:0/2031104930 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a24159560 0x7f6a24169cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:43.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.282+0000 7f6a289ce700 1 -- 172.21.15.98:0/2031104930 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6a240b0760 msgr2=0x7f6a240aecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:43.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.282+0000 7f6a289ce700 1 --2- 172.21.15.98:0/2031104930 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6a240b0760 0x7f6a240aecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:43.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.282+0000 7f6a289ce700 1 -- 172.21.15.98:0/2031104930 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a24159560 msgr2=0x7f6a24169cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:43.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.282+0000 7f6a289ce700 1 --2- 172.21.15.98:0/2031104930 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a24159560 0x7f6a24169cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:43.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.282+0000 7f6a289ce700 1 -- 172.21.15.98:0/2031104930 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a18009420 con 0x7f6a240b49a0 2023-10-06T05:07:43.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.282+0000 7f6a299d0700 1 --2- 172.21.15.98:0/2031104930 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a24159560 0x7f6a24169cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:07:43.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.282+0000 7f6a289ce700 1 --2- 172.21.15.98:0/2031104930 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a240b49a0 0x7f6a2415bfd0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f6a18013000 tx=0x7f6a18017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:43.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.283+0000 7f6a217fa700 1 -- 172.21.15.98:0/2031104930 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a18000d10 con 0x7f6a240b49a0 2023-10-06T05:07:43.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.283+0000 7f6a217fa700 1 -- 172.21.15.98:0/2031104930 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a1800bb20 con 0x7f6a240b49a0 2023-10-06T05:07:43.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.283+0000 7f6a2b433700 1 -- 172.21.15.98:0/2031104930 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a2409a180 con 0x7f6a240b49a0 2023-10-06T05:07:43.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.283+0000 7f6a217fa700 1 -- 172.21.15.98:0/2031104930 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a18011670 con 0x7f6a240b49a0 2023-10-06T05:07:43.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.283+0000 7f6a2b433700 1 -- 172.21.15.98:0/2031104930 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a2409dca0 con 0x7f6a240b49a0 2023-10-06T05:07:43.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.284+0000 7f6a2b433700 1 -- 172.21.15.98:0/2031104930 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a2409de90 con 0x7f6a240b49a0 2023-10-06T05:07:43.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.284+0000 7f6a217fa700 1 -- 172.21.15.98:0/2031104930 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f6a18011a20 con 0x7f6a240b49a0 2023-10-06T05:07:43.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.284+0000 7f6a217fa700 1 --2- 172.21.15.98:0/2031104930 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f6a10066f30 0x7f6a100693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:43.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.285+0000 7f6a291cf700 1 --2- 172.21.15.98:0/2031104930 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f6a10066f30 0x7f6a100693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:43.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.285+0000 7f6a217fa700 1 -- 172.21.15.98:0/2031104930 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6867+0+0 (secure 0 0 0) 0x7f6a18085fa0 con 0x7f6a240b49a0 2023-10-06T05:07:43.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.285+0000 7f6a291cf700 1 --2- 172.21.15.98:0/2031104930 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f6a10066f30 0x7f6a100693e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f6a1400f510 tx=0x7f6a1400b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:43.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.289+0000 7f6a217fa700 1 -- 172.21.15.98:0/2031104930 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6a180b75a0 con 0x7f6a240b49a0 2023-10-06T05:07:43.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.513+0000 7f6a2b433700 1 -- 172.21.15.98:0/2031104930 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6a2409a670 con 0x7f6a240b49a0 2023-10-06T05:07:43.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.514+0000 7f6a217fa700 1 -- 172.21.15.98:0/2031104930 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4596 (secure 0 0 0) 0x7f6a2409de90 con 0x7f6a240b49a0 2023-10-06T05:07:43.515 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:07:43.515 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4285,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6834/925250632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":925250632},{"type":"v1","addr":"172.21.15.174:6834","nonce":925250632}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4288,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/1629449810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6833","nonce":1629449810},{"type":"v1","addr":"172.21.15.174:6835","nonce":1629449810}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4299,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/3961327990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":3961327990},{"type":"v1","addr":"172.21.15.98:6836","nonce":3961327990}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-06T05:06:57.086683+0000","modified":"2023-10-06T05:07:09.773654+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.98:6837/2683239094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":2683239094},{"type":"v1","addr":"172.21.15.98:6837","nonce":2683239094}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-06T05:07:43.517 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.516+0000 7f6a2b433700 1 -- 172.21.15.98:0/2031104930 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f6a10066f30 msgr2=0x7f6a100693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:43.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.516+0000 7f6a2b433700 1 --2- 172.21.15.98:0/2031104930 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f6a10066f30 0x7f6a100693e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f6a1400f510 tx=0x7f6a1400b040 comp rx=0 tx=0).stop 2023-10-06T05:07:43.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.516+0000 7f6a2b433700 1 -- 172.21.15.98:0/2031104930 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a240b49a0 msgr2=0x7f6a2415bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:43.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.516+0000 7f6a2b433700 1 --2- 172.21.15.98:0/2031104930 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a240b49a0 0x7f6a2415bfd0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f6a18013000 tx=0x7f6a18017000 comp rx=0 tx=0).stop 2023-10-06T05:07:43.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.516+0000 7f6a2b433700 1 -- 172.21.15.98:0/2031104930 shutdown_connections 2023-10-06T05:07:43.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.516+0000 7f6a2b433700 1 --2- 172.21.15.98:0/2031104930 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6a24159560 0x7f6a24169cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:43.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.516+0000 7f6a2b433700 1 --2- 172.21.15.98:0/2031104930 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6a240b0760 0x7f6a240aecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:43.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.516+0000 7f6a2b433700 1 --2- 172.21.15.98:0/2031104930 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f6a10066f30 0x7f6a100693e0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:43.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.516+0000 7f6a2b433700 1 --2- 172.21.15.98:0/2031104930 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a240b49a0 0x7f6a2415bfd0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:43.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.516+0000 7f6a2b433700 1 -- 172.21.15.98:0/2031104930 >> 172.21.15.98:0/2031104930 conn(0x7f6a240997f0 msgr2=0x7f6a24158a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:43.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.516+0000 7f6a2b433700 1 -- 172.21.15.98:0/2031104930 shutdown_connections 2023-10-06T05:07:43.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.516+0000 7f6a2b433700 1 -- 172.21.15.98:0/2031104930 wait complete. 2023-10-06T05:07:43.519 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 8 2023-10-06T05:07:43.530 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T05:07:43.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.643+0000 7fd836281700 1 Processor -- start 2023-10-06T05:07:43.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.643+0000 7fd836281700 1 -- start start 2023-10-06T05:07:43.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.644+0000 7fd836281700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fd830153c30 0x7fd8300685c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:43.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.644+0000 7fd836281700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd830156d60 0x7fd830068b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:43.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.644+0000 7fd836281700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd830157620 con 0x7fd8300cbf90 2023-10-06T05:07:43.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.644+0000 7fd836281700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd83006eb50 con 0x7fd830156d60 2023-10-06T05:07:43.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.644+0000 7fd836281700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd830157160 con 0x7fd830153c30 2023-10-06T05:07:43.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.644+0000 7fd835a80700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd830156d60 0x7fd830068b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:43.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.644+0000 7fd835a80700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd830156d60 0x7fd830068b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60406/0 (socket says 172.21.15.98:60406) 2023-10-06T05:07:43.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.644+0000 7fd835a80700 1 -- 172.21.15.98:0/359225395 learned_addr learned my addr 172.21.15.98:0/359225395 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:43.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.644+0000 7fd83527f700 1 --2- 172.21.15.98:0/359225395 >> v2:172.21.15.174:3300/0 conn(0x7fd830153c30 0x7fd8300685c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:43.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.644+0000 7fd835a80700 1 -- 172.21.15.98:0/359225395 >> v2:172.21.15.174:3300/0 conn(0x7fd830153c30 msgr2=0x7fd8300685c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:43.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.644+0000 7fd835a80700 1 --2- 172.21.15.98:0/359225395 >> v2:172.21.15.174:3300/0 conn(0x7fd830153c30 0x7fd8300685c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:43.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.644+0000 7fd835a80700 1 -- 172.21.15.98:0/359225395 >> v1:172.21.15.174:6789/0 conn(0x7fd8300cbf90 legacy=0x7fd830156650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:43.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.644+0000 7fd835a80700 1 -- 172.21.15.98:0/359225395 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd830157340 con 0x7fd830156d60 2023-10-06T05:07:43.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.644+0000 7fd83527f700 1 --2- 172.21.15.98:0/359225395 >> v2:172.21.15.174:3300/0 conn(0x7fd830153c30 0x7fd8300685c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:07:43.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.645+0000 7fd835a80700 1 --2- 172.21.15.98:0/359225395 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd830156d60 0x7fd830068b00 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fd830069150 tx=0x7fd828009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=957b46f0caf2398a server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:43.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.645+0000 7fd82ffff700 1 -- 172.21.15.98:0/359225395 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd82800b820 con 0x7fd830156d60 2023-10-06T05:07:43.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.645+0000 7fd82ffff700 1 -- 172.21.15.98:0/359225395 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd82800b980 con 0x7fd830156d60 2023-10-06T05:07:43.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.645+0000 7fd82ffff700 1 -- 172.21.15.98:0/359225395 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd82800bd20 con 0x7fd830156d60 2023-10-06T05:07:43.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.645+0000 7fd836281700 1 -- 172.21.15.98:0/359225395 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd830156d60 msgr2=0x7fd830068b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:43.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.645+0000 7fd836281700 1 --2- 172.21.15.98:0/359225395 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd830156d60 0x7fd830068b00 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fd830069150 tx=0x7fd828009250 comp rx=0 tx=0).stop 2023-10-06T05:07:43.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.645+0000 7fd836281700 1 -- 172.21.15.98:0/359225395 shutdown_connections 2023-10-06T05:07:43.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.645+0000 7fd836281700 1 --2- 172.21.15.98:0/359225395 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd830156d60 0x7fd830068b00 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:43.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.645+0000 7fd836281700 1 --2- 172.21.15.98:0/359225395 >> v2:172.21.15.174:3300/0 conn(0x7fd830153c30 0x7fd8300685c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:43.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.645+0000 7fd836281700 1 -- 172.21.15.98:0/359225395 >> 172.21.15.98:0/359225395 conn(0x7fd8300c9c10 msgr2=0x7fd83006dac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:43.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.645+0000 7fd836281700 1 -- 172.21.15.98:0/359225395 shutdown_connections 2023-10-06T05:07:43.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.645+0000 7fd836281700 1 -- 172.21.15.98:0/359225395 wait complete. 2023-10-06T05:07:43.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.646+0000 7fd836281700 1 Processor -- start 2023-10-06T05:07:43.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.646+0000 7fd836281700 1 -- start start 2023-10-06T05:07:43.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.647+0000 7fd836281700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8300cbf90 0x7fd830169390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:43.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.647+0000 7fd836281700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd830153c30 0x7fd830163190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:43.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.647+0000 7fd836281700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd830156d60 0x7fd8301550b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:43.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.647+0000 7fd836281700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd830169c50 con 0x7fd8300cbf90 2023-10-06T05:07:43.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.647+0000 7fd836281700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd830169f30 con 0x7fd830153c30 2023-10-06T05:07:43.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.647+0000 7fd836281700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd83016a210 con 0x7fd830156d60 2023-10-06T05:07:43.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.647+0000 7fd83527f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8300cbf90 0x7fd830169390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:43.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.647+0000 7fd83527f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8300cbf90 0x7fd830169390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60422/0 (socket says 172.21.15.98:60422) 2023-10-06T05:07:43.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.647+0000 7fd83527f700 1 -- 172.21.15.98:0/3856840150 learned_addr learned my addr 172.21.15.98:0/3856840150 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:43.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.647+0000 7fd835a80700 1 --2- 172.21.15.98:0/3856840150 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd830156d60 0x7fd8301550b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:43.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.647+0000 7fd834a7e700 1 --2- 172.21.15.98:0/3856840150 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd830153c30 0x7fd830163190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:43.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.648+0000 7fd83527f700 1 -- 172.21.15.98:0/3856840150 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd830156d60 msgr2=0x7fd8301550b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:43.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.648+0000 7fd83527f700 1 --2- 172.21.15.98:0/3856840150 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd830156d60 0x7fd8301550b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:43.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.648+0000 7fd83527f700 1 -- 172.21.15.98:0/3856840150 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd830153c30 msgr2=0x7fd830163190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:43.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.648+0000 7fd83527f700 1 --2- 172.21.15.98:0/3856840150 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd830153c30 0x7fd830163190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:43.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.648+0000 7fd83527f700 1 -- 172.21.15.98:0/3856840150 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8280094f0 con 0x7fd8300cbf90 2023-10-06T05:07:43.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.648+0000 7fd83527f700 1 --2- 172.21.15.98:0/3856840150 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8300cbf90 0x7fd830169390 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fd82000a400 tx=0x7fd820007910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:43.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.648+0000 7fd82d7fa700 1 -- 172.21.15.98:0/3856840150 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd82000d650 con 0x7fd8300cbf90 2023-10-06T05:07:43.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.648+0000 7fd82d7fa700 1 -- 172.21.15.98:0/3856840150 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd82000d7b0 con 0x7fd8300cbf90 2023-10-06T05:07:43.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.648+0000 7fd836281700 1 -- 172.21.15.98:0/3856840150 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8300c4990 con 0x7fd8300cbf90 2023-10-06T05:07:43.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.648+0000 7fd82d7fa700 1 -- 172.21.15.98:0/3856840150 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd82000db50 con 0x7fd8300cbf90 2023-10-06T05:07:43.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.648+0000 7fd836281700 1 -- 172.21.15.98:0/3856840150 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8300c45b0 con 0x7fd8300cbf90 2023-10-06T05:07:43.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.649+0000 7fd836281700 1 -- 172.21.15.98:0/3856840150 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8300ca860 con 0x7fd8300cbf90 2023-10-06T05:07:43.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.649+0000 7fd82d7fa700 1 -- 172.21.15.98:0/3856840150 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fd820015440 con 0x7fd8300cbf90 2023-10-06T05:07:43.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.650+0000 7fd82d7fa700 1 --2- 172.21.15.98:0/3856840150 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fd81c066f30 0x7fd81c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:43.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.650+0000 7fd82d7fa700 1 -- 172.21.15.98:0/3856840150 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6867+0+0 (secure 0 0 0) 0x7fd820085270 con 0x7fd8300cbf90 2023-10-06T05:07:43.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.650+0000 7fd834a7e700 1 --2- 172.21.15.98:0/3856840150 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fd81c066f30 0x7fd81c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:43.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.650+0000 7fd834a7e700 1 --2- 172.21.15.98:0/3856840150 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fd81c066f30 0x7fd81c0693e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fd824006fd0 tx=0x7fd824008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:43.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.654+0000 7fd82d7fa700 1 -- 172.21.15.98:0/3856840150 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd82004e5a0 con 0x7fd8300cbf90 2023-10-06T05:07:43.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.841+0000 7fd836281700 1 -- 172.21.15.98:0/3856840150 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd83005f230 con 0x7fd8300cbf90 2023-10-06T05:07:43.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.842+0000 7fd82d7fa700 1 -- 172.21.15.98:0/3856840150 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v31) v1 ==== 74+0+18801 (secure 0 0 0) 0x7fd820052100 con 0x7fd8300cbf90 2023-10-06T05:07:43.844 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:07:43.844 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":31,"fsid":"f50c5df5-cad5-47a4-9bb1-8c791d4311ec","created":"2023-10-06T05:06:32.971175+0000","modified":"2023-10-06T05:07:39.938013+0000","last_up_change":"2023-10-06T05:06:44.016489+0000","last_in_change":"2023-10-06T05:06:35.544511+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-06T05:06:44.559671+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-06T05:06:51.432161+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-10-06T05:06:55.589057+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-10-06T05:06:56.041687+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-10-06T05:06:59.754099+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d9495d95-da75-4c3b-8927-e9bae6a8b093","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":90627},{"type":"v1","addr":"172.21.15.98:6801","nonce":90627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":90627},{"type":"v1","addr":"172.21.15.98:6803","nonce":90627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":90627},{"type":"v1","addr":"172.21.15.98:6807","nonce":90627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":90627},{"type":"v1","addr":"172.21.15.98:6805","nonce":90627}]},"public_addr":"172.21.15.98:6801/90627","cluster_addr":"172.21.15.98:6803/90627","heartbeat_back_addr":"172.21.15.98:6807/90627","heartbeat_front_addr":"172.21.15.98:6805/90627","state":["exists","up"]},{"osd":1,"uuid":"f25ba5be-d65e-4707-a134-154092deb754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":90628},{"type":"v1","addr":"172.21.15.98:6809","nonce":90628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":90628},{"type":"v1","addr":"172.21.15.98:6811","nonce":90628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":90628},{"type":"v1","addr":"172.21.15.98:6815","nonce":90628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":90628},{"type":"v1","addr":"172.21.15.98:6813","nonce":90628}]},"public_addr":"172.21.15.98:6809/90628","cluster_addr":"172.21.15.98:6811/90628","heartbeat_back_addr":"172.21.15.98:6815/90628","heartbeat_front_addr":"172.21.15.98:6813/90628","state":["exists","up"]},{"osd":2,"uuid":"5ab9e028-945c-43e2-8c95-3b0b4cf38e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":90662},{"type":"v1","addr":"172.21.15.98:6818","nonce":90662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":90662},{"type":"v1","addr":"172.21.15.98:6822","nonce":90662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":90662},{"type":"v1","addr":"172.21.15.98:6830","nonce":90662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":90662},{"type":"v1","addr":"172.21.15.98:6826","nonce":90662}]},"public_addr":"172.21.15.98:6818/90662","cluster_addr":"172.21.15.98:6822/90662","heartbeat_back_addr":"172.21.15.98:6830/90662","heartbeat_front_addr":"172.21.15.98:6826/90662","state":["exists","up"]},{"osd":3,"uuid":"3bb803ff-1738-46bf-be62-2267bf378822","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6817","nonce":90663},{"type":"v1","addr":"172.21.15.98:6819","nonce":90663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6821","nonce":90663},{"type":"v1","addr":"172.21.15.98:6823","nonce":90663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6829","nonce":90663},{"type":"v1","addr":"172.21.15.98:6831","nonce":90663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6825","nonce":90663},{"type":"v1","addr":"172.21.15.98:6827","nonce":90663}]},"public_addr":"172.21.15.98:6819/90663","cluster_addr":"172.21.15.98:6823/90663","heartbeat_back_addr":"172.21.15.98:6831/90663","heartbeat_front_addr":"172.21.15.98:6827/90663","state":["exists","up"]},{"osd":4,"uuid":"7d57a4c9-6462-43f6-b64a-7f9e9b4353d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":90847},{"type":"v1","addr":"172.21.15.174:6801","nonce":90847}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":90847},{"type":"v1","addr":"172.21.15.174:6803","nonce":90847}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":90847},{"type":"v1","addr":"172.21.15.174:6807","nonce":90847}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":90847},{"type":"v1","addr":"172.21.15.174:6805","nonce":90847}]},"public_addr":"172.21.15.174:6801/90847","cluster_addr":"172.21.15.174:6803/90847","heartbeat_back_addr":"172.21.15.174:6807/90847","heartbeat_front_addr":"172.21.15.174:6805/90847","state":["exists","up"]},{"osd":5,"uuid":"83287ea4-e44d-4af9-bc77-36d47c60d4a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":90848},{"type":"v1","addr":"172.21.15.174:6809","nonce":90848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":90848},{"type":"v1","addr":"172.21.15.174:6811","nonce":90848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":90848},{"type":"v1","addr":"172.21.15.174:6815","nonce":90848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":90848},{"type":"v1","addr":"172.21.15.174:6813","nonce":90848}]},"public_addr":"172.21.15.174:6809/90848","cluster_addr":"172.21.15.174:6811/90848","heartbeat_back_addr":"172.21.15.174:6815/90848","heartbeat_front_addr":"172.21.15.174:6813/90848","state":["exists","up"]},{"osd":6,"uuid":"496a6272-892e-43c4-a145-206623c22f99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":90877},{"type":"v1","addr":"172.21.15.174:6817","nonce":90877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":90877},{"type":"v1","addr":"172.21.15.174:6819","nonce":90877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":90877},{"type":"v1","addr":"172.21.15.174:6823","nonce":90877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":90877},{"type":"v1","addr":"172.21.15.174:6821","nonce":90877}]},"public_addr":"172.21.15.174:6817/90877","cluster_addr":"172.21.15.174:6819/90877","heartbeat_back_addr":"172.21.15.174:6823/90877","heartbeat_front_addr":"172.21.15.174:6821/90877","state":["exists","up"]},{"osd":7,"uuid":"49967e34-8b07-4bab-a13e-0c16d6a0f3a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":90878},{"type":"v1","addr":"172.21.15.174:6825","nonce":90878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":90878},{"type":"v1","addr":"172.21.15.174:6827","nonce":90878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":90878},{"type":"v1","addr":"172.21.15.174:6831","nonce":90878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":90878},{"type":"v1","addr":"172.21.15.174:6829","nonce":90878}]},"public_addr":"172.21.15.174:6825/90878","cluster_addr":"172.21.15.174:6827/90878","heartbeat_back_addr":"172.21.15.174:6831/90878","heartbeat_front_addr":"172.21.15.174:6829/90878","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.446479+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.514266+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.794108+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.810894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.731919+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.717756+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.737435+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.508682+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.9","mappings":[{"from":1,"to":5}]},{"pgid":"4.c","mappings":[{"from":3,"to":2}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T05:07:43.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.844+0000 7fd836281700 1 -- 172.21.15.98:0/3856840150 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fd81c066f30 msgr2=0x7fd81c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:43.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.844+0000 7fd836281700 1 --2- 172.21.15.98:0/3856840150 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fd81c066f30 0x7fd81c0693e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fd824006fd0 tx=0x7fd824008040 comp rx=0 tx=0).stop 2023-10-06T05:07:43.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.844+0000 7fd836281700 1 -- 172.21.15.98:0/3856840150 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8300cbf90 msgr2=0x7fd830169390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:43.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.844+0000 7fd836281700 1 --2- 172.21.15.98:0/3856840150 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8300cbf90 0x7fd830169390 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fd82000a400 tx=0x7fd820007910 comp rx=0 tx=0).stop 2023-10-06T05:07:43.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.845+0000 7fd836281700 1 -- 172.21.15.98:0/3856840150 shutdown_connections 2023-10-06T05:07:43.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.845+0000 7fd836281700 1 --2- 172.21.15.98:0/3856840150 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd830156d60 0x7fd8301550b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:43.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.845+0000 7fd836281700 1 --2- 172.21.15.98:0/3856840150 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fd81c066f30 0x7fd81c0693e0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:43.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.845+0000 7fd836281700 1 --2- 172.21.15.98:0/3856840150 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd8300cbf90 0x7fd830169390 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:43.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.845+0000 7fd836281700 1 --2- 172.21.15.98:0/3856840150 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd830153c30 0x7fd830163190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:43.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.845+0000 7fd836281700 1 -- 172.21.15.98:0/3856840150 >> 172.21.15.98:0/3856840150 conn(0x7fd8300c9c10 msgr2=0x7fd830153210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:43.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.845+0000 7fd836281700 1 -- 172.21.15.98:0/3856840150 shutdown_connections 2023-10-06T05:07:43.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.845+0000 7fd836281700 1 -- 172.21.15.98:0/3856840150 wait complete. 2023-10-06T05:07:43.858 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-06T05:07:43.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.972+0000 7f999c969700 1 Processor -- start 2023-10-06T05:07:43.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.972+0000 7f999c969700 1 -- start start 2023-10-06T05:07:43.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.973+0000 7f999c969700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9998159440 0x7f99980b3090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:43.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.973+0000 7f999c969700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f99980b35d0 0x7f99980cbc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:43.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.973+0000 7f999c969700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99980cb7a0 con 0x7f99980cc170 2023-10-06T05:07:43.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.973+0000 7f999c969700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99980b0860 con 0x7f9998159440 2023-10-06T05:07:43.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.973+0000 7f999c969700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99980cb150 con 0x7f99980b35d0 2023-10-06T05:07:43.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.973+0000 7f99977fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9998159440 0x7f99980b3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:43.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.973+0000 7f99977fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9998159440 0x7f99980b3090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60434/0 (socket says 172.21.15.98:60434) 2023-10-06T05:07:43.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.973+0000 7f99977fe700 1 -- 172.21.15.98:0/3403860572 learned_addr learned my addr 172.21.15.98:0/3403860572 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:43.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.973+0000 7f9996ffd700 1 --2- 172.21.15.98:0/3403860572 >> v2:172.21.15.174:3300/0 conn(0x7f99980b35d0 0x7f99980cbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:43.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.973+0000 7f99977fe700 1 -- 172.21.15.98:0/3403860572 >> v2:172.21.15.174:3300/0 conn(0x7f99980b35d0 msgr2=0x7f99980cbc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:43.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.973+0000 7f99977fe700 1 --2- 172.21.15.98:0/3403860572 >> v2:172.21.15.174:3300/0 conn(0x7f99980b35d0 0x7f99980cbc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:43.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.973+0000 7f99977fe700 1 -- 172.21.15.98:0/3403860572 >> v1:172.21.15.174:6789/0 conn(0x7f99980cc170 legacy=0x7f99980b2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:43.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.973+0000 7f99977fe700 1 -- 172.21.15.98:0/3403860572 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99980cb430 con 0x7f9998159440 2023-10-06T05:07:43.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.973+0000 7f9996ffd700 1 --2- 172.21.15.98:0/3403860572 >> v2:172.21.15.174:3300/0 conn(0x7f99980b35d0 0x7f99980cbc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:07:43.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.973+0000 7f99977fe700 1 --2- 172.21.15.98:0/3403860572 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9998159440 0x7f99980b3090 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f99980d0220 tx=0x7f9988009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcccf6d77593763d server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:43.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.974+0000 7f99967fc700 1 -- 172.21.15.98:0/3403860572 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f998800b820 con 0x7f9998159440 2023-10-06T05:07:43.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.974+0000 7f99967fc700 1 -- 172.21.15.98:0/3403860572 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f998800b980 con 0x7f9998159440 2023-10-06T05:07:43.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.974+0000 7f99967fc700 1 -- 172.21.15.98:0/3403860572 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f998800bd20 con 0x7f9998159440 2023-10-06T05:07:43.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.974+0000 7f999c969700 1 -- 172.21.15.98:0/3403860572 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9998159440 msgr2=0x7f99980b3090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:43.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.974+0000 7f999c969700 1 --2- 172.21.15.98:0/3403860572 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9998159440 0x7f99980b3090 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f99980d0220 tx=0x7f9988009250 comp rx=0 tx=0).stop 2023-10-06T05:07:43.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.974+0000 7f999c969700 1 -- 172.21.15.98:0/3403860572 shutdown_connections 2023-10-06T05:07:43.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.974+0000 7f999c969700 1 --2- 172.21.15.98:0/3403860572 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9998159440 0x7f99980b3090 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:43.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.974+0000 7f999c969700 1 --2- 172.21.15.98:0/3403860572 >> v2:172.21.15.174:3300/0 conn(0x7f99980b35d0 0x7f99980cbc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:43.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.974+0000 7f999c969700 1 -- 172.21.15.98:0/3403860572 >> 172.21.15.98:0/3403860572 conn(0x7f999806ece0 msgr2=0x7f999806f0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:43.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.974+0000 7f999c969700 1 -- 172.21.15.98:0/3403860572 shutdown_connections 2023-10-06T05:07:43.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.974+0000 7f999c969700 1 -- 172.21.15.98:0/3403860572 wait complete. 2023-10-06T05:07:43.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.975+0000 7f999c969700 1 Processor -- start 2023-10-06T05:07:43.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.975+0000 7f999c969700 1 -- start start 2023-10-06T05:07:43.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.976+0000 7f999c969700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f99980b35d0 0x7f99980c4920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:43.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.976+0000 7f999c969700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99980cc170 0x7f99980c9830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:43.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.976+0000 7f999c969700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9998159440 0x7f99980c8190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:43.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.976+0000 7f999c969700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f999815a430 con 0x7f99980cc170 2023-10-06T05:07:43.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.976+0000 7f999c969700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99980c8bc0 con 0x7f9998159440 2023-10-06T05:07:43.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.976+0000 7f999c969700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9998159b00 con 0x7f99980b35d0 2023-10-06T05:07:43.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.976+0000 7f9996ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99980cc170 0x7f99980c9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:43.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.976+0000 7f9996ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99980cc170 0x7f99980c9830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60442/0 (socket says 172.21.15.98:60442) 2023-10-06T05:07:43.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.976+0000 7f9996ffd700 1 -- 172.21.15.98:0/3332233159 learned_addr learned my addr 172.21.15.98:0/3332233159 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:43.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.977+0000 7f99977fe700 1 --2- 172.21.15.98:0/3332233159 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f99980b35d0 0x7f99980c4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:43.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.977+0000 7f9997fff700 1 --2- 172.21.15.98:0/3332233159 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9998159440 0x7f99980c8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:43.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.977+0000 7f9996ffd700 1 -- 172.21.15.98:0/3332233159 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f99980b35d0 msgr2=0x7f99980c4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:43.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.977+0000 7f9996ffd700 1 --2- 172.21.15.98:0/3332233159 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f99980b35d0 0x7f99980c4920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:43.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.977+0000 7f9996ffd700 1 -- 172.21.15.98:0/3332233159 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9998159440 msgr2=0x7f99980c8190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:43.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.977+0000 7f9996ffd700 1 --2- 172.21.15.98:0/3332233159 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9998159440 0x7f99980c8190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:43.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.977+0000 7f9996ffd700 1 -- 172.21.15.98:0/3332233159 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99880094f0 con 0x7f99980cc170 2023-10-06T05:07:43.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.977+0000 7f9996ffd700 1 --2- 172.21.15.98:0/3332233159 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99980cc170 0x7f99980c9830 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f998c00c830 tx=0x7f998c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:43.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.977+0000 7f997f7fe700 1 -- 172.21.15.98:0/3332233159 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f998c011600 con 0x7f99980cc170 2023-10-06T05:07:43.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.977+0000 7f997f7fe700 1 -- 172.21.15.98:0/3332233159 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f998c011760 con 0x7f99980cc170 2023-10-06T05:07:43.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.977+0000 7f999c969700 1 -- 172.21.15.98:0/3332233159 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9998159de0 con 0x7f99980cc170 2023-10-06T05:07:43.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.977+0000 7f997f7fe700 1 -- 172.21.15.98:0/3332233159 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f998c011b20 con 0x7f99980cc170 2023-10-06T05:07:43.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.977+0000 7f999c969700 1 -- 172.21.15.98:0/3332233159 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99980c4e60 con 0x7f99980cc170 2023-10-06T05:07:43.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.978+0000 7f999c969700 1 -- 172.21.15.98:0/3332233159 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f999815a5e0 con 0x7f99980cc170 2023-10-06T05:07:43.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.978+0000 7f997f7fe700 1 -- 172.21.15.98:0/3332233159 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f998c00a3e0 con 0x7f99980cc170 2023-10-06T05:07:43.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.978+0000 7f997f7fe700 1 --2- 172.21.15.98:0/3332233159 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f9984066f30 0x7f99840693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:43.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.979+0000 7f997f7fe700 1 -- 172.21.15.98:0/3332233159 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6867+0+0 (secure 0 0 0) 0x7f998c084e50 con 0x7f99980cc170 2023-10-06T05:07:43.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.979+0000 7f99977fe700 1 --2- 172.21.15.98:0/3332233159 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f9984066f30 0x7f99840693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:43.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.979+0000 7f99977fe700 1 --2- 172.21.15.98:0/3332233159 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f9984066f30 0x7f99840693e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f99980d0220 tx=0x7f9988016000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:43.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:43.983+0000 7f997f7fe700 1 -- 172.21.15.98:0/3332233159 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f999815a5e0 con 0x7f99980cc170 2023-10-06T05:07:44.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:44.210+0000 7f999c969700 1 -- 172.21.15.98:0/3332233159 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f999815a570 con 0x7f99980cc170 2023-10-06T05:07:44.212 INFO:tasks.ceph.mon.a.smithi098.stderr:2023-10-06T05:07:44.211+0000 7fd5c817d700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-06T05:07:44.220 INFO:tasks.ceph.mds.c.smithi098.stderr: -3713> 2023-10-06T05:07:02.689+0000 7faba9a8f700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-06T05:07:44.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:44.220+0000 7f997f7fe700 1 -- 172.21.15.98:0/3332233159 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v9) v1 ==== 161+0+0 (secure 0 0 0) 0x7f998c015090 con 0x7f99980cc170 2023-10-06T05:07:44.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:44.222+0000 7f999c969700 1 -- 172.21.15.98:0/3332233159 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f9984066f30 msgr2=0x7f99840693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:44.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:44.222+0000 7f999c969700 1 --2- 172.21.15.98:0/3332233159 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f9984066f30 0x7f99840693e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f99980d0220 tx=0x7f9988016000 comp rx=0 tx=0).stop 2023-10-06T05:07:44.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:44.222+0000 7f999c969700 1 -- 172.21.15.98:0/3332233159 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99980cc170 msgr2=0x7f99980c9830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:44.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:44.222+0000 7f999c969700 1 --2- 172.21.15.98:0/3332233159 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99980cc170 0x7f99980c9830 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f998c00c830 tx=0x7f998c008040 comp rx=0 tx=0).stop 2023-10-06T05:07:44.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:44.222+0000 7f999c969700 1 -- 172.21.15.98:0/3332233159 shutdown_connections 2023-10-06T05:07:44.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:44.222+0000 7f999c969700 1 --2- 172.21.15.98:0/3332233159 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9998159440 0x7f99980c8190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:44.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:44.222+0000 7f999c969700 1 --2- 172.21.15.98:0/3332233159 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f99980b35d0 0x7f99980c4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:44.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:44.222+0000 7f999c969700 1 --2- 172.21.15.98:0/3332233159 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f9984066f30 0x7f99840693e0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:44.225 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:44.222+0000 7f999c969700 1 --2- 172.21.15.98:0/3332233159 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f99980cc170 0x7f99980c9830 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:44.225 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:44.222+0000 7f999c969700 1 -- 172.21.15.98:0/3332233159 >> 172.21.15.98:0/3332233159 conn(0x7f999806ece0 msgr2=0x7f99981585f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:44.225 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:44.222+0000 7f999c969700 1 -- 172.21.15.98:0/3332233159 shutdown_connections 2023-10-06T05:07:44.225 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:44.222+0000 7f999c969700 1 -- 172.21.15.98:0/3332233159 wait complete. 2023-10-06T05:07:44.225 INFO:teuthology.orchestra.run.smithi098.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-06T05:07:44.236 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-06T05:07:44.246 INFO:tasks.ceph.mds.c.smithi098.stdout:starting mds.c at 2023-10-06T05:07:44.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:44.352+0000 7f038ce11700 1 Processor -- start 2023-10-06T05:07:44.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:44.352+0000 7f038ce11700 1 -- start start 2023-10-06T05:07:44.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:44.353+0000 7f038ce11700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f03880b1830 0x7f03881565f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:44.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:44.353+0000 7f038ce11700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f038815a600 con 0x7f03880b07c0 2023-10-06T05:07:44.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:44.353+0000 7f038ce11700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03880b0cd0 con 0x7f0388153970 2023-10-06T05:07:44.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:44.353+0000 7f038ce11700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0388159cd0 con 0x7f03880b1830 2023-10-06T05:07:44.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:44.353+0000 7f0386ffd700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f03880b1830 0x7f03881565f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:44.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:44.353+0000 7f0387fff700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f03880b07c0 0x7f0388156b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:40454/0 (socket says 172.21.15.98:40454) 2023-10-06T05:07:44.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:44.353+0000 7f0387fff700 1 -- 172.21.15.98:0/1445408956 learned_addr learned my addr 172.21.15.98:0/1445408956 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:07:44.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:44.353+0000 7f0386ffd700 1 -- 172.21.15.98:0/1445408956 >> v1:172.21.15.174:6789/0 conn(0x7f0388153970 legacy=0x7f03880b1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:44.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:44.353+0000 7f0386ffd700 1 -- 172.21.15.98:0/1445408956 >> v1:172.21.15.98:6789/0 conn(0x7f03880b07c0 legacy=0x7f0388156b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:44.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:44.353+0000 7f0386ffd700 1 -- 172.21.15.98:0/1445408956 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0388159fb0 con 0x7f03880b1830 2023-10-06T05:07:44.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:44.353+0000 7f03867fc700 1 -- 172.21.15.98:0/1445408956 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1508132563 0 0) 0x7f038815a600 con 0x7f03880b07c0 2023-10-06T05:07:44.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:44.353+0000 7f0386ffd700 1 --2- 172.21.15.98:0/1445408956 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03880b1830 0x7f03881565f0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f037c00bb90 tx=0x7f037c00b430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9240006a06b8be33 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:44.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:44.354+0000 7f03867fc700 1 -- 172.21.15.98:0/1445408956 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f037c00db60 con 0x7f03880b1830 2023-10-06T05:07:44.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:44.354+0000 7f03867fc700 1 -- 172.21.15.98:0/1445408956 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f037c00dcc0 con 0x7f03880b1830 2023-10-06T05:07:44.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:44.354+0000 7f03867fc700 1 -- 172.21.15.98:0/1445408956 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f037c0106a0 con 0x7f03880b1830 2023-10-06T05:07:44.357 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:44.354+0000 7f038ce11700 1 -- 172.21.15.98:0/1445408956 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03880b1830 msgr2=0x7f03881565f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:44.357 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:44.354+0000 7f038ce11700 1 --2- 172.21.15.98:0/1445408956 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03880b1830 0x7f03881565f0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f037c00bb90 tx=0x7f037c00b430 comp rx=0 tx=0).stop 2023-10-06T05:07:44.357 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:44.354+0000 7f038ce11700 1 -- 172.21.15.98:0/1445408956 shutdown_connections 2023-10-06T05:07:44.357 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:44.354+0000 7f038ce11700 1 --2- 172.21.15.98:0/1445408956 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03880b1830 0x7f03881565f0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:44.357 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:44.354+0000 7f038ce11700 1 -- 172.21.15.98:0/1445408956 >> 172.21.15.98:0/1445408956 conn(0x7f0388069140 msgr2=0x7f03880aea70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:44.357 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:44.354+0000 7f038ce11700 1 -- 172.21.15.98:0/1445408956 shutdown_connections 2023-10-06T05:07:44.357 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:44.354+0000 7f038ce11700 1 -- 172.21.15.98:0/1445408956 wait complete. 2023-10-06T05:07:44.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:44.355+0000 7f038ce11700 1 Processor -- start 2023-10-06T05:07:44.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:44.355+0000 7f038ce11700 1 -- start start 2023-10-06T05:07:44.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:44.356+0000 7f038ce11700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f03880b07c0 0x7f038814eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:44.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:44.356+0000 7f038ce11700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03880b1830 0x7f0388147bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:44.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:44.356+0000 7f038ce11700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0388153970 0x7f03880c9ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:44.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:44.356+0000 7f038ce11700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f038814e910 con 0x7f03880b1830 2023-10-06T05:07:44.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:44.356+0000 7f038ce11700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f038814e300 con 0x7f0388153970 2023-10-06T05:07:44.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:44.356+0000 7f038ce11700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0388151aa0 con 0x7f03880b07c0 2023-10-06T05:07:44.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:44.356+0000 7f0386ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03880b1830 0x7f0388147bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:44.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:44.356+0000 7f0386ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03880b1830 0x7f0388147bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60466/0 (socket says 172.21.15.98:60466) 2023-10-06T05:07:44.361 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:44.356+0000 7f0386ffd700 1 -- 172.21.15.98:0/2938476347 learned_addr learned my addr 172.21.15.98:0/2938476347 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:44.361 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:44.356+0000 7f0387fff700 1 --2- 172.21.15.98:0/2938476347 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0388153970 0x7f03880c9ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:44.361 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:44.356+0000 7f03877fe700 1 --2- 172.21.15.98:0/2938476347 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f03880b07c0 0x7f038814eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:44.361 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:44.356+0000 7f0386ffd700 1 -- 172.21.15.98:0/2938476347 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f03880b07c0 msgr2=0x7f038814eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:44.361 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:44.356+0000 7f0386ffd700 1 --2- 172.21.15.98:0/2938476347 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f03880b07c0 0x7f038814eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:44.361 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:44.356+0000 7f0386ffd700 1 -- 172.21.15.98:0/2938476347 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0388153970 msgr2=0x7f03880c9ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:44.361 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:44.356+0000 7f0386ffd700 1 --2- 172.21.15.98:0/2938476347 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0388153970 0x7f03880c9ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:44.361 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:44.356+0000 7f0386ffd700 1 -- 172.21.15.98:0/2938476347 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f037c00b6d0 con 0x7f03880b1830 2023-10-06T05:07:44.361 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:44.356+0000 7f03877fe700 1 --2- 172.21.15.98:0/2938476347 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f03880b07c0 0x7f038814eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:07:44.361 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:44.357+0000 7f0386ffd700 1 --2- 172.21.15.98:0/2938476347 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03880b1830 0x7f0388147bd0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f037c00b6a0 tx=0x7f037c01a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:44.361 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:44.357+0000 7f036f7fe700 1 -- 172.21.15.98:0/2938476347 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f037c018970 con 0x7f03880b1830 2023-10-06T05:07:44.362 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:44.357+0000 7f036f7fe700 1 -- 172.21.15.98:0/2938476347 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f037c01d050 con 0x7f03880b1830 2023-10-06T05:07:44.362 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:44.357+0000 7f038ce11700 1 -- 172.21.15.98:0/2938476347 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f038814e5a0 con 0x7f03880b1830 2023-10-06T05:07:44.362 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:44.357+0000 7f036f7fe700 1 -- 172.21.15.98:0/2938476347 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f037c018d40 con 0x7f03880b1830 2023-10-06T05:07:44.362 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:44.357+0000 7f038ce11700 1 -- 172.21.15.98:0/2938476347 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03880ca220 con 0x7f03880b1830 2023-10-06T05:07:44.362 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:44.358+0000 7f038ce11700 1 -- 172.21.15.98:0/2938476347 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f038814e730 con 0x7f03880b1830 2023-10-06T05:07:44.362 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:44.358+0000 7f036f7fe700 1 -- 172.21.15.98:0/2938476347 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f037c023680 con 0x7f03880b1830 2023-10-06T05:07:44.362 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:44.358+0000 7f036f7fe700 1 --2- 172.21.15.98:0/2938476347 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f0374066f30 0x7f03740693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:44.362 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:44.359+0000 7f036f7fe700 1 -- 172.21.15.98:0/2938476347 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6953+0+0 (secure 0 0 0) 0x7f037c014070 con 0x7f03880b1830 2023-10-06T05:07:44.362 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:44.359+0000 7f03877fe700 1 --2- 172.21.15.98:0/2938476347 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f0374066f30 0x7f03740693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:44.362 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:44.359+0000 7f03877fe700 1 --2- 172.21.15.98:0/2938476347 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f0374066f30 0x7f03740693e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f037800c330 tx=0x7f03780074d0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:44.363 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:44.363+0000 7f036f7fe700 1 -- 172.21.15.98:0/2938476347 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f037c056240 con 0x7f03880b1830 2023-10-06T05:07:44.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:44.589+0000 7f038ce11700 1 -- 172.21.15.98:0/2938476347 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f038805f230 con 0x7f03880b1830 2023-10-06T05:07:45.225 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:45.224+0000 7f036f7fe700 1 -- 172.21.15.98:0/2938476347 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v10) v1 ==== 104+0+0 (secure 0 0 0) 0x7f037c059860 con 0x7f03880b1830 2023-10-06T05:07:45.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:45.227+0000 7f038ce11700 1 -- 172.21.15.98:0/2938476347 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f0374066f30 msgr2=0x7f03740693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:45.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:45.227+0000 7f038ce11700 1 --2- 172.21.15.98:0/2938476347 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f0374066f30 0x7f03740693e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f037800c330 tx=0x7f03780074d0 comp rx=0 tx=0).stop 2023-10-06T05:07:45.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:45.227+0000 7f038ce11700 1 -- 172.21.15.98:0/2938476347 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03880b1830 msgr2=0x7f0388147bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:45.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:45.227+0000 7f038ce11700 1 --2- 172.21.15.98:0/2938476347 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03880b1830 0x7f0388147bd0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f037c00b6a0 tx=0x7f037c01a000 comp rx=0 tx=0).stop 2023-10-06T05:07:45.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:45.227+0000 7f038ce11700 1 -- 172.21.15.98:0/2938476347 shutdown_connections 2023-10-06T05:07:45.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:45.227+0000 7f038ce11700 1 --2- 172.21.15.98:0/2938476347 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0388153970 0x7f03880c9ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:45.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:45.227+0000 7f038ce11700 1 --2- 172.21.15.98:0/2938476347 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f03880b07c0 0x7f038814eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:45.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:45.227+0000 7f038ce11700 1 --2- 172.21.15.98:0/2938476347 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f0374066f30 0x7f03740693e0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:45.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:45.227+0000 7f038ce11700 1 --2- 172.21.15.98:0/2938476347 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03880b1830 0x7f0388147bd0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:45.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:45.227+0000 7f038ce11700 1 -- 172.21.15.98:0/2938476347 >> 172.21.15.98:0/2938476347 conn(0x7f0388069140 msgr2=0x7f03880ae8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:45.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:45.227+0000 7f038ce11700 1 -- 172.21.15.98:0/2938476347 shutdown_connections 2023-10-06T05:07:45.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:45.227+0000 7f038ce11700 1 -- 172.21.15.98:0/2938476347 wait complete. 2023-10-06T05:07:45.241 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-06T05:07:45.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:45.398+0000 7f5ef2840700 1 Processor -- start 2023-10-06T05:07:45.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:45.398+0000 7f5ef2840700 1 -- start start 2023-10-06T05:07:45.401 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:45.399+0000 7f5ef2840700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5eec153990 0x7f5eec15a370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:45.401 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:45.399+0000 7f5ef2840700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f5eec0cca70 0x7f5eec0d02b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:45.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:45.399+0000 7f5ef2840700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5eec0cdbf0 con 0x7f5eec0d07f0 2023-10-06T05:07:45.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:45.399+0000 7f5ef2840700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5eec0cb4c0 con 0x7f5eec153990 2023-10-06T05:07:45.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:45.399+0000 7f5ef2840700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5eec15a8b0 con 0x7f5eec0cca70 2023-10-06T05:07:45.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:45.399+0000 7f5ef183e700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5eec153990 0x7f5eec15a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:45.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:45.399+0000 7f5ef203f700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f5eec0d07f0 0x7f5eec15cf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:40466/0 (socket says 172.21.15.98:40466) 2023-10-06T05:07:45.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:45.399+0000 7f5ef203f700 1 -- 172.21.15.98:0/3826312618 learned_addr learned my addr 172.21.15.98:0/3826312618 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:07:45.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:45.399+0000 7f5ef103d700 1 --2- 172.21.15.98:0/3826312618 >> v2:172.21.15.174:3300/0 conn(0x7f5eec0cca70 0x7f5eec0d02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:45.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:45.399+0000 7f5ef183e700 1 -- 172.21.15.98:0/3826312618 >> v2:172.21.15.174:3300/0 conn(0x7f5eec0cca70 msgr2=0x7f5eec0d02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:45.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:45.399+0000 7f5ef183e700 1 --2- 172.21.15.98:0/3826312618 >> v2:172.21.15.174:3300/0 conn(0x7f5eec0cca70 0x7f5eec0d02b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:45.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:45.399+0000 7f5ef183e700 1 -- 172.21.15.98:0/3826312618 >> v1:172.21.15.98:6789/0 conn(0x7f5eec0d07f0 legacy=0x7f5eec15cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:45.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:45.399+0000 7f5ef183e700 1 -- 172.21.15.98:0/3826312618 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eec0d0bf0 con 0x7f5eec153990 2023-10-06T05:07:45.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:45.399+0000 7f5ef083c700 1 -- 172.21.15.98:0/3826312618 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1390719176 0 0) 0x7f5eec0cdbf0 con 0x7f5eec0d07f0 2023-10-06T05:07:45.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:45.399+0000 7f5ef183e700 1 --2- 172.21.15.98:0/3826312618 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5eec153990 0x7f5eec15a370 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f5eec0d0250 tx=0x7f5ee4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a618b94029f40410 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:45.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:45.399+0000 7f5ef083c700 1 -- 172.21.15.98:0/3826312618 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee400e070 con 0x7f5eec153990 2023-10-06T05:07:45.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:45.400+0000 7f5ef083c700 1 -- 172.21.15.98:0/3826312618 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ee400bc30 con 0x7f5eec153990 2023-10-06T05:07:45.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:45.400+0000 7f5ef083c700 1 -- 172.21.15.98:0/3826312618 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee4012430 con 0x7f5eec153990 2023-10-06T05:07:45.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:45.400+0000 7f5ef2840700 1 -- 172.21.15.98:0/3826312618 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5eec153990 msgr2=0x7f5eec15a370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:45.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:45.400+0000 7f5ef2840700 1 --2- 172.21.15.98:0/3826312618 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5eec153990 0x7f5eec15a370 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f5eec0d0250 tx=0x7f5ee4009780 comp rx=0 tx=0).stop 2023-10-06T05:07:45.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:45.400+0000 7f5ef2840700 1 -- 172.21.15.98:0/3826312618 shutdown_connections 2023-10-06T05:07:45.404 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:45.400+0000 7f5ef2840700 1 --2- 172.21.15.98:0/3826312618 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5eec153990 0x7f5eec15a370 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:45.404 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:45.400+0000 7f5ef2840700 1 --2- 172.21.15.98:0/3826312618 >> v2:172.21.15.174:3300/0 conn(0x7f5eec0cca70 0x7f5eec0d02b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:45.404 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:45.400+0000 7f5ef2840700 1 -- 172.21.15.98:0/3826312618 >> 172.21.15.98:0/3826312618 conn(0x7f5eec0680c0 msgr2=0x7f5eec068490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:45.404 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:45.400+0000 7f5ef2840700 1 -- 172.21.15.98:0/3826312618 shutdown_connections 2023-10-06T05:07:45.404 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:45.400+0000 7f5ef2840700 1 -- 172.21.15.98:0/3826312618 wait complete. 2023-10-06T05:07:45.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:45.401+0000 7f5ef2840700 1 Processor -- start 2023-10-06T05:07:45.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:45.401+0000 7f5ef2840700 1 -- start start 2023-10-06T05:07:45.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:45.402+0000 7f5ef2840700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5eec0cca70 0x7f5eec151550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:45.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:45.402+0000 7f5ef2840700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5eec0d07f0 0x7f5eec0c7db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:45.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:45.402+0000 7f5ef2840700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5eec153990 0x7f5eec0b0f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:45.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:45.402+0000 7f5ef2840700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eec16b2b0 con 0x7f5eec0cca70 2023-10-06T05:07:45.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:45.402+0000 7f5ef2840700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eec151a90 con 0x7f5eec153990 2023-10-06T05:07:45.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:45.402+0000 7f5ef2840700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5eec16a9c0 con 0x7f5eec0d07f0 2023-10-06T05:07:45.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:45.402+0000 7f5ef183e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5eec0cca70 0x7f5eec151550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:45.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:45.402+0000 7f5ef183e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5eec0cca70 0x7f5eec151550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60484/0 (socket says 172.21.15.98:60484) 2023-10-06T05:07:45.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:45.402+0000 7f5ef183e700 1 -- 172.21.15.98:0/1117528603 learned_addr learned my addr 172.21.15.98:0/1117528603 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:45.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:45.402+0000 7f5ef183e700 1 -- 172.21.15.98:0/1117528603 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5eec0d07f0 msgr2=0x7f5eec0c7db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:45.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:45.402+0000 7f5ef183e700 1 --2- 172.21.15.98:0/1117528603 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5eec0d07f0 0x7f5eec0c7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:45.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:45.402+0000 7f5ef183e700 1 -- 172.21.15.98:0/1117528603 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5eec153990 msgr2=0x7f5eec0b0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:45.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:45.402+0000 7f5ef183e700 1 --2- 172.21.15.98:0/1117528603 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5eec153990 0x7f5eec0b0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:45.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:45.402+0000 7f5ef183e700 1 -- 172.21.15.98:0/1117528603 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ee4009a20 con 0x7f5eec0cca70 2023-10-06T05:07:45.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:45.402+0000 7f5ef183e700 1 --2- 172.21.15.98:0/1117528603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5eec0cca70 0x7f5eec151550 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f5ee40099f0 tx=0x7f5ee4019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:45.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:45.403+0000 7f5ee9ffb700 1 -- 172.21.15.98:0/1117528603 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee4027470 con 0x7f5eec0cca70 2023-10-06T05:07:45.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:45.403+0000 7f5ee9ffb700 1 -- 172.21.15.98:0/1117528603 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ee4027610 con 0x7f5eec0cca70 2023-10-06T05:07:45.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:45.403+0000 7f5ef2840700 1 -- 172.21.15.98:0/1117528603 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5eec0c8720 con 0x7f5eec0cca70 2023-10-06T05:07:45.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:45.403+0000 7f5ee9ffb700 1 -- 172.21.15.98:0/1117528603 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee40279d0 con 0x7f5eec0cca70 2023-10-06T05:07:45.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:45.403+0000 7f5ef2840700 1 -- 172.21.15.98:0/1117528603 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5eec16af40 con 0x7f5eec0cca70 2023-10-06T05:07:45.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:45.404+0000 7f5ef2840700 1 -- 172.21.15.98:0/1117528603 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5eec05f230 con 0x7f5eec0cca70 2023-10-06T05:07:45.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:45.404+0000 7f5ee9ffb700 1 -- 172.21.15.98:0/1117528603 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f5ee4024070 con 0x7f5eec0cca70 2023-10-06T05:07:45.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:45.404+0000 7f5ee9ffb700 1 --2- 172.21.15.98:0/1117528603 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f5ed8066f30 0x7f5ed80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:45.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:45.405+0000 7f5ef103d700 1 --2- 172.21.15.98:0/1117528603 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f5ed8066f30 0x7f5ed80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:45.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:45.405+0000 7f5ee9ffb700 1 -- 172.21.15.98:0/1117528603 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6953+0+0 (secure 0 0 0) 0x7f5ee4087070 con 0x7f5eec0cca70 2023-10-06T05:07:45.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:45.405+0000 7f5ef103d700 1 --2- 172.21.15.98:0/1117528603 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f5ed8066f30 0x7f5ed80693e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f5ee000b910 tx=0x7f5ee0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:45.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:45.409+0000 7f5ee9ffb700 1 -- 172.21.15.98:0/1117528603 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5ee4053e00 con 0x7f5eec0cca70 2023-10-06T05:07:45.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:45.592+0000 7f5ef2840700 1 -- 172.21.15.98:0/1117528603 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f5eec0ccfe0 con 0x7f5eec0cca70 2023-10-06T05:07:46.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:46.228+0000 7f5ee9ffb700 1 -- 172.21.15.98:0/1117528603 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v33) v1 ==== 181+0+0 (secure 0 0 0) 0x7f5ee402c0d0 con 0x7f5eec0cca70 2023-10-06T05:07:46.233 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:46.231+0000 7f5ef2840700 1 -- 172.21.15.98:0/1117528603 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f5ed8066f30 msgr2=0x7f5ed80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:46.233 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:46.231+0000 7f5ef2840700 1 --2- 172.21.15.98:0/1117528603 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f5ed8066f30 0x7f5ed80693e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f5ee000b910 tx=0x7f5ee0008040 comp rx=0 tx=0).stop 2023-10-06T05:07:46.233 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:46.231+0000 7f5ef2840700 1 -- 172.21.15.98:0/1117528603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5eec0cca70 msgr2=0x7f5eec151550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:46.233 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:46.231+0000 7f5ef2840700 1 --2- 172.21.15.98:0/1117528603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5eec0cca70 0x7f5eec151550 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f5ee40099f0 tx=0x7f5ee4019000 comp rx=0 tx=0).stop 2023-10-06T05:07:46.233 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:46.231+0000 7f5ef2840700 1 -- 172.21.15.98:0/1117528603 shutdown_connections 2023-10-06T05:07:46.233 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:46.231+0000 7f5ef2840700 1 --2- 172.21.15.98:0/1117528603 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5eec153990 0x7f5eec0b0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:46.233 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:46.231+0000 7f5ef2840700 1 --2- 172.21.15.98:0/1117528603 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f5ed8066f30 0x7f5ed80693e0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:46.233 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:46.231+0000 7f5ef2840700 1 --2- 172.21.15.98:0/1117528603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5eec0cca70 0x7f5eec151550 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:46.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:46.231+0000 7f5ef2840700 1 --2- 172.21.15.98:0/1117528603 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5eec0d07f0 0x7f5eec0c7db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:46.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:46.231+0000 7f5ef2840700 1 -- 172.21.15.98:0/1117528603 >> 172.21.15.98:0/1117528603 conn(0x7f5eec0680c0 msgr2=0x7f5eec152fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:46.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:46.232+0000 7f5ef2840700 1 -- 172.21.15.98:0/1117528603 shutdown_connections 2023-10-06T05:07:46.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:46.232+0000 7f5ef2840700 1 -- 172.21.15.98:0/1117528603 wait complete. 2023-10-06T05:07:46.234 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'cephfs_metadata' removed 2023-10-06T05:07:46.245 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-06T05:07:46.365 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:46.363+0000 7f836cc47700 1 Processor -- start 2023-10-06T05:07:46.365 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:46.363+0000 7f836cc47700 1 -- start start 2023-10-06T05:07:46.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:46.364+0000 7f836cc47700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f83680b0720 0x7f83680b10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:46.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:46.364+0000 7f836cc47700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f83680c3090 0x7f83680c45c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:46.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:46.364+0000 7f836cc47700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83680c4130 con 0x7f83680c4b00 2023-10-06T05:07:46.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:46.364+0000 7f836cc47700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83680c3520 con 0x7f83680b0720 2023-10-06T05:07:46.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:46.364+0000 7f836cc47700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83680c3800 con 0x7f83680c3090 2023-10-06T05:07:46.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:46.365+0000 7f836659c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f83680b0720 0x7f83680b10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:46.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:46.365+0000 7f836659c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f83680b0720 0x7f83680b10b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60496/0 (socket says 172.21.15.98:60496) 2023-10-06T05:07:46.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:46.365+0000 7f836659c700 1 -- 172.21.15.98:0/3917872773 learned_addr learned my addr 172.21.15.98:0/3917872773 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:46.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:46.365+0000 7f8365d9b700 1 --2- 172.21.15.98:0/3917872773 >> v2:172.21.15.174:3300/0 conn(0x7f83680c3090 0x7f83680c45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:46.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:46.365+0000 7f836659c700 1 -- 172.21.15.98:0/3917872773 >> v2:172.21.15.174:3300/0 conn(0x7f83680c3090 msgr2=0x7f83680c45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:46.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:46.365+0000 7f836659c700 1 --2- 172.21.15.98:0/3917872773 >> v2:172.21.15.174:3300/0 conn(0x7f83680c3090 0x7f83680c45c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:46.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:46.365+0000 7f836659c700 1 -- 172.21.15.98:0/3917872773 >> v1:172.21.15.174:6789/0 conn(0x7f83680c4b00 legacy=0x7f83680c8ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:46.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:46.365+0000 7f836659c700 1 -- 172.21.15.98:0/3917872773 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83680c3ae0 con 0x7f83680b0720 2023-10-06T05:07:46.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:46.365+0000 7f8365d9b700 1 --2- 172.21.15.98:0/3917872773 >> v2:172.21.15.174:3300/0 conn(0x7f83680c3090 0x7f83680c45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:07:46.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:46.365+0000 7f836659c700 1 --2- 172.21.15.98:0/3917872773 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83680b0720 0x7f83680b10b0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f836806fc30 tx=0x7f8358009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f00e616ac02122b server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:46.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:46.365+0000 7f836559a700 1 -- 172.21.15.98:0/3917872773 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f835800e070 con 0x7f83680b0720 2023-10-06T05:07:46.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:46.365+0000 7f836559a700 1 -- 172.21.15.98:0/3917872773 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f835800baf0 con 0x7f83680b0720 2023-10-06T05:07:46.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:46.365+0000 7f836559a700 1 -- 172.21.15.98:0/3917872773 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83580092e0 con 0x7f83680b0720 2023-10-06T05:07:46.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:46.366+0000 7f836cc47700 1 -- 172.21.15.98:0/3917872773 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83680b0720 msgr2=0x7f83680b10b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:46.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:46.366+0000 7f836cc47700 1 --2- 172.21.15.98:0/3917872773 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83680b0720 0x7f83680b10b0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f836806fc30 tx=0x7f8358009840 comp rx=0 tx=0).stop 2023-10-06T05:07:46.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:46.366+0000 7f836cc47700 1 -- 172.21.15.98:0/3917872773 shutdown_connections 2023-10-06T05:07:46.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:46.366+0000 7f836cc47700 1 --2- 172.21.15.98:0/3917872773 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83680b0720 0x7f83680b10b0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:46.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:46.366+0000 7f836cc47700 1 --2- 172.21.15.98:0/3917872773 >> v2:172.21.15.174:3300/0 conn(0x7f83680c3090 0x7f83680c45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:46.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:46.366+0000 7f836cc47700 1 -- 172.21.15.98:0/3917872773 >> 172.21.15.98:0/3917872773 conn(0x7f8368087fc0 msgr2=0x7f83680ce2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:46.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:46.366+0000 7f836cc47700 1 -- 172.21.15.98:0/3917872773 shutdown_connections 2023-10-06T05:07:46.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:46.366+0000 7f836cc47700 1 -- 172.21.15.98:0/3917872773 wait complete. 2023-10-06T05:07:46.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:46.367+0000 7f836cc47700 1 Processor -- start 2023-10-06T05:07:46.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:46.367+0000 7f836cc47700 1 -- start start 2023-10-06T05:07:46.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:46.368+0000 7f836cc47700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83680b0720 0x7f8368152cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:46.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:46.368+0000 7f836cc47700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f83680c3090 0x7f836815a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:46.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:46.368+0000 7f836cc47700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f83680c4b00 0x7f83680c55b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:46.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:46.368+0000 7f836cc47700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8368153e10 con 0x7f83680b0720 2023-10-06T05:07:46.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:46.368+0000 7f836cc47700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83680b2130 con 0x7f83680c4b00 2023-10-06T05:07:46.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:46.368+0000 7f836cc47700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8368153570 con 0x7f83680c3090 2023-10-06T05:07:46.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:46.368+0000 7f836659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83680b0720 0x7f8368152cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:46.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:46.368+0000 7f836659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83680b0720 0x7f8368152cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60502/0 (socket says 172.21.15.98:60502) 2023-10-06T05:07:46.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:46.368+0000 7f836659c700 1 -- 172.21.15.98:0/2025345854 learned_addr learned my addr 172.21.15.98:0/2025345854 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:46.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:46.368+0000 7f8366d9d700 1 --2- 172.21.15.98:0/2025345854 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f83680c4b00 0x7f83680c55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:46.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:46.368+0000 7f8365d9b700 1 --2- 172.21.15.98:0/2025345854 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f83680c3090 0x7f836815a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:46.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:46.368+0000 7f836659c700 1 -- 172.21.15.98:0/2025345854 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f83680c3090 msgr2=0x7f836815a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:46.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:46.368+0000 7f836659c700 1 --2- 172.21.15.98:0/2025345854 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f83680c3090 0x7f836815a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:46.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:46.368+0000 7f836659c700 1 -- 172.21.15.98:0/2025345854 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f83680c4b00 msgr2=0x7f83680c55b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:46.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:46.368+0000 7f836659c700 1 --2- 172.21.15.98:0/2025345854 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f83680c4b00 0x7f83680c55b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:46.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:46.368+0000 7f836659c700 1 -- 172.21.15.98:0/2025345854 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8358009ae0 con 0x7f83680b0720 2023-10-06T05:07:46.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:46.368+0000 7f8365d9b700 1 --2- 172.21.15.98:0/2025345854 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f83680c3090 0x7f836815a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:07:46.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:46.368+0000 7f836659c700 1 --2- 172.21.15.98:0/2025345854 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83680b0720 0x7f8368152cc0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f835800b770 tx=0x7f8358017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:46.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:46.368+0000 7f83567fc700 1 -- 172.21.15.98:0/2025345854 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8358012aa0 con 0x7f83680b0720 2023-10-06T05:07:46.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:46.368+0000 7f83567fc700 1 -- 172.21.15.98:0/2025345854 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8358012c20 con 0x7f83680b0720 2023-10-06T05:07:46.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:46.368+0000 7f836cc47700 1 -- 172.21.15.98:0/2025345854 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8368153850 con 0x7f83680b0720 2023-10-06T05:07:46.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:46.368+0000 7f83567fc700 1 -- 172.21.15.98:0/2025345854 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f835801d420 con 0x7f83680b0720 2023-10-06T05:07:46.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:46.368+0000 7f836cc47700 1 -- 172.21.15.98:0/2025345854 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8368153200 con 0x7f83680b0720 2023-10-06T05:07:46.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:46.369+0000 7f836cc47700 1 -- 172.21.15.98:0/2025345854 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8368064a90 con 0x7f83680b0720 2023-10-06T05:07:46.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:46.370+0000 7f83567fc700 1 -- 172.21.15.98:0/2025345854 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f835801d830 con 0x7f83680b0720 2023-10-06T05:07:46.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:46.370+0000 7f83567fc700 1 --2- 172.21.15.98:0/2025345854 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f8348066f30 0x7f83480693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:46.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:46.370+0000 7f83567fc700 1 -- 172.21.15.98:0/2025345854 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6515+0+0 (secure 0 0 0) 0x7f8358085fb0 con 0x7f83680b0720 2023-10-06T05:07:46.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:46.370+0000 7f8365d9b700 1 --2- 172.21.15.98:0/2025345854 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f8348066f30 0x7f83480693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:46.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:46.370+0000 7f8365d9b700 1 --2- 172.21.15.98:0/2025345854 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f8348066f30 0x7f83480693e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f836806c140 tx=0x7f835c006f90 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:46.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:46.374+0000 7f83567fc700 1 -- 172.21.15.98:0/2025345854 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f83580b75a0 con 0x7f83680b0720 2023-10-06T05:07:46.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:46.558+0000 7f836cc47700 1 -- 172.21.15.98:0/2025345854 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f83680c3dc0 con 0x7f83680b0720 2023-10-06T05:07:47.237 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.236+0000 7f83567fc700 1 -- 172.21.15.98:0/2025345854 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v34) v1 ==== 169+0+0 (secure 0 0 0) 0x7f8358052ef0 con 0x7f83680b0720 2023-10-06T05:07:47.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.238+0000 7f836cc47700 1 -- 172.21.15.98:0/2025345854 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f8348066f30 msgr2=0x7f83480693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:47.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.238+0000 7f836cc47700 1 --2- 172.21.15.98:0/2025345854 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f8348066f30 0x7f83480693e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f836806c140 tx=0x7f835c006f90 comp rx=0 tx=0).stop 2023-10-06T05:07:47.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.238+0000 7f836cc47700 1 -- 172.21.15.98:0/2025345854 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83680b0720 msgr2=0x7f8368152cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:47.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.238+0000 7f836cc47700 1 --2- 172.21.15.98:0/2025345854 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83680b0720 0x7f8368152cc0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f835800b770 tx=0x7f8358017000 comp rx=0 tx=0).stop 2023-10-06T05:07:47.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.238+0000 7f836cc47700 1 -- 172.21.15.98:0/2025345854 shutdown_connections 2023-10-06T05:07:47.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.238+0000 7f836cc47700 1 --2- 172.21.15.98:0/2025345854 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f83680c4b00 0x7f83680c55b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:47.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.238+0000 7f836cc47700 1 --2- 172.21.15.98:0/2025345854 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f8348066f30 0x7f83480693e0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:47.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.238+0000 7f836cc47700 1 --2- 172.21.15.98:0/2025345854 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83680b0720 0x7f8368152cc0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:47.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.238+0000 7f836cc47700 1 --2- 172.21.15.98:0/2025345854 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f83680c3090 0x7f836815a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:47.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.238+0000 7f836cc47700 1 -- 172.21.15.98:0/2025345854 >> 172.21.15.98:0/2025345854 conn(0x7f8368087fc0 msgr2=0x7f8368155980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:47.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.238+0000 7f836cc47700 1 -- 172.21.15.98:0/2025345854 shutdown_connections 2023-10-06T05:07:47.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.238+0000 7f836cc47700 1 -- 172.21.15.98:0/2025345854 wait complete. 2023-10-06T05:07:47.241 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'cephfs_data' removed 2023-10-06T05:07:47.252 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2023-10-06T05:07:47.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.373+0000 7f8a5a3e4700 1 Processor -- start 2023-10-06T05:07:47.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.373+0000 7f8a5a3e4700 1 -- start start 2023-10-06T05:07:47.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.374+0000 7f8a5a3e4700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8a540b2360 0x7f8a540b08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:47.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.374+0000 7f8a5a3e4700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a540b27d0 con 0x7f8a5406ece0 2023-10-06T05:07:47.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.374+0000 7f8a5a3e4700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a540b2ce0 con 0x7f8a540aede0 2023-10-06T05:07:47.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.374+0000 7f8a5a3e4700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5406fb40 con 0x7f8a540b2360 2023-10-06T05:07:47.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.374+0000 7f8a53fff700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f8a5406ece0 0x7f8a5406f0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:40476/0 (socket says 172.21.15.98:40476) 2023-10-06T05:07:47.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.374+0000 7f8a53fff700 1 -- 172.21.15.98:0/1426009505 learned_addr learned my addr 172.21.15.98:0/1426009505 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:07:47.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.374+0000 7f8a537fe700 1 --2- 172.21.15.98:0/1426009505 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8a540b2360 0x7f8a540b08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:47.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.374+0000 7f8a52ffd700 1 -- 172.21.15.98:0/1426009505 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3305200210 0 0) 0x7f8a540b27d0 con 0x7f8a5406ece0 2023-10-06T05:07:47.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.374+0000 7f8a52ffd700 1 -- 172.21.15.98:0/1426009505 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a3c003570 con 0x7f8a5406ece0 2023-10-06T05:07:47.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.374+0000 7f8a52ffd700 1 -- 172.21.15.98:0/1426009505 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 871120364 0 0) 0x7f8a3c003570 con 0x7f8a5406ece0 2023-10-06T05:07:47.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.374+0000 7f8a52ffd700 1 -- 172.21.15.98:0/1426009505 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8a540b2360 msgr2=0x7f8a540b08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:47.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.374+0000 7f8a52ffd700 1 --2- 172.21.15.98:0/1426009505 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8a540b2360 0x7f8a540b08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:47.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.374+0000 7f8a52ffd700 1 -- 172.21.15.98:0/1426009505 >> v1:172.21.15.174:6789/0 conn(0x7f8a540aede0 legacy=0x7f8a540af240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:47.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.374+0000 7f8a52ffd700 1 -- 172.21.15.98:0/1426009505 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a540b0410 con 0x7f8a5406ece0 2023-10-06T05:07:47.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.375+0000 7f8a52ffd700 1 -- 172.21.15.98:0/1426009505 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1543991876 0 0) 0x7f8a44001470 con 0x7f8a5406ece0 2023-10-06T05:07:47.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.375+0000 7f8a52ffd700 1 -- 172.21.15.98:0/1426009505 >> v1:172.21.15.98:6789/0 conn(0x7f8a5406ece0 legacy=0x7f8a5406f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:47.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.375+0000 7f8a52ffd700 1 --2- 172.21.15.98:0/1426009505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a3c003a90 0x7f8a3c003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:47.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.375+0000 7f8a52ffd700 1 --2- 172.21.15.98:0/1426009505 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8a540b2360 0x7f8a3c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:47.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.375+0000 7f8a52ffd700 1 --2- 172.21.15.98:0/1426009505 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a3c004d50 0x7f8a3c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:47.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.375+0000 7f8a52ffd700 1 -- 172.21.15.98:0/1426009505 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a3c00bb10 con 0x7f8a3c003a90 2023-10-06T05:07:47.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.375+0000 7f8a52ffd700 1 -- 172.21.15.98:0/1426009505 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a3c00bc80 con 0x7f8a3c004d50 2023-10-06T05:07:47.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.375+0000 7f8a52ffd700 1 -- 172.21.15.98:0/1426009505 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a3c00be40 con 0x7f8a540b2360 2023-10-06T05:07:47.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.375+0000 7f8a53fff700 1 --2- 172.21.15.98:0/1426009505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a3c003a90 0x7f8a3c003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:47.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.375+0000 7f8a537fe700 1 --2- 172.21.15.98:0/1426009505 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8a540b2360 0x7f8a3c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:47.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.375+0000 7f8a58981700 1 --2- 172.21.15.98:0/1426009505 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a3c004d50 0x7f8a3c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:47.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.375+0000 7f8a53fff700 1 -- 172.21.15.98:0/1426009505 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8a540b2360 msgr2=0x7f8a3c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:47.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.375+0000 7f8a53fff700 1 --2- 172.21.15.98:0/1426009505 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8a540b2360 0x7f8a3c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:47.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.375+0000 7f8a53fff700 1 -- 172.21.15.98:0/1426009505 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a3c004d50 msgr2=0x7f8a3c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:47.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.375+0000 7f8a53fff700 1 --2- 172.21.15.98:0/1426009505 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a3c004d50 0x7f8a3c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:47.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.375+0000 7f8a53fff700 1 -- 172.21.15.98:0/1426009505 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a3c003570 con 0x7f8a3c003a90 2023-10-06T05:07:47.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.375+0000 7f8a58981700 1 --2- 172.21.15.98:0/1426009505 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a3c004d50 0x7f8a3c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:07:47.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.375+0000 7f8a53fff700 1 --2- 172.21.15.98:0/1426009505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a3c003a90 0x7f8a3c003e60 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f8a4400c2d0 tx=0x7f8a44001df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18fb03652b4c471e server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:47.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.375+0000 7f8a52ffd700 1 -- 172.21.15.98:0/1426009505 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a4400cbc0 con 0x7f8a3c003a90 2023-10-06T05:07:47.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.375+0000 7f8a52ffd700 1 -- 172.21.15.98:0/1426009505 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a4400cd20 con 0x7f8a3c003a90 2023-10-06T05:07:47.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.375+0000 7f8a52ffd700 1 -- 172.21.15.98:0/1426009505 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a44005b70 con 0x7f8a3c003a90 2023-10-06T05:07:47.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.376+0000 7f8a5a3e4700 1 -- 172.21.15.98:0/1426009505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a3c003a90 msgr2=0x7f8a3c003e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:47.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.376+0000 7f8a5a3e4700 1 --2- 172.21.15.98:0/1426009505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a3c003a90 0x7f8a3c003e60 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f8a4400c2d0 tx=0x7f8a44001df0 comp rx=0 tx=0).stop 2023-10-06T05:07:47.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.376+0000 7f8a58981700 1 -- 172.21.15.98:0/1426009505 reap_dead start 2023-10-06T05:07:47.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.376+0000 7f8a5a3e4700 1 -- 172.21.15.98:0/1426009505 shutdown_connections 2023-10-06T05:07:47.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.376+0000 7f8a5a3e4700 1 -- 172.21.15.98:0/1426009505 >> 172.21.15.98:0/1426009505 conn(0x7f8a540682f0 msgr2=0x7f8a540686f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:47.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.376+0000 7f8a5a3e4700 1 -- 172.21.15.98:0/1426009505 shutdown_connections 2023-10-06T05:07:47.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.376+0000 7f8a5a3e4700 1 -- 172.21.15.98:0/1426009505 wait complete. 2023-10-06T05:07:47.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.377+0000 7f8a5a3e4700 1 Processor -- start 2023-10-06T05:07:47.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.377+0000 7f8a5a3e4700 1 -- start start 2023-10-06T05:07:47.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.378+0000 7f8a5a3e4700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a3c003a90 0x7f8a54158050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:47.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.378+0000 7f8a5a3e4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a54158590 0x7f8a540c9b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:47.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.378+0000 7f8a5a3e4700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8a540ca090 0x7f8a54159cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:47.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.378+0000 7f8a5a3e4700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a540c96c0 con 0x7f8a54158590 2023-10-06T05:07:47.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.378+0000 7f8a5a3e4700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a540c8ab0 con 0x7f8a3c003a90 2023-10-06T05:07:47.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.378+0000 7f8a5a3e4700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a540c8d90 con 0x7f8a540ca090 2023-10-06T05:07:47.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.378+0000 7f8a537fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a54158590 0x7f8a540c9b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:47.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.378+0000 7f8a537fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a54158590 0x7f8a540c9b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60518/0 (socket says 172.21.15.98:60518) 2023-10-06T05:07:47.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.378+0000 7f8a537fe700 1 -- 172.21.15.98:0/4128862933 learned_addr learned my addr 172.21.15.98:0/4128862933 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:47.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.378+0000 7f8a53fff700 1 --2- 172.21.15.98:0/4128862933 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a3c003a90 0x7f8a54158050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:47.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.378+0000 7f8a58981700 1 --2- 172.21.15.98:0/4128862933 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8a540ca090 0x7f8a54159cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:47.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.378+0000 7f8a537fe700 1 -- 172.21.15.98:0/4128862933 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8a540ca090 msgr2=0x7f8a54159cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:47.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.378+0000 7f8a537fe700 1 --2- 172.21.15.98:0/4128862933 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8a540ca090 0x7f8a54159cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:47.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.378+0000 7f8a537fe700 1 -- 172.21.15.98:0/4128862933 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a3c003a90 msgr2=0x7f8a54158050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:47.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.378+0000 7f8a537fe700 1 --2- 172.21.15.98:0/4128862933 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a3c003a90 0x7f8a54158050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:47.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.378+0000 7f8a537fe700 1 -- 172.21.15.98:0/4128862933 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a540c9070 con 0x7f8a54158590 2023-10-06T05:07:47.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.378+0000 7f8a537fe700 1 --2- 172.21.15.98:0/4128862933 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a54158590 0x7f8a540c9b50 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f8a48011d40 tx=0x7f8a480091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:47.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.378+0000 7f8a3bfff700 1 -- 172.21.15.98:0/4128862933 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a4800fa10 con 0x7f8a54158590 2023-10-06T05:07:47.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.378+0000 7f8a3bfff700 1 -- 172.21.15.98:0/4128862933 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a4800fb70 con 0x7f8a54158590 2023-10-06T05:07:47.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.378+0000 7f8a5a3e4700 1 -- 172.21.15.98:0/4128862933 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a540c9350 con 0x7f8a54158590 2023-10-06T05:07:47.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.378+0000 7f8a3bfff700 1 -- 172.21.15.98:0/4128862933 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a48014440 con 0x7f8a54158590 2023-10-06T05:07:47.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.378+0000 7f8a5a3e4700 1 -- 172.21.15.98:0/4128862933 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a5415a200 con 0x7f8a54158590 2023-10-06T05:07:47.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.379+0000 7f8a5a3e4700 1 -- 172.21.15.98:0/4128862933 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a54064a90 con 0x7f8a54158590 2023-10-06T05:07:47.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.380+0000 7f8a3bfff700 1 -- 172.21.15.98:0/4128862933 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f8a48014870 con 0x7f8a54158590 2023-10-06T05:07:47.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.380+0000 7f8a3bfff700 1 --2- 172.21.15.98:0/4128862933 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f8a40066fb0 0x7f8a40069460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:47.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.380+0000 7f8a3bfff700 1 -- 172.21.15.98:0/4128862933 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6043+0+0 (secure 0 0 0) 0x7f8a48018030 con 0x7f8a54158590 2023-10-06T05:07:47.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.380+0000 7f8a53fff700 1 --2- 172.21.15.98:0/4128862933 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f8a40066fb0 0x7f8a40069460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:47.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.381+0000 7f8a53fff700 1 --2- 172.21.15.98:0/4128862933 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f8a40066fb0 0x7f8a40069460 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f8a44008fd0 tx=0x7f8a44015000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:47.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.384+0000 7f8a3bfff700 1 -- 172.21.15.98:0/4128862933 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8a480520d0 con 0x7f8a54158590 2023-10-06T05:07:47.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:47.567+0000 7f8a5a3e4700 1 -- 172.21.15.98:0/4128862933 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f8a540c9800 con 0x7f8a54158590 2023-10-06T05:07:48.238 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:48.237+0000 7f8a3bfff700 1 -- 172.21.15.98:0/4128862933 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v35) v1 ==== 178+0+0 (secure 0 0 0) 0x7f8a480556f0 con 0x7f8a54158590 2023-10-06T05:07:48.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:48.239+0000 7f8a5a3e4700 1 -- 172.21.15.98:0/4128862933 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f8a40066fb0 msgr2=0x7f8a40069460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:48.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:48.240+0000 7f8a5a3e4700 1 --2- 172.21.15.98:0/4128862933 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f8a40066fb0 0x7f8a40069460 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f8a44008fd0 tx=0x7f8a44015000 comp rx=0 tx=0).stop 2023-10-06T05:07:48.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:48.240+0000 7f8a5a3e4700 1 -- 172.21.15.98:0/4128862933 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a54158590 msgr2=0x7f8a540c9b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:48.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:48.240+0000 7f8a5a3e4700 1 --2- 172.21.15.98:0/4128862933 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a54158590 0x7f8a540c9b50 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f8a48011d40 tx=0x7f8a480091a0 comp rx=0 tx=0).stop 2023-10-06T05:07:48.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:48.240+0000 7f8a5a3e4700 1 -- 172.21.15.98:0/4128862933 shutdown_connections 2023-10-06T05:07:48.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:48.240+0000 7f8a5a3e4700 1 --2- 172.21.15.98:0/4128862933 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8a540ca090 0x7f8a54159cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:48.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:48.240+0000 7f8a5a3e4700 1 --2- 172.21.15.98:0/4128862933 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a3c003a90 0x7f8a54158050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:48.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:48.240+0000 7f8a5a3e4700 1 --2- 172.21.15.98:0/4128862933 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f8a40066fb0 0x7f8a40069460 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:48.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:48.240+0000 7f8a5a3e4700 1 --2- 172.21.15.98:0/4128862933 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a54158590 0x7f8a540c9b50 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:48.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:48.240+0000 7f8a5a3e4700 1 -- 172.21.15.98:0/4128862933 >> 172.21.15.98:0/4128862933 conn(0x7f8a540682f0 msgr2=0x7f8a540ccc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:48.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:48.240+0000 7f8a5a3e4700 1 -- 172.21.15.98:0/4128862933 shutdown_connections 2023-10-06T05:07:48.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:48.240+0000 7f8a5a3e4700 1 -- 172.21.15.98:0/4128862933 wait complete. 2023-10-06T05:07:48.242 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'cephfs_data_ec' removed 2023-10-06T05:07:48.255 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-06T05:07:48.255 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-06T05:07:48.255 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-06T05:07:48.255 INFO:teuthology.orchestra.run:waiting for 300 2023-10-06T05:07:48.255 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-06T05:07:48.256 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-06T05:07:48.256 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-06T05:07:48.256 INFO:teuthology.orchestra.run:waiting for 300 2023-10-06T05:07:48.256 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-06T05:07:48.256 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-06T05:07:48.256 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-06T05:07:48.257 INFO:teuthology.orchestra.run:waiting for 300 2023-10-06T05:07:48.257 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-06T05:07:48.257 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-06T05:07:48.257 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-06T05:07:48.257 INFO:teuthology.orchestra.run:waiting for 300 2023-10-06T05:07:48.264 INFO:tasks.ceph.mds.a:Stopped 2023-10-06T05:07:48.265 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-06T05:07:48.266 INFO:tasks.ceph.mds.c:Stopped 2023-10-06T05:07:48.266 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-06T05:07:48.268 INFO:tasks.ceph.mds.b:Stopped 2023-10-06T05:07:48.268 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-06T05:07:48.269 INFO:tasks.ceph.mds.d:Stopped 2023-10-06T05:07:48.269 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-06T05:07:48.307 INFO:tasks.ceph.mds.a:Started 2023-10-06T05:07:48.307 DEBUG:teuthology.parallel:result is None 2023-10-06T05:07:48.308 INFO:tasks.ceph.mds.c:Started 2023-10-06T05:07:48.308 DEBUG:teuthology.parallel:result is None 2023-10-06T05:07:48.310 INFO:tasks.ceph.mds.b:Started 2023-10-06T05:07:48.311 DEBUG:teuthology.parallel:result is None 2023-10-06T05:07:48.311 INFO:tasks.ceph.mds.d:Started 2023-10-06T05:07:48.311 DEBUG:teuthology.parallel:result is None 2023-10-06T05:07:48.312 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-06T05:07:48.380 INFO:tasks.ceph.mds.a.smithi098.stdout:starting mds.a at 2023-10-06T05:07:48.380 INFO:tasks.ceph.mds.c.smithi098.stdout:starting mds.c at 2023-10-06T05:07:48.388 INFO:tasks.ceph.mds.b.smithi174.stdout:starting mds.b at 2023-10-06T05:07:48.388 INFO:tasks.ceph.mds.d.smithi174.stdout:starting mds.d at 2023-10-06T05:07:48.435 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:48.434+0000 7fd59e1c7700 1 Processor -- start 2023-10-06T05:07:48.435 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:48.434+0000 7fd59e1c7700 1 -- start start 2023-10-06T05:07:48.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:48.434+0000 7fd59e1c7700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fd598159560 0x7fd5980aa530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:48.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:48.434+0000 7fd59e1c7700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd5980b49a0 0x7fd5980aaa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:48.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:48.435+0000 7fd59e1c7700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5980b0c60 con 0x7fd5980b0760 2023-10-06T05:07:48.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:48.435+0000 7fd59e1c7700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5980b0f40 con 0x7fd598159560 2023-10-06T05:07:48.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:48.435+0000 7fd59e1c7700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5980b1220 con 0x7fd5980b49a0 2023-10-06T05:07:48.438 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:48.435+0000 7fd596ffd700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd5980b49a0 0x7fd5980aaa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:48.438 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:48.435+0000 7fd5977fe700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fd598159560 0x7fd5980aa530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:48.438 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:48.435+0000 7fd596ffd700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd5980b49a0 0x7fd5980aaa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.98:57680/0 (socket says 172.21.15.98:57680) 2023-10-06T05:07:48.438 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:48.435+0000 7fd596ffd700 1 -- 172.21.15.98:0/1070240549 learned_addr learned my addr 172.21.15.98:0/1070240549 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:48.438 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:48.435+0000 7fd597fff700 1 --1- 172.21.15.98:0/1070240549 >> v1:172.21.15.174:6789/0 conn(0x7fd5980b0760 0x7fd5980b3890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.98:49660/0 (socket says 172.21.15.98:49660) 2023-10-06T05:07:48.438 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:48.435+0000 7fd596ffd700 1 -- 172.21.15.98:0/1070240549 >> v2:172.21.15.174:3300/0 conn(0x7fd598159560 msgr2=0x7fd5980aa530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:48.438 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:48.435+0000 7fd596ffd700 1 --2- 172.21.15.98:0/1070240549 >> v2:172.21.15.174:3300/0 conn(0x7fd598159560 0x7fd5980aa530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:48.438 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:48.435+0000 7fd596ffd700 1 -- 172.21.15.98:0/1070240549 >> v1:172.21.15.174:6789/0 conn(0x7fd5980b0760 legacy=0x7fd5980b3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:48.438 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:48.435+0000 7fd596ffd700 1 -- 172.21.15.98:0/1070240549 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5980b1500 con 0x7fd5980b49a0 2023-10-06T05:07:48.438 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:48.435+0000 7fd5977fe700 1 --2- 172.21.15.98:0/1070240549 >> v2:172.21.15.174:3300/0 conn(0x7fd598159560 0x7fd5980aa530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T05:07:48.438 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:48.435+0000 7fd596ffd700 1 --2- 172.21.15.98:0/1070240549 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd5980b49a0 0x7fd5980aaa70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd58c0099b0 tx=0x7fd58c009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6029d09631d4b83e server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:48.438 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:48.435+0000 7fd5967fc700 1 -- 172.21.15.98:0/1070240549 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd58c009ea0 con 0x7fd5980b49a0 2023-10-06T05:07:48.439 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:48.436+0000 7fd5967fc700 1 -- 172.21.15.98:0/1070240549 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd58c00b860 con 0x7fd5980b49a0 2023-10-06T05:07:48.439 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:48.436+0000 7fd5967fc700 1 -- 172.21.15.98:0/1070240549 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd58c00bc00 con 0x7fd5980b49a0 2023-10-06T05:07:48.439 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:48.436+0000 7fd59e1c7700 1 -- 172.21.15.98:0/1070240549 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd5980b49a0 msgr2=0x7fd5980aaa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:48.439 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:48.436+0000 7fd59e1c7700 1 --2- 172.21.15.98:0/1070240549 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd5980b49a0 0x7fd5980aaa70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd58c0099b0 tx=0x7fd58c009250 comp rx=0 tx=0).stop 2023-10-06T05:07:48.439 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:48.436+0000 7fd59e1c7700 1 -- 172.21.15.98:0/1070240549 shutdown_connections 2023-10-06T05:07:48.439 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:48.436+0000 7fd59e1c7700 1 --2- 172.21.15.98:0/1070240549 >> v2:172.21.15.174:3300/0 conn(0x7fd598159560 0x7fd5980aa530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:48.439 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:48.436+0000 7fd59e1c7700 1 --2- 172.21.15.98:0/1070240549 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd5980b49a0 0x7fd5980aaa70 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:48.439 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:48.436+0000 7fd59e1c7700 1 -- 172.21.15.98:0/1070240549 >> 172.21.15.98:0/1070240549 conn(0x7fd5980997f0 msgr2=0x7fd59806f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:48.439 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:48.436+0000 7fd59e1c7700 1 -- 172.21.15.98:0/1070240549 shutdown_connections 2023-10-06T05:07:48.439 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:48.436+0000 7fd59e1c7700 1 -- 172.21.15.98:0/1070240549 wait complete. 2023-10-06T05:07:48.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:48.437+0000 7fd59e1c7700 1 Processor -- start 2023-10-06T05:07:48.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:48.437+0000 7fd59e1c7700 1 -- start start 2023-10-06T05:07:48.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:48.438+0000 7fd59e1c7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5980b0760 0x7fd598156a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:48.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:48.438+0000 7fd59e1c7700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd5980b49a0 0x7fd59815a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:48.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:48.438+0000 7fd59e1c7700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd598159560 0x7fd59809d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:48.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:48.438+0000 7fd59e1c7700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd59809a050 con 0x7fd5980b0760 2023-10-06T05:07:48.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:48.438+0000 7fd59e1c7700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd59809a330 con 0x7fd598159560 2023-10-06T05:07:48.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:48.438+0000 7fd59e1c7700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd59815ab90 con 0x7fd5980b49a0 2023-10-06T05:07:48.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:48.438+0000 7fd5977fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5980b0760 0x7fd598156a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:48.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:48.438+0000 7fd5977fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5980b0760 0x7fd598156a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:60574/0 (socket says 172.21.15.98:60574) 2023-10-06T05:07:48.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:48.438+0000 7fd5977fe700 1 -- 172.21.15.98:0/707683810 learned_addr learned my addr 172.21.15.98:0/707683810 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:48.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:48.438+0000 7fd596ffd700 1 --2- 172.21.15.98:0/707683810 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd5980b49a0 0x7fd59815a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:48.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:48.438+0000 7fd597fff700 1 --2- 172.21.15.98:0/707683810 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd598159560 0x7fd59809d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:48.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:48.438+0000 7fd5977fe700 1 -- 172.21.15.98:0/707683810 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd5980b49a0 msgr2=0x7fd59815a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:48.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:48.438+0000 7fd5977fe700 1 --2- 172.21.15.98:0/707683810 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd5980b49a0 0x7fd59815a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:48.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:48.438+0000 7fd5977fe700 1 -- 172.21.15.98:0/707683810 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd598159560 msgr2=0x7fd59809d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:48.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:48.438+0000 7fd5977fe700 1 --2- 172.21.15.98:0/707683810 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd598159560 0x7fd59809d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:48.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:48.438+0000 7fd5977fe700 1 -- 172.21.15.98:0/707683810 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5880093c0 con 0x7fd5980b0760 2023-10-06T05:07:48.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:48.439+0000 7fd5977fe700 1 --2- 172.21.15.98:0/707683810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5980b0760 0x7fd598156a10 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fd58800ec80 tx=0x7fd588009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:48.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:48.439+0000 7fd57f7fe700 1 -- 172.21.15.98:0/707683810 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd588005a00 con 0x7fd5980b0760 2023-10-06T05:07:48.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:48.439+0000 7fd57f7fe700 1 -- 172.21.15.98:0/707683810 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd588005b60 con 0x7fd5980b0760 2023-10-06T05:07:48.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:48.439+0000 7fd59e1c7700 1 -- 172.21.15.98:0/707683810 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd58c0094f0 con 0x7fd5980b0760 2023-10-06T05:07:48.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:48.439+0000 7fd57f7fe700 1 -- 172.21.15.98:0/707683810 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd58800c850 con 0x7fd5980b0760 2023-10-06T05:07:48.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:48.439+0000 7fd59e1c7700 1 -- 172.21.15.98:0/707683810 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd59815b150 con 0x7fd5980b0760 2023-10-06T05:07:48.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:48.440+0000 7fd59e1c7700 1 -- 172.21.15.98:0/707683810 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd598158190 con 0x7fd5980b0760 2023-10-06T05:07:48.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:48.440+0000 7fd57f7fe700 1 -- 172.21.15.98:0/707683810 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fd58800cbc0 con 0x7fd5980b0760 2023-10-06T05:07:48.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:48.440+0000 7fd57f7fe700 1 --2- 172.21.15.98:0/707683810 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fd584066f30 0x7fd5840693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:48.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:48.440+0000 7fd57f7fe700 1 -- 172.21.15.98:0/707683810 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5612+0+0 (secure 0 0 0) 0x7fd5880848a0 con 0x7fd5980b0760 2023-10-06T05:07:48.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:48.441+0000 7fd596ffd700 1 --2- 172.21.15.98:0/707683810 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fd584066f30 0x7fd5840693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:48.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:48.441+0000 7fd596ffd700 1 --2- 172.21.15.98:0/707683810 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fd584066f30 0x7fd5840693e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd58c0099b0 tx=0x7fd58c017000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:48.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:48.445+0000 7fd57f7fe700 1 -- 172.21.15.98:0/707683810 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd588051b70 con 0x7fd5980b0760 2023-10-06T05:07:48.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:48.653+0000 7fd59e1c7700 1 -- 172.21.15.98:0/707683810 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fd598064a90 con 0x7fd5980b0760 2023-10-06T05:07:49.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.257+0000 7fd57f7fe700 1 -- 172.21.15.98:0/707683810 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v37) v1 ==== 97+0+0 (secure 0 0 0) 0x7fd588055190 con 0x7fd5980b0760 2023-10-06T05:07:49.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.259+0000 7fd59e1c7700 1 -- 172.21.15.98:0/707683810 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fd584066f30 msgr2=0x7fd5840693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:49.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.259+0000 7fd59e1c7700 1 --2- 172.21.15.98:0/707683810 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fd584066f30 0x7fd5840693e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd58c0099b0 tx=0x7fd58c017000 comp rx=0 tx=0).stop 2023-10-06T05:07:49.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.259+0000 7fd59e1c7700 1 -- 172.21.15.98:0/707683810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5980b0760 msgr2=0x7fd598156a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:49.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.259+0000 7fd59e1c7700 1 --2- 172.21.15.98:0/707683810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5980b0760 0x7fd598156a10 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fd58800ec80 tx=0x7fd588009120 comp rx=0 tx=0).stop 2023-10-06T05:07:49.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.259+0000 7fd59e1c7700 1 -- 172.21.15.98:0/707683810 shutdown_connections 2023-10-06T05:07:49.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.259+0000 7fd59e1c7700 1 --2- 172.21.15.98:0/707683810 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd598159560 0x7fd59809d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:49.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.259+0000 7fd59e1c7700 1 --2- 172.21.15.98:0/707683810 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fd584066f30 0x7fd5840693e0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:49.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.259+0000 7fd59e1c7700 1 --2- 172.21.15.98:0/707683810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5980b0760 0x7fd598156a10 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:49.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.259+0000 7fd59e1c7700 1 --2- 172.21.15.98:0/707683810 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd5980b49a0 0x7fd59815a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:49.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.259+0000 7fd59e1c7700 1 -- 172.21.15.98:0/707683810 >> 172.21.15.98:0/707683810 conn(0x7fd5980997f0 msgr2=0x7fd598158870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:49.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.259+0000 7fd59e1c7700 1 -- 172.21.15.98:0/707683810 shutdown_connections 2023-10-06T05:07:49.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.259+0000 7fd59e1c7700 1 -- 172.21.15.98:0/707683810 wait complete. 2023-10-06T05:07:49.648 INFO:teuthology.orchestra.run.smithi098.stderr: removed all blocklist entries 2023-10-06T05:07:49.648 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-06T05:07:49.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.764+0000 7fbe22ef8700 1 Processor -- start 2023-10-06T05:07:49.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.764+0000 7fbe22ef8700 1 -- start start 2023-10-06T05:07:49.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.765+0000 7fbe22ef8700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbe1c1577f0 0x7fbe1c0ae8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:49.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.765+0000 7fbe22ef8700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe1c0cae20 con 0x7fbe1c157420 2023-10-06T05:07:49.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.765+0000 7fbe22ef8700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe1c0aee90 con 0x7fbe1c0ac440 2023-10-06T05:07:49.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.765+0000 7fbe22ef8700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe1c0af170 con 0x7fbe1c1577f0 2023-10-06T05:07:49.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.765+0000 7fbe20c94700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fbe1c157420 0x7fbe1c0cc260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:55694/0 (socket says 172.21.15.98:55694) 2023-10-06T05:07:49.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.765+0000 7fbe20c94700 1 -- 172.21.15.98:0/1125714864 learned_addr learned my addr 172.21.15.98:0/1125714864 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:07:49.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.765+0000 7fbe1bfff700 1 --2- 172.21.15.98:0/1125714864 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbe1c1577f0 0x7fbe1c0ae8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:49.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.766+0000 7fbe1b7fe700 1 -- 172.21.15.98:0/1125714864 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4147627901 0 0) 0x7fbe1c0cae20 con 0x7fbe1c157420 2023-10-06T05:07:49.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.766+0000 7fbe1b7fe700 1 -- 172.21.15.98:0/1125714864 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe08003570 con 0x7fbe1c157420 2023-10-06T05:07:49.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.766+0000 7fbe1b7fe700 1 -- 172.21.15.98:0/1125714864 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 313895328 0 0) 0x7fbe1c0aee90 con 0x7fbe1c0ac440 2023-10-06T05:07:49.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.766+0000 7fbe1b7fe700 1 -- 172.21.15.98:0/1125714864 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe1c0cae20 con 0x7fbe1c0ac440 2023-10-06T05:07:49.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.766+0000 7fbe1b7fe700 1 -- 172.21.15.98:0/1125714864 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1702902073 0 0) 0x7fbe08003570 con 0x7fbe1c157420 2023-10-06T05:07:49.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.766+0000 7fbe1b7fe700 1 -- 172.21.15.98:0/1125714864 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbe1c1577f0 msgr2=0x7fbe1c0ae8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:49.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.766+0000 7fbe1b7fe700 1 --2- 172.21.15.98:0/1125714864 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbe1c1577f0 0x7fbe1c0ae8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:49.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.766+0000 7fbe1b7fe700 1 -- 172.21.15.98:0/1125714864 >> v1:172.21.15.174:6789/0 conn(0x7fbe1c0ac440 legacy=0x7fbe1c0685c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:49.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.766+0000 7fbe1b7fe700 1 -- 172.21.15.98:0/1125714864 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe1c0af450 con 0x7fbe1c157420 2023-10-06T05:07:49.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.766+0000 7fbe1b7fe700 1 -- 172.21.15.98:0/1125714864 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1543991876 0 0) 0x7fbe140018f0 con 0x7fbe1c157420 2023-10-06T05:07:49.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.766+0000 7fbe1b7fe700 1 -- 172.21.15.98:0/1125714864 >> v1:172.21.15.98:6789/0 conn(0x7fbe1c157420 legacy=0x7fbe1c0cc260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:49.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.766+0000 7fbe1b7fe700 1 --2- 172.21.15.98:0/1125714864 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbe08003c20 0x7fbe08004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:49.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.766+0000 7fbe1b7fe700 1 --2- 172.21.15.98:0/1125714864 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe08004810 0x7fbe08004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:49.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.766+0000 7fbe1b7fe700 1 --2- 172.21.15.98:0/1125714864 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbe1c1577f0 0x7fbe080052e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:49.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.766+0000 7fbe1b7fe700 1 -- 172.21.15.98:0/1125714864 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe1c0af170 con 0x7fbe08004810 2023-10-06T05:07:49.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.766+0000 7fbe1b7fe700 1 -- 172.21.15.98:0/1125714864 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe08010080 con 0x7fbe08003c20 2023-10-06T05:07:49.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.766+0000 7fbe1b7fe700 1 -- 172.21.15.98:0/1125714864 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe08010240 con 0x7fbe1c1577f0 2023-10-06T05:07:49.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.766+0000 7fbe1bfff700 1 --2- 172.21.15.98:0/1125714864 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe08004810 0x7fbe08004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:49.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.766+0000 7fbe20c94700 1 --2- 172.21.15.98:0/1125714864 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbe08003c20 0x7fbe08004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:49.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.766+0000 7fbe21495700 1 --2- 172.21.15.98:0/1125714864 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbe1c1577f0 0x7fbe080052e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:49.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.766+0000 7fbe1bfff700 1 -- 172.21.15.98:0/1125714864 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbe1c1577f0 msgr2=0x7fbe080052e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:49.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.766+0000 7fbe1bfff700 1 --2- 172.21.15.98:0/1125714864 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbe1c1577f0 0x7fbe080052e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:49.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.766+0000 7fbe1bfff700 1 -- 172.21.15.98:0/1125714864 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbe08003c20 msgr2=0x7fbe08004010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:49.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.766+0000 7fbe1bfff700 1 --2- 172.21.15.98:0/1125714864 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbe08003c20 0x7fbe08004010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:49.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.766+0000 7fbe1bfff700 1 -- 172.21.15.98:0/1125714864 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe08003570 con 0x7fbe08004810 2023-10-06T05:07:49.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.767+0000 7fbe1bfff700 1 --2- 172.21.15.98:0/1125714864 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe08004810 0x7fbe08004c80 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fbe1c15c150 tx=0x7fbe0c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e25ba0325180080 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:49.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.767+0000 7fbe1b7fe700 1 -- 172.21.15.98:0/1125714864 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe0c00e710 con 0x7fbe08004810 2023-10-06T05:07:49.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.767+0000 7fbe1b7fe700 1 -- 172.21.15.98:0/1125714864 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe0c00e870 con 0x7fbe08004810 2023-10-06T05:07:49.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.767+0000 7fbe1b7fe700 1 -- 172.21.15.98:0/1125714864 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe0c00ec30 con 0x7fbe08004810 2023-10-06T05:07:49.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.767+0000 7fbe22ef8700 1 -- 172.21.15.98:0/1125714864 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe08004810 msgr2=0x7fbe08004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:49.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.767+0000 7fbe22ef8700 1 --2- 172.21.15.98:0/1125714864 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe08004810 0x7fbe08004c80 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fbe1c15c150 tx=0x7fbe0c0091a0 comp rx=0 tx=0).stop 2023-10-06T05:07:49.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.767+0000 7fbe21495700 1 -- 172.21.15.98:0/1125714864 reap_dead start 2023-10-06T05:07:49.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.767+0000 7fbe22ef8700 1 -- 172.21.15.98:0/1125714864 shutdown_connections 2023-10-06T05:07:49.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.767+0000 7fbe22ef8700 1 -- 172.21.15.98:0/1125714864 >> 172.21.15.98:0/1125714864 conn(0x7fbe1c06de40 msgr2=0x7fbe1c06e240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:49.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.767+0000 7fbe22ef8700 1 -- 172.21.15.98:0/1125714864 shutdown_connections 2023-10-06T05:07:49.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.767+0000 7fbe22ef8700 1 -- 172.21.15.98:0/1125714864 wait complete. 2023-10-06T05:07:49.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.768+0000 7fbe22ef8700 1 Processor -- start 2023-10-06T05:07:49.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.769+0000 7fbe22ef8700 1 -- start start 2023-10-06T05:07:49.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.769+0000 7fbe22ef8700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbe08004810 0x7fbe1c15ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:49.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.769+0000 7fbe22ef8700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe1c1426e0 0x7fbe1c159ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:49.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.769+0000 7fbe22ef8700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbe1c15a3e0 0x7fbe1c0ad860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:49.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.769+0000 7fbe22ef8700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe1c15b600 con 0x7fbe1c1426e0 2023-10-06T05:07:49.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.769+0000 7fbe22ef8700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe1c142b50 con 0x7fbe1c15a3e0 2023-10-06T05:07:49.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.769+0000 7fbe22ef8700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe1c159860 con 0x7fbe08004810 2023-10-06T05:07:49.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.769+0000 7fbe1bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe1c1426e0 0x7fbe1c159ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:49.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.769+0000 7fbe1bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe1c1426e0 0x7fbe1c159ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47358/0 (socket says 172.21.15.98:47358) 2023-10-06T05:07:49.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.769+0000 7fbe1bfff700 1 -- 172.21.15.98:0/3772812430 learned_addr learned my addr 172.21.15.98:0/3772812430 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:49.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.769+0000 7fbe20c94700 1 --2- 172.21.15.98:0/3772812430 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbe08004810 0x7fbe1c15ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:49.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.769+0000 7fbe21495700 1 --2- 172.21.15.98:0/3772812430 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbe1c15a3e0 0x7fbe1c0ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:49.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.770+0000 7fbe1bfff700 1 -- 172.21.15.98:0/3772812430 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbe08004810 msgr2=0x7fbe1c15ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:49.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.770+0000 7fbe1bfff700 1 --2- 172.21.15.98:0/3772812430 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbe08004810 0x7fbe1c15ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:49.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.770+0000 7fbe1bfff700 1 -- 172.21.15.98:0/3772812430 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbe1c15a3e0 msgr2=0x7fbe1c0ad860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:49.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.770+0000 7fbe1bfff700 1 --2- 172.21.15.98:0/3772812430 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbe1c15a3e0 0x7fbe1c0ad860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:49.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.770+0000 7fbe1bfff700 1 -- 172.21.15.98:0/3772812430 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe1c15a9f0 con 0x7fbe1c1426e0 2023-10-06T05:07:49.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.770+0000 7fbe1bfff700 1 --2- 172.21.15.98:0/3772812430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe1c1426e0 0x7fbe1c159ea0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fbe0800bc40 tx=0x7fbe0c01a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:49.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.770+0000 7fbe18ff9700 1 -- 172.21.15.98:0/3772812430 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe0c022070 con 0x7fbe1c1426e0 2023-10-06T05:07:49.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.770+0000 7fbe18ff9700 1 -- 172.21.15.98:0/3772812430 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe0c00e9e0 con 0x7fbe1c1426e0 2023-10-06T05:07:49.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.770+0000 7fbe22ef8700 1 -- 172.21.15.98:0/3772812430 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe1c15b600 con 0x7fbe1c1426e0 2023-10-06T05:07:49.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.770+0000 7fbe18ff9700 1 -- 172.21.15.98:0/3772812430 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe0c01c450 con 0x7fbe1c1426e0 2023-10-06T05:07:49.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.770+0000 7fbe22ef8700 1 -- 172.21.15.98:0/3772812430 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe1c15afb0 con 0x7fbe1c1426e0 2023-10-06T05:07:49.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.771+0000 7fbe22ef8700 1 -- 172.21.15.98:0/3772812430 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe1c15a7b0 con 0x7fbe1c1426e0 2023-10-06T05:07:49.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.771+0000 7fbe18ff9700 1 -- 172.21.15.98:0/3772812430 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fbe0c01c800 con 0x7fbe1c1426e0 2023-10-06T05:07:49.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.771+0000 7fbe18ff9700 1 --2- 172.21.15.98:0/3772812430 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fbdfc066f30 0x7fbdfc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:49.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.772+0000 7fbe18ff9700 1 -- 172.21.15.98:0/3772812430 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5526+0+0 (secure 0 0 0) 0x7fbe0c085cf0 con 0x7fbe1c1426e0 2023-10-06T05:07:49.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.772+0000 7fbe20c94700 1 --2- 172.21.15.98:0/3772812430 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fbdfc066f30 0x7fbdfc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:49.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.772+0000 7fbe20c94700 1 --2- 172.21.15.98:0/3772812430 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fbdfc066f30 0x7fbdfc0693e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fbe14001190 tx=0x7fbe14009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:49.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:49.776+0000 7fbe18ff9700 1 -- 172.21.15.98:0/3772812430 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbe1c15a7b0 con 0x7fbe1c1426e0 2023-10-06T05:07:50.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.034+0000 7fbe22ef8700 1 -- 172.21.15.98:0/3772812430 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fbe1c15a7b0 con 0x7fbe1c1426e0 2023-10-06T05:07:50.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.035+0000 7fbe18ff9700 1 -- 172.21.15.98:0/3772812430 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v4) v1 ==== 80+0+6645 (secure 0 0 0) 0x7fbe1c15a7b0 con 0x7fbe1c1426e0 2023-10-06T05:07:50.036 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:07:50.036 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-10-06T05:07:50.036 INFO:teuthology.orchestra.run.smithi098.stdout: "auth_dump": [ 2023-10-06T05:07:50.036 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-10-06T05:07:50.036 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "mds.a", 2023-10-06T05:07:50.036 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC3lR9l8/dMFhAANvojBT5KMbxAA/zmYdKwdQ==", 2023-10-06T05:07:50.037 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-10-06T05:07:50.037 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow", 2023-10-06T05:07:50.037 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow *", 2023-10-06T05:07:50.037 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow *", 2023-10-06T05:07:50.037 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-10-06T05:07:50.037 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-10-06T05:07:50.037 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-10-06T05:07:50.037 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-10-06T05:07:50.037 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "mds.b", 2023-10-06T05:07:50.037 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC3lR9lvbV3IRAAqPaSuCsrRugVa4cioHdaeQ==", 2023-10-06T05:07:50.037 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-10-06T05:07:50.037 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow", 2023-10-06T05:07:50.038 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow *", 2023-10-06T05:07:50.038 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow *", 2023-10-06T05:07:50.038 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-10-06T05:07:50.038 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-10-06T05:07:50.038 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-10-06T05:07:50.038 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-10-06T05:07:50.038 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "mds.c", 2023-10-06T05:07:50.038 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC3lR9lstsTHRAAFofaYDUECIUfeln+MEyJ9w==", 2023-10-06T05:07:50.038 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-10-06T05:07:50.038 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow", 2023-10-06T05:07:50.038 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow *", 2023-10-06T05:07:50.039 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow *", 2023-10-06T05:07:50.039 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-10-06T05:07:50.039 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-10-06T05:07:50.039 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-10-06T05:07:50.039 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-10-06T05:07:50.039 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "mds.d", 2023-10-06T05:07:50.039 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC3lR9lLGE6KBAAIYl9F6qfRBCRgk1I78L4wA==", 2023-10-06T05:07:50.039 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-10-06T05:07:50.039 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow", 2023-10-06T05:07:50.039 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow *", 2023-10-06T05:07:50.039 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow *", 2023-10-06T05:07:50.040 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-10-06T05:07:50.040 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-10-06T05:07:50.040 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-10-06T05:07:50.040 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-10-06T05:07:50.040 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "osd.0", 2023-10-06T05:07:50.040 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC7lR9lWNtrEBAAT2EktzoBT2dsJmiioQ4ySg==", 2023-10-06T05:07:50.040 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-10-06T05:07:50.040 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow profile osd", 2023-10-06T05:07:50.040 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile osd", 2023-10-06T05:07:50.040 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-10-06T05:07:50.040 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-10-06T05:07:50.041 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-10-06T05:07:50.041 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-10-06T05:07:50.041 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "osd.1", 2023-10-06T05:07:50.041 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC+lR9l/JepDRAA8FyqlCk/OfnLWtD5Ilnjzw==", 2023-10-06T05:07:50.041 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-10-06T05:07:50.041 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow profile osd", 2023-10-06T05:07:50.041 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile osd", 2023-10-06T05:07:50.041 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-10-06T05:07:50.041 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-10-06T05:07:50.041 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-10-06T05:07:50.041 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-10-06T05:07:50.041 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "osd.2", 2023-10-06T05:07:50.042 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQDBlR9ljslNERAAqnf/WmMPMgNrW4T5pK+59w==", 2023-10-06T05:07:50.042 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-10-06T05:07:50.042 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow profile osd", 2023-10-06T05:07:50.042 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile osd", 2023-10-06T05:07:50.042 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-10-06T05:07:50.042 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-10-06T05:07:50.042 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-10-06T05:07:50.042 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-10-06T05:07:50.042 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "osd.3", 2023-10-06T05:07:50.042 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQDElR9lMzfaDhAAhtX4bPxsE/x9EldgsnfWdA==", 2023-10-06T05:07:50.042 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-10-06T05:07:50.043 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow profile osd", 2023-10-06T05:07:50.043 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile osd", 2023-10-06T05:07:50.043 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-10-06T05:07:50.043 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-10-06T05:07:50.043 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-10-06T05:07:50.043 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-10-06T05:07:50.043 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "osd.4", 2023-10-06T05:07:50.043 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQDKlR9leHV3EhAAwOyH0FPXKgz7jRmckY7F4A==", 2023-10-06T05:07:50.043 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-10-06T05:07:50.043 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow profile osd", 2023-10-06T05:07:50.043 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile osd", 2023-10-06T05:07:50.043 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-10-06T05:07:50.044 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-10-06T05:07:50.044 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-10-06T05:07:50.044 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-10-06T05:07:50.044 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "osd.5", 2023-10-06T05:07:50.044 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQDNlR9l/OGeEBAAc/1Dn4zBLTGZDHOV+DpyCA==", 2023-10-06T05:07:50.044 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-10-06T05:07:50.044 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow profile osd", 2023-10-06T05:07:50.044 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile osd", 2023-10-06T05:07:50.044 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-10-06T05:07:50.044 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-10-06T05:07:50.044 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-10-06T05:07:50.044 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-10-06T05:07:50.045 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "osd.6", 2023-10-06T05:07:50.045 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQDQlR9ldKHMDhAAzSvXhA/oE0CN/06HES/Vtg==", 2023-10-06T05:07:50.045 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-10-06T05:07:50.045 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow profile osd", 2023-10-06T05:07:50.045 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile osd", 2023-10-06T05:07:50.045 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-10-06T05:07:50.045 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-10-06T05:07:50.045 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-10-06T05:07:50.045 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-10-06T05:07:50.045 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "osd.7", 2023-10-06T05:07:50.045 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQDTlR9lKsrYCxAAarQ0keRxrs5848xZQhBl0Q==", 2023-10-06T05:07:50.045 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-10-06T05:07:50.046 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow profile osd", 2023-10-06T05:07:50.046 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile osd", 2023-10-06T05:07:50.046 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-10-06T05:07:50.046 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-10-06T05:07:50.046 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-10-06T05:07:50.046 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-10-06T05:07:50.046 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "client.0", 2023-10-06T05:07:50.046 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC3lR9lpjoPLBAAAYJorK9BQnAyELQt3AGqDw==", 2023-10-06T05:07:50.046 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-10-06T05:07:50.046 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow", 2023-10-06T05:07:50.046 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow r", 2023-10-06T05:07:50.046 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow rw", 2023-10-06T05:07:50.047 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow rwx" 2023-10-06T05:07:50.047 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-10-06T05:07:50.047 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-10-06T05:07:50.047 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-10-06T05:07:50.047 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "client.1", 2023-10-06T05:07:50.047 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC3lR9lDCdkNBAANLue0SCLNp7ySmlhD97ARQ==", 2023-10-06T05:07:50.047 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-10-06T05:07:50.047 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow", 2023-10-06T05:07:50.047 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow r", 2023-10-06T05:07:50.047 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow rw", 2023-10-06T05:07:50.047 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow rwx" 2023-10-06T05:07:50.048 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-10-06T05:07:50.048 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-10-06T05:07:50.048 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-10-06T05:07:50.048 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "client.2", 2023-10-06T05:07:50.048 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC3lR9lbPdqMRAAB0QpBNi/jn+CQupZAWupMw==", 2023-10-06T05:07:50.048 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-10-06T05:07:50.048 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow", 2023-10-06T05:07:50.048 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow r", 2023-10-06T05:07:50.048 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow rw", 2023-10-06T05:07:50.048 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow rwx" 2023-10-06T05:07:50.048 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-10-06T05:07:50.048 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-10-06T05:07:50.049 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-10-06T05:07:50.049 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "client.3", 2023-10-06T05:07:50.049 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC3lR9lAPzEORAAD6oTL9GOgguazIoZRPMpAA==", 2023-10-06T05:07:50.049 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-10-06T05:07:50.049 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow", 2023-10-06T05:07:50.049 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow r", 2023-10-06T05:07:50.049 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow rw", 2023-10-06T05:07:50.049 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow rwx" 2023-10-06T05:07:50.049 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-10-06T05:07:50.049 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-10-06T05:07:50.049 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-10-06T05:07:50.049 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "client.admin", 2023-10-06T05:07:50.050 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC2lR9lPUQWNxAA/2AZIE6Ye+dPvk929oSmZA==", 2023-10-06T05:07:50.050 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-10-06T05:07:50.050 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow *", 2023-10-06T05:07:50.050 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": "allow *", 2023-10-06T05:07:50.050 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow *", 2023-10-06T05:07:50.050 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-10-06T05:07:50.050 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-10-06T05:07:50.050 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-10-06T05:07:50.050 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-10-06T05:07:50.050 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "client.bootstrap-mds", 2023-10-06T05:07:50.050 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQDYlR9ligjqORAAagRZv+12mYJj/chCJIdatw==", 2023-10-06T05:07:50.050 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-10-06T05:07:50.051 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile bootstrap-mds" 2023-10-06T05:07:50.051 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-10-06T05:07:50.051 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-10-06T05:07:50.051 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-10-06T05:07:50.051 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "client.bootstrap-mgr", 2023-10-06T05:07:50.051 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQDYlR9lghDqORAAayWGZnITvqS9LSlr1R3p1Q==", 2023-10-06T05:07:50.051 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-10-06T05:07:50.051 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile bootstrap-mgr" 2023-10-06T05:07:50.051 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-10-06T05:07:50.051 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-10-06T05:07:50.051 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-10-06T05:07:50.051 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "client.bootstrap-osd", 2023-10-06T05:07:50.052 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQDYlR9lyRjqORAAwgOhVNDMCSEDhk6Hv8xfpg==", 2023-10-06T05:07:50.052 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-10-06T05:07:50.052 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile bootstrap-osd" 2023-10-06T05:07:50.052 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-10-06T05:07:50.052 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-10-06T05:07:50.052 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-10-06T05:07:50.052 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "client.bootstrap-rbd", 2023-10-06T05:07:50.052 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQDYlR9lqyDqORAASHBgKdc+PWuMGBb3wMwPqQ==", 2023-10-06T05:07:50.052 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-10-06T05:07:50.052 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile bootstrap-rbd" 2023-10-06T05:07:50.052 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-10-06T05:07:50.053 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-10-06T05:07:50.053 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-10-06T05:07:50.053 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "client.bootstrap-rbd-mirror", 2023-10-06T05:07:50.053 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQDYlR9leSfqORAA0ooertu6bcaJp+3WrRuAGA==", 2023-10-06T05:07:50.053 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-10-06T05:07:50.053 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2023-10-06T05:07:50.053 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-10-06T05:07:50.053 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-10-06T05:07:50.053 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-10-06T05:07:50.053 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "client.bootstrap-rgw", 2023-10-06T05:07:50.053 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQDYlR9lai7qORAACPbC/YhnQEn4tYqVvpArug==", 2023-10-06T05:07:50.053 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-10-06T05:07:50.053 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile bootstrap-rgw" 2023-10-06T05:07:50.054 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-10-06T05:07:50.054 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-10-06T05:07:50.054 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-10-06T05:07:50.054 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "mgr.x", 2023-10-06T05:07:50.054 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC3lR9leW8+ExAAGn+V3TntGoC+CdBjnyWBuw==", 2023-10-06T05:07:50.054 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-10-06T05:07:50.054 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow *", 2023-10-06T05:07:50.054 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile mgr", 2023-10-06T05:07:50.054 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-10-06T05:07:50.054 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-10-06T05:07:50.054 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-10-06T05:07:50.055 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-10-06T05:07:50.055 INFO:teuthology.orchestra.run.smithi098.stdout: "entity": "mgr.y", 2023-10-06T05:07:50.055 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "AQC3lR9lop4MEBAAaic3jAzQKe2Yz+BX80cK4Q==", 2023-10-06T05:07:50.055 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": { 2023-10-06T05:07:50.055 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": "allow *", 2023-10-06T05:07:50.055 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": "allow profile mgr", 2023-10-06T05:07:50.055 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": "allow *" 2023-10-06T05:07:50.055 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-10-06T05:07:50.055 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-10-06T05:07:50.055 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-10-06T05:07:50.055 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-10-06T05:07:50.057 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.036+0000 7fbe22ef8700 1 -- 172.21.15.98:0/3772812430 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fbdfc066f30 msgr2=0x7fbdfc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:50.057 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.036+0000 7fbe22ef8700 1 --2- 172.21.15.98:0/3772812430 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fbdfc066f30 0x7fbdfc0693e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fbe14001190 tx=0x7fbe14009040 comp rx=0 tx=0).stop 2023-10-06T05:07:50.057 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.036+0000 7fbe22ef8700 1 -- 172.21.15.98:0/3772812430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe1c1426e0 msgr2=0x7fbe1c159ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:50.057 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.036+0000 7fbe22ef8700 1 --2- 172.21.15.98:0/3772812430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe1c1426e0 0x7fbe1c159ea0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fbe0800bc40 tx=0x7fbe0c01a000 comp rx=0 tx=0).stop 2023-10-06T05:07:50.057 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.036+0000 7fbe22ef8700 1 -- 172.21.15.98:0/3772812430 shutdown_connections 2023-10-06T05:07:50.057 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.036+0000 7fbe22ef8700 1 --2- 172.21.15.98:0/3772812430 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbe1c15a3e0 0x7fbe1c0ad860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:50.057 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.036+0000 7fbe22ef8700 1 --2- 172.21.15.98:0/3772812430 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbe08004810 0x7fbe1c15ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:50.058 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.036+0000 7fbe22ef8700 1 --2- 172.21.15.98:0/3772812430 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fbdfc066f30 0x7fbdfc0693e0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:50.058 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.036+0000 7fbe22ef8700 1 --2- 172.21.15.98:0/3772812430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe1c1426e0 0x7fbe1c159ea0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:50.058 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.036+0000 7fbe22ef8700 1 -- 172.21.15.98:0/3772812430 >> 172.21.15.98:0/3772812430 conn(0x7fbe1c06de40 msgr2=0x7fbe1c1559b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:50.058 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.037+0000 7fbe22ef8700 1 -- 172.21.15.98:0/3772812430 shutdown_connections 2023-10-06T05:07:50.058 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.037+0000 7fbe22ef8700 1 -- 172.21.15.98:0/3772812430 wait complete. 2023-10-06T05:07:50.058 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2023-10-06T05:07:50.173 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.172+0000 7fb984b6d700 1 Processor -- start 2023-10-06T05:07:50.173 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.172+0000 7fb984b6d700 1 -- start start 2023-10-06T05:07:50.175 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.173+0000 7fb984b6d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb9800b2810 0x7fb9800cc080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:50.175 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.173+0000 7fb984b6d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9800cb7a0 con 0x7fb980159440 2023-10-06T05:07:50.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.173+0000 7fb984b6d700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9800b0860 con 0x7fb9800cbc30 2023-10-06T05:07:50.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.173+0000 7fb984b6d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9800cb150 con 0x7fb9800b2810 2023-10-06T05:07:50.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.173+0000 7fb97ffff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb9800b2810 0x7fb9800cc080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:50.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.173+0000 7fb97f7fe700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fb980159440 0x7fb9800b3090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:55700/0 (socket says 172.21.15.98:55700) 2023-10-06T05:07:50.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.173+0000 7fb97f7fe700 1 -- 172.21.15.98:0/1392971538 learned_addr learned my addr 172.21.15.98:0/1392971538 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:07:50.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.173+0000 7fb97ffff700 1 -- 172.21.15.98:0/1392971538 >> v1:172.21.15.174:6789/0 conn(0x7fb9800cbc30 legacy=0x7fb9800b2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:50.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.173+0000 7fb97ffff700 1 -- 172.21.15.98:0/1392971538 >> v1:172.21.15.98:6789/0 conn(0x7fb980159440 legacy=0x7fb9800b3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:50.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.173+0000 7fb97e7fc700 1 -- 172.21.15.98:0/1392971538 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 623348489 0 0) 0x7fb9800cb7a0 con 0x7fb980159440 2023-10-06T05:07:50.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.173+0000 7fb97ffff700 1 -- 172.21.15.98:0/1392971538 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9800cb430 con 0x7fb9800b2810 2023-10-06T05:07:50.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.174+0000 7fb97ffff700 1 --2- 172.21.15.98:0/1392971538 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9800b2810 0x7fb9800cc080 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fb978009e60 tx=0x7fb978009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d30b2c11b8e7918 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:50.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.174+0000 7fb97e7fc700 1 -- 172.21.15.98:0/1392971538 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb97800e070 con 0x7fb9800b2810 2023-10-06T05:07:50.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.174+0000 7fb97e7fc700 1 -- 172.21.15.98:0/1392971538 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb97800bc30 con 0x7fb9800b2810 2023-10-06T05:07:50.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.174+0000 7fb97e7fc700 1 -- 172.21.15.98:0/1392971538 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb978012430 con 0x7fb9800b2810 2023-10-06T05:07:50.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.174+0000 7fb984b6d700 1 -- 172.21.15.98:0/1392971538 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9800b2810 msgr2=0x7fb9800cc080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:50.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.174+0000 7fb984b6d700 1 --2- 172.21.15.98:0/1392971538 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9800b2810 0x7fb9800cc080 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fb978009e60 tx=0x7fb978009780 comp rx=0 tx=0).stop 2023-10-06T05:07:50.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.174+0000 7fb984b6d700 1 -- 172.21.15.98:0/1392971538 shutdown_connections 2023-10-06T05:07:50.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.174+0000 7fb984b6d700 1 --2- 172.21.15.98:0/1392971538 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9800b2810 0x7fb9800cc080 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:50.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.174+0000 7fb984b6d700 1 -- 172.21.15.98:0/1392971538 >> 172.21.15.98:0/1392971538 conn(0x7fb98006ece0 msgr2=0x7fb98006f0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:50.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.174+0000 7fb984b6d700 1 -- 172.21.15.98:0/1392971538 shutdown_connections 2023-10-06T05:07:50.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.174+0000 7fb984b6d700 1 -- 172.21.15.98:0/1392971538 wait complete. 2023-10-06T05:07:50.180 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.175+0000 7fb984b6d700 1 Processor -- start 2023-10-06T05:07:50.180 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.176+0000 7fb984b6d700 1 -- start start 2023-10-06T05:07:50.180 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.176+0000 7fb984b6d700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb9800b2810 0x7fb9801556b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:50.180 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.176+0000 7fb984b6d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9800cbc30 0x7fb98015c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:50.180 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.176+0000 7fb984b6d700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb980159440 0x7fb98015a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:50.180 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.176+0000 7fb984b6d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb980155bf0 con 0x7fb9800cbc30 2023-10-06T05:07:50.181 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.176+0000 7fb984b6d700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9800ca070 con 0x7fb980159440 2023-10-06T05:07:50.181 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.176+0000 7fb984b6d700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9800c9460 con 0x7fb9800b2810 2023-10-06T05:07:50.181 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.176+0000 7fb97effd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9800cbc30 0x7fb98015c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:50.181 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.176+0000 7fb97effd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9800cbc30 0x7fb98015c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47378/0 (socket says 172.21.15.98:47378) 2023-10-06T05:07:50.181 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.176+0000 7fb97effd700 1 -- 172.21.15.98:0/478776654 learned_addr learned my addr 172.21.15.98:0/478776654 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:50.181 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.176+0000 7fb97f7fe700 1 --2- 172.21.15.98:0/478776654 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb9800b2810 0x7fb9801556b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:50.181 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.176+0000 7fb97ffff700 1 --2- 172.21.15.98:0/478776654 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb980159440 0x7fb98015a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:50.181 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.177+0000 7fb97effd700 1 -- 172.21.15.98:0/478776654 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb9800b2810 msgr2=0x7fb9801556b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:50.181 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.177+0000 7fb97effd700 1 --2- 172.21.15.98:0/478776654 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb9800b2810 0x7fb9801556b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:50.181 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.177+0000 7fb97effd700 1 -- 172.21.15.98:0/478776654 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb980159440 msgr2=0x7fb98015a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:50.182 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.177+0000 7fb97effd700 1 --2- 172.21.15.98:0/478776654 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb980159440 0x7fb98015a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:50.182 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.177+0000 7fb97effd700 1 -- 172.21.15.98:0/478776654 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb978009a20 con 0x7fb9800cbc30 2023-10-06T05:07:50.182 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.177+0000 7fb97f7fe700 1 --2- 172.21.15.98:0/478776654 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb9800b2810 0x7fb9801556b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:07:50.182 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.177+0000 7fb97effd700 1 --2- 172.21.15.98:0/478776654 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9800cbc30 0x7fb98015c420 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fb97400c7c0 tx=0x7fb974007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:50.182 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.177+0000 7fb9677fe700 1 -- 172.21.15.98:0/478776654 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb974011590 con 0x7fb9800cbc30 2023-10-06T05:07:50.182 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.177+0000 7fb9677fe700 1 -- 172.21.15.98:0/478776654 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb974011710 con 0x7fb9800cbc30 2023-10-06T05:07:50.182 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.177+0000 7fb984b6d700 1 -- 172.21.15.98:0/478776654 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9800c9740 con 0x7fb9800cbc30 2023-10-06T05:07:50.182 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.177+0000 7fb9677fe700 1 -- 172.21.15.98:0/478776654 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb974011ad0 con 0x7fb9800cbc30 2023-10-06T05:07:50.182 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.177+0000 7fb984b6d700 1 -- 172.21.15.98:0/478776654 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9800c9d00 con 0x7fb9800cbc30 2023-10-06T05:07:50.182 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.178+0000 7fb984b6d700 1 -- 172.21.15.98:0/478776654 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb98005f230 con 0x7fb9800cbc30 2023-10-06T05:07:50.182 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.178+0000 7fb9677fe700 1 -- 172.21.15.98:0/478776654 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fb97401b3b0 con 0x7fb9800cbc30 2023-10-06T05:07:50.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.179+0000 7fb9677fe700 1 --2- 172.21.15.98:0/478776654 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fb96c066f30 0x7fb96c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:50.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.179+0000 7fb9677fe700 1 -- 172.21.15.98:0/478776654 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5526+0+0 (secure 0 0 0) 0x7fb974083a80 con 0x7fb9800cbc30 2023-10-06T05:07:50.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.179+0000 7fb97f7fe700 1 --2- 172.21.15.98:0/478776654 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fb96c066f30 0x7fb96c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:50.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.179+0000 7fb97f7fe700 1 --2- 172.21.15.98:0/478776654 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fb96c066f30 0x7fb96c0693e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb970001890 tx=0x7fb970009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:50.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.183+0000 7fb9677fe700 1 -- 172.21.15.98:0/478776654 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb974050da0 con 0x7fb9800cbc30 2023-10-06T05:07:50.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.446+0000 7fb984b6d700 1 -- 172.21.15.98:0/478776654 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7fb980159810 con 0x7fb9800cbc30 2023-10-06T05:07:50.453 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.452+0000 7fb9677fe700 1 -- 172.21.15.98:0/478776654 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v5) v1 ==== 90+0+0 (secure 0 0 0) 0x7fb974017070 con 0x7fb9800cbc30 2023-10-06T05:07:50.455 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.454+0000 7fb984b6d700 1 -- 172.21.15.98:0/478776654 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fb96c066f30 msgr2=0x7fb96c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:50.455 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.454+0000 7fb984b6d700 1 --2- 172.21.15.98:0/478776654 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fb96c066f30 0x7fb96c0693e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb970001890 tx=0x7fb970009040 comp rx=0 tx=0).stop 2023-10-06T05:07:50.455 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.454+0000 7fb984b6d700 1 -- 172.21.15.98:0/478776654 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9800cbc30 msgr2=0x7fb98015c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:50.455 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.454+0000 7fb984b6d700 1 --2- 172.21.15.98:0/478776654 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9800cbc30 0x7fb98015c420 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fb97400c7c0 tx=0x7fb974007f60 comp rx=0 tx=0).stop 2023-10-06T05:07:50.455 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.454+0000 7fb984b6d700 1 -- 172.21.15.98:0/478776654 shutdown_connections 2023-10-06T05:07:50.455 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.454+0000 7fb984b6d700 1 --2- 172.21.15.98:0/478776654 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb980159440 0x7fb98015a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:50.456 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.454+0000 7fb984b6d700 1 --2- 172.21.15.98:0/478776654 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb9800b2810 0x7fb9801556b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:50.456 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.454+0000 7fb984b6d700 1 --2- 172.21.15.98:0/478776654 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fb96c066f30 0x7fb96c0693e0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:50.456 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.454+0000 7fb984b6d700 1 --2- 172.21.15.98:0/478776654 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9800cbc30 0x7fb98015c420 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:50.456 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.454+0000 7fb984b6d700 1 -- 172.21.15.98:0/478776654 >> 172.21.15.98:0/478776654 conn(0x7fb98006ece0 msgr2=0x7fb9800b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:50.456 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.454+0000 7fb984b6d700 1 -- 172.21.15.98:0/478776654 shutdown_connections 2023-10-06T05:07:50.456 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.454+0000 7fb984b6d700 1 -- 172.21.15.98:0/478776654 wait complete. 2023-10-06T05:07:50.467 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2023-10-06T05:07:50.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.580+0000 7f11ccdf5700 1 Processor -- start 2023-10-06T05:07:50.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.580+0000 7f11ccdf5700 1 -- start start 2023-10-06T05:07:50.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.581+0000 7f11ccdf5700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11c815db10 0x7f11c80cd290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:50.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.581+0000 7f11ccdf5700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f11c80ad5e0 0x7f11c80ca0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:50.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.581+0000 7f11ccdf5700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11c80ca600 con 0x7f11c80ad210 2023-10-06T05:07:50.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.581+0000 7f11ccdf5700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11c80cecf0 con 0x7f11c80ad5e0 2023-10-06T05:07:50.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.581+0000 7f11ccdf5700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11c80cefd0 con 0x7f11c815db10 2023-10-06T05:07:50.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.581+0000 7f11c6d9d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f11c80ad5e0 0x7f11c80ca0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:50.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.581+0000 7f11c6d9d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f11c80ad5e0 0x7f11c80ca0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47384/0 (socket says 172.21.15.98:47384) 2023-10-06T05:07:50.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.581+0000 7f11c6d9d700 1 -- 172.21.15.98:0/3884974604 learned_addr learned my addr 172.21.15.98:0/3884974604 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:50.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.582+0000 7f11c6d9d700 1 -- 172.21.15.98:0/3884974604 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11c815db10 msgr2=0x7f11c80cd290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:50.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.582+0000 7f11bdd9b700 1 --2- 172.21.15.98:0/3884974604 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11c815db10 0x7f11c80cd290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:50.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.582+0000 7f11c6d9d700 1 --2- 172.21.15.98:0/3884974604 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11c815db10 0x7f11c80cd290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:50.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.582+0000 7f11c6d9d700 1 -- 172.21.15.98:0/3884974604 >> v1:172.21.15.174:6789/0 conn(0x7f11c80ad210 legacy=0x7f11c815d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:50.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.582+0000 7f11c6d9d700 1 -- 172.21.15.98:0/3884974604 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11c80cca90 con 0x7f11c80ad5e0 2023-10-06T05:07:50.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.582+0000 7f11bdd9b700 1 --2- 172.21.15.98:0/3884974604 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11c815db10 0x7f11c80cd290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:07:50.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.582+0000 7f11c6d9d700 1 --2- 172.21.15.98:0/3884974604 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11c80ad5e0 0x7f11c80ca0c0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f11c0009f20 tx=0x7f11c0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa4c1fae4e3b46d2 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:50.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.582+0000 7f11c5d9b700 1 -- 172.21.15.98:0/3884974604 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c000e070 con 0x7f11c80ad5e0 2023-10-06T05:07:50.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.582+0000 7f11c5d9b700 1 -- 172.21.15.98:0/3884974604 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11c000ba40 con 0x7f11c80ad5e0 2023-10-06T05:07:50.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.582+0000 7f11c5d9b700 1 -- 172.21.15.98:0/3884974604 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c00092e0 con 0x7f11c80ad5e0 2023-10-06T05:07:50.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.582+0000 7f11ccdf5700 1 -- 172.21.15.98:0/3884974604 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11c80ad5e0 msgr2=0x7f11c80ca0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:50.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.582+0000 7f11ccdf5700 1 --2- 172.21.15.98:0/3884974604 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11c80ad5e0 0x7f11c80ca0c0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f11c0009f20 tx=0x7f11c0009840 comp rx=0 tx=0).stop 2023-10-06T05:07:50.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.582+0000 7f11ccdf5700 1 -- 172.21.15.98:0/3884974604 shutdown_connections 2023-10-06T05:07:50.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.582+0000 7f11ccdf5700 1 --2- 172.21.15.98:0/3884974604 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11c80ad5e0 0x7f11c80ca0c0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:50.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.582+0000 7f11ccdf5700 1 --2- 172.21.15.98:0/3884974604 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11c815db10 0x7f11c80cd290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:50.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.582+0000 7f11ccdf5700 1 -- 172.21.15.98:0/3884974604 >> 172.21.15.98:0/3884974604 conn(0x7f11c8067d90 msgr2=0x7f11c80ce270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:50.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.582+0000 7f11ccdf5700 1 -- 172.21.15.98:0/3884974604 shutdown_connections 2023-10-06T05:07:50.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.582+0000 7f11ccdf5700 1 -- 172.21.15.98:0/3884974604 wait complete. 2023-10-06T05:07:50.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.584+0000 7f11ccdf5700 1 Processor -- start 2023-10-06T05:07:50.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.584+0000 7f11ccdf5700 1 -- start start 2023-10-06T05:07:50.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.584+0000 7f11ccdf5700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11c80ad210 0x7f11c815b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:50.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.584+0000 7f11ccdf5700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11c80ad5e0 0x7f11c80b3ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:50.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.584+0000 7f11ccdf5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11c815db10 0x7f11c80b18c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:50.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.584+0000 7f11ccdf5700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11c80c1cd0 con 0x7f11c815db10 2023-10-06T05:07:50.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.584+0000 7f11ccdf5700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11c80c6bd0 con 0x7f11c80ad210 2023-10-06T05:07:50.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.584+0000 7f11ccdf5700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11c815b6a0 con 0x7f11c80ad5e0 2023-10-06T05:07:50.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.584+0000 7f11c6d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11c815db10 0x7f11c80b18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:50.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.584+0000 7f11c6d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11c815db10 0x7f11c80b18c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47392/0 (socket says 172.21.15.98:47392) 2023-10-06T05:07:50.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.585+0000 7f11c6d9d700 1 -- 172.21.15.98:0/2379681739 learned_addr learned my addr 172.21.15.98:0/2379681739 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:50.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.585+0000 7f11c6d9d700 1 -- 172.21.15.98:0/2379681739 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11c80ad5e0 msgr2=0x7f11c80b3ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:50.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.585+0000 7f11c6d9d700 1 --2- 172.21.15.98:0/2379681739 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11c80ad5e0 0x7f11c80b3ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:50.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.585+0000 7f11c6d9d700 1 -- 172.21.15.98:0/2379681739 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11c80ad210 msgr2=0x7f11c815b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:50.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.585+0000 7f11c659c700 1 --2- 172.21.15.98:0/2379681739 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11c80ad210 0x7f11c815b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:50.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.585+0000 7f11c6d9d700 1 --2- 172.21.15.98:0/2379681739 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11c80ad210 0x7f11c815b160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:50.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.585+0000 7f11c6d9d700 1 -- 172.21.15.98:0/2379681739 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11c0009ae0 con 0x7f11c815db10 2023-10-06T05:07:50.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.585+0000 7f11c659c700 1 --2- 172.21.15.98:0/2379681739 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11c80ad210 0x7f11c815b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:07:50.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.585+0000 7f11c6d9d700 1 --2- 172.21.15.98:0/2379681739 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11c815db10 0x7f11c80b18c0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f11c8070480 tx=0x7f11c0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:50.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.585+0000 7f11beffd700 1 -- 172.21.15.98:0/2379681739 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c0012e60 con 0x7f11c815db10 2023-10-06T05:07:50.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.585+0000 7f11beffd700 1 -- 172.21.15.98:0/2379681739 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11c00092e0 con 0x7f11c815db10 2023-10-06T05:07:50.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.585+0000 7f11ccdf5700 1 -- 172.21.15.98:0/2379681739 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11c80c5fc0 con 0x7f11c815db10 2023-10-06T05:07:50.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.585+0000 7f11beffd700 1 -- 172.21.15.98:0/2379681739 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c001e660 con 0x7f11c815db10 2023-10-06T05:07:50.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.585+0000 7f11ccdf5700 1 -- 172.21.15.98:0/2379681739 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11c80c6580 con 0x7f11c815db10 2023-10-06T05:07:50.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.586+0000 7f11ccdf5700 1 -- 172.21.15.98:0/2379681739 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11c80b0bf0 con 0x7f11c815db10 2023-10-06T05:07:50.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.587+0000 7f11beffd700 1 -- 172.21.15.98:0/2379681739 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f11c001e9d0 con 0x7f11c815db10 2023-10-06T05:07:50.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.587+0000 7f11beffd700 1 --2- 172.21.15.98:0/2379681739 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f11a8066f30 0x7f11a80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:50.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.587+0000 7f11beffd700 1 -- 172.21.15.98:0/2379681739 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5526+0+0 (secure 0 0 0) 0x7f11c00860a0 con 0x7f11c815db10 2023-10-06T05:07:50.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.587+0000 7f11c659c700 1 --2- 172.21.15.98:0/2379681739 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f11a8066f30 0x7f11a80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:50.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.587+0000 7f11c659c700 1 --2- 172.21.15.98:0/2379681739 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f11a8066f30 0x7f11a80693e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f11b80028b0 tx=0x7f11b8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:50.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.591+0000 7f11beffd700 1 -- 172.21.15.98:0/2379681739 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f11c0053470 con 0x7f11c815db10 2023-10-06T05:07:50.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.848+0000 7f11ccdf5700 1 -- 172.21.15.98:0/2379681739 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7f11c80c6d40 con 0x7f11c815db10 2023-10-06T05:07:50.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.853+0000 7f11beffd700 1 -- 172.21.15.98:0/2379681739 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v6) v1 ==== 90+0+0 (secure 0 0 0) 0x7f11c0056a90 con 0x7f11c815db10 2023-10-06T05:07:50.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.855+0000 7f11ccdf5700 1 -- 172.21.15.98:0/2379681739 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f11a8066f30 msgr2=0x7f11a80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:50.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.855+0000 7f11ccdf5700 1 --2- 172.21.15.98:0/2379681739 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f11a8066f30 0x7f11a80693e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f11b80028b0 tx=0x7f11b8009040 comp rx=0 tx=0).stop 2023-10-06T05:07:50.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.855+0000 7f11ccdf5700 1 -- 172.21.15.98:0/2379681739 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11c815db10 msgr2=0x7f11c80b18c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:50.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.855+0000 7f11ccdf5700 1 --2- 172.21.15.98:0/2379681739 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11c815db10 0x7f11c80b18c0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f11c8070480 tx=0x7f11c0017000 comp rx=0 tx=0).stop 2023-10-06T05:07:50.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.855+0000 7f11ccdf5700 1 -- 172.21.15.98:0/2379681739 shutdown_connections 2023-10-06T05:07:50.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.855+0000 7f11ccdf5700 1 --2- 172.21.15.98:0/2379681739 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f11a8066f30 0x7f11a80693e0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:50.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.855+0000 7f11ccdf5700 1 --2- 172.21.15.98:0/2379681739 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11c815db10 0x7f11c80b18c0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:50.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.855+0000 7f11ccdf5700 1 --2- 172.21.15.98:0/2379681739 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11c80ad210 0x7f11c815b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:50.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.855+0000 7f11ccdf5700 1 --2- 172.21.15.98:0/2379681739 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11c80ad5e0 0x7f11c80b3ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:50.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.855+0000 7f11ccdf5700 1 -- 172.21.15.98:0/2379681739 >> 172.21.15.98:0/2379681739 conn(0x7f11c8067d90 msgr2=0x7f11c80ce060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:50.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.856+0000 7f11ccdf5700 1 -- 172.21.15.98:0/2379681739 shutdown_connections 2023-10-06T05:07:50.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.856+0000 7f11ccdf5700 1 -- 172.21.15.98:0/2379681739 wait complete. 2023-10-06T05:07:50.869 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2023-10-06T05:07:50.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.983+0000 7f884b59e700 1 Processor -- start 2023-10-06T05:07:50.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.983+0000 7f884b59e700 1 -- start start 2023-10-06T05:07:50.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.984+0000 7f884b59e700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8844159560 0x7f88440aa530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:50.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.984+0000 7f884b59e700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f88440b49a0 0x7f88440aaa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:50.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.984+0000 7f884b59e700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88440b0c60 con 0x7f88440b0760 2023-10-06T05:07:50.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.984+0000 7f884b59e700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88440b0f40 con 0x7f88440b49a0 2023-10-06T05:07:50.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.984+0000 7f884b59e700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88440b1220 con 0x7f8844159560 2023-10-06T05:07:50.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.984+0000 7f8848b39700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f88440b49a0 0x7f88440aaa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:50.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.984+0000 7f8848b39700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f88440b49a0 0x7f88440aaa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47394/0 (socket says 172.21.15.98:47394) 2023-10-06T05:07:50.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.984+0000 7f8848b39700 1 -- 172.21.15.98:0/3309701125 learned_addr learned my addr 172.21.15.98:0/3309701125 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:50.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.984+0000 7f8848b39700 1 -- 172.21.15.98:0/3309701125 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8844159560 msgr2=0x7f88440aa530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:50.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.984+0000 7f884933a700 1 --2- 172.21.15.98:0/3309701125 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8844159560 0x7f88440aa530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:50.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.984+0000 7f8848b39700 1 --2- 172.21.15.98:0/3309701125 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8844159560 0x7f88440aa530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:50.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.984+0000 7f8848b39700 1 -- 172.21.15.98:0/3309701125 >> v1:172.21.15.174:6789/0 conn(0x7f88440b0760 legacy=0x7f88440b3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:50.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.984+0000 7f8848b39700 1 -- 172.21.15.98:0/3309701125 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88440b1500 con 0x7f88440b49a0 2023-10-06T05:07:50.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.984+0000 7f884933a700 1 --2- 172.21.15.98:0/3309701125 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8844159560 0x7f88440aa530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:07:50.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.985+0000 7f8848b39700 1 --2- 172.21.15.98:0/3309701125 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88440b49a0 0x7f88440aaa70 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f88340099b0 tx=0x7f8834009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb020d557b6e5e69 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:50.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.985+0000 7f883bfff700 1 -- 172.21.15.98:0/3309701125 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f883400b820 con 0x7f88440b49a0 2023-10-06T05:07:50.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.985+0000 7f883bfff700 1 -- 172.21.15.98:0/3309701125 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f883400b980 con 0x7f88440b49a0 2023-10-06T05:07:50.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.985+0000 7f883bfff700 1 -- 172.21.15.98:0/3309701125 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f883400bd20 con 0x7f88440b49a0 2023-10-06T05:07:50.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.985+0000 7f884b59e700 1 -- 172.21.15.98:0/3309701125 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88440b49a0 msgr2=0x7f88440aaa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:50.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.985+0000 7f884b59e700 1 --2- 172.21.15.98:0/3309701125 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88440b49a0 0x7f88440aaa70 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f88340099b0 tx=0x7f8834009250 comp rx=0 tx=0).stop 2023-10-06T05:07:50.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.985+0000 7f884b59e700 1 -- 172.21.15.98:0/3309701125 shutdown_connections 2023-10-06T05:07:50.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.985+0000 7f884b59e700 1 --2- 172.21.15.98:0/3309701125 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8844159560 0x7f88440aa530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:50.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.985+0000 7f884b59e700 1 --2- 172.21.15.98:0/3309701125 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88440b49a0 0x7f88440aaa70 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:50.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.985+0000 7f884b59e700 1 -- 172.21.15.98:0/3309701125 >> 172.21.15.98:0/3309701125 conn(0x7f88440997f0 msgr2=0x7f884406f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:50.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.985+0000 7f884b59e700 1 -- 172.21.15.98:0/3309701125 shutdown_connections 2023-10-06T05:07:50.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.985+0000 7f884b59e700 1 -- 172.21.15.98:0/3309701125 wait complete. 2023-10-06T05:07:50.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.986+0000 7f884b59e700 1 Processor -- start 2023-10-06T05:07:50.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.986+0000 7f884b59e700 1 -- start start 2023-10-06T05:07:50.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.987+0000 7f884b59e700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f88440b0760 0x7f884409d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:50.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.987+0000 7f884b59e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88440b49a0 0x7f88440cde40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:50.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.987+0000 7f884b59e700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8844159560 0x7f88440ce380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:50.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.987+0000 7f884b59e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f884416aaf0 con 0x7f88440b49a0 2023-10-06T05:07:50.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.987+0000 7f884b59e700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88440ce8c0 con 0x7f8844159560 2023-10-06T05:07:50.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.987+0000 7f884b59e700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88440ceb70 con 0x7f88440b0760 2023-10-06T05:07:50.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.987+0000 7f8848b39700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88440b49a0 0x7f88440cde40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:50.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.987+0000 7f8848b39700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88440b49a0 0x7f88440cde40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47396/0 (socket says 172.21.15.98:47396) 2023-10-06T05:07:50.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.987+0000 7f8848b39700 1 -- 172.21.15.98:0/1077870920 learned_addr learned my addr 172.21.15.98:0/1077870920 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:50.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.988+0000 7f884933a700 1 --2- 172.21.15.98:0/1077870920 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f88440b0760 0x7f884409d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:50.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.988+0000 7f8849b3b700 1 --2- 172.21.15.98:0/1077870920 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8844159560 0x7f88440ce380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:50.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.988+0000 7f8848b39700 1 -- 172.21.15.98:0/1077870920 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f88440b0760 msgr2=0x7f884409d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:50.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.988+0000 7f8848b39700 1 --2- 172.21.15.98:0/1077870920 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f88440b0760 0x7f884409d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:50.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.988+0000 7f8848b39700 1 -- 172.21.15.98:0/1077870920 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8844159560 msgr2=0x7f88440ce380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:50.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.988+0000 7f8848b39700 1 --2- 172.21.15.98:0/1077870920 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8844159560 0x7f88440ce380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:50.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.988+0000 7f8848b39700 1 -- 172.21.15.98:0/1077870920 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88340094f0 con 0x7f88440b49a0 2023-10-06T05:07:50.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.988+0000 7f8848b39700 1 --2- 172.21.15.98:0/1077870920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88440b49a0 0x7f88440cde40 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f8834009290 tx=0x7f8834016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:50.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.988+0000 7f88397fa700 1 -- 172.21.15.98:0/1077870920 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8834011c10 con 0x7f88440b49a0 2023-10-06T05:07:50.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.988+0000 7f88397fa700 1 -- 172.21.15.98:0/1077870920 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8834020070 con 0x7f88440b49a0 2023-10-06T05:07:50.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.988+0000 7f884b59e700 1 -- 172.21.15.98:0/1077870920 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88440cee50 con 0x7f88440b49a0 2023-10-06T05:07:50.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.988+0000 7f88397fa700 1 -- 172.21.15.98:0/1077870920 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f883401b450 con 0x7f88440b49a0 2023-10-06T05:07:50.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.988+0000 7f884b59e700 1 -- 172.21.15.98:0/1077870920 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88440cf410 con 0x7f88440b49a0 2023-10-06T05:07:50.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.989+0000 7f884b59e700 1 -- 172.21.15.98:0/1077870920 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f884416aca0 con 0x7f88440b49a0 2023-10-06T05:07:50.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.990+0000 7f88397fa700 1 -- 172.21.15.98:0/1077870920 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f8834022030 con 0x7f88440b49a0 2023-10-06T05:07:50.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.990+0000 7f88397fa700 1 --2- 172.21.15.98:0/1077870920 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f8828066f30 0x7f88280693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:50.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.990+0000 7f88397fa700 1 -- 172.21.15.98:0/1077870920 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5526+0+0 (secure 0 0 0) 0x7f8834086310 con 0x7f88440b49a0 2023-10-06T05:07:50.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.990+0000 7f884933a700 1 --2- 172.21.15.98:0/1077870920 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f8828066f30 0x7f88280693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:50.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.990+0000 7f884933a700 1 --2- 172.21.15.98:0/1077870920 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f8828066f30 0x7f88280693e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f88440af710 tx=0x7f8840008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:50.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:50.994+0000 7f88397fa700 1 -- 172.21.15.98:0/1077870920 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f884416aca0 con 0x7f88440b49a0 2023-10-06T05:07:51.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.255+0000 7f884b59e700 1 -- 172.21.15.98:0/1077870920 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7f884416aca0 con 0x7f88440b49a0 2023-10-06T05:07:51.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.261+0000 7f88397fa700 1 -- 172.21.15.98:0/1077870920 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v7) v1 ==== 90+0+0 (secure 0 0 0) 0x7f884416aca0 con 0x7f88440b49a0 2023-10-06T05:07:51.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.263+0000 7f884b59e700 1 -- 172.21.15.98:0/1077870920 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f8828066f30 msgr2=0x7f88280693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:51.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.263+0000 7f884b59e700 1 --2- 172.21.15.98:0/1077870920 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f8828066f30 0x7f88280693e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f88440af710 tx=0x7f8840008040 comp rx=0 tx=0).stop 2023-10-06T05:07:51.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.263+0000 7f884b59e700 1 -- 172.21.15.98:0/1077870920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88440b49a0 msgr2=0x7f88440cde40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:51.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.263+0000 7f884b59e700 1 --2- 172.21.15.98:0/1077870920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88440b49a0 0x7f88440cde40 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f8834009290 tx=0x7f8834016000 comp rx=0 tx=0).stop 2023-10-06T05:07:51.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.263+0000 7f884b59e700 1 -- 172.21.15.98:0/1077870920 shutdown_connections 2023-10-06T05:07:51.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.263+0000 7f884b59e700 1 --2- 172.21.15.98:0/1077870920 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8844159560 0x7f88440ce380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:51.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.263+0000 7f884b59e700 1 --2- 172.21.15.98:0/1077870920 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f88440b0760 0x7f884409d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:51.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.263+0000 7f884b59e700 1 --2- 172.21.15.98:0/1077870920 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f8828066f30 0x7f88280693e0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:51.267 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.263+0000 7f884b59e700 1 --2- 172.21.15.98:0/1077870920 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88440b49a0 0x7f88440cde40 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:51.267 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.263+0000 7f884b59e700 1 -- 172.21.15.98:0/1077870920 >> 172.21.15.98:0/1077870920 conn(0x7f88440997f0 msgr2=0x7f88440afce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:51.267 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.264+0000 7f884b59e700 1 -- 172.21.15.98:0/1077870920 shutdown_connections 2023-10-06T05:07:51.267 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.264+0000 7f884b59e700 1 -- 172.21.15.98:0/1077870920 wait complete. 2023-10-06T05:07:51.278 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2023-10-06T05:07:51.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.390+0000 7f31629f3700 1 Processor -- start 2023-10-06T05:07:51.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.390+0000 7f31629f3700 1 -- start start 2023-10-06T05:07:51.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.391+0000 7f31629f3700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f315c159560 0x7f315c0aa530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:51.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.391+0000 7f31629f3700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f315c0b0760 0x7f315c0b1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:51.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.391+0000 7f31629f3700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f315c0b0c60 con 0x7f315c0b49a0 2023-10-06T05:07:51.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.391+0000 7f31629f3700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f315c0b0f40 con 0x7f315c0b0760 2023-10-06T05:07:51.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.391+0000 7f31629f3700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f315c0b1220 con 0x7f315c159560 2023-10-06T05:07:51.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.391+0000 7f3160f90700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f315c0b0760 0x7f315c0b1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:51.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.391+0000 7f3160f90700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f315c0b0760 0x7f315c0b1bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47410/0 (socket says 172.21.15.98:47410) 2023-10-06T05:07:51.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.391+0000 7f3160f90700 1 -- 172.21.15.98:0/3136620060 learned_addr learned my addr 172.21.15.98:0/3136620060 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:51.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.392+0000 7f315bfff700 1 --2- 172.21.15.98:0/3136620060 >> v2:172.21.15.174:3300/0 conn(0x7f315c159560 0x7f315c0aa530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:51.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.392+0000 7f3160f90700 1 -- 172.21.15.98:0/3136620060 >> v2:172.21.15.174:3300/0 conn(0x7f315c159560 msgr2=0x7f315c0aa530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:51.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.392+0000 7f3160f90700 1 --2- 172.21.15.98:0/3136620060 >> v2:172.21.15.174:3300/0 conn(0x7f315c159560 0x7f315c0aa530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:51.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.392+0000 7f3160f90700 1 -- 172.21.15.98:0/3136620060 >> v1:172.21.15.174:6789/0 conn(0x7f315c0b49a0 legacy=0x7f315c0aaa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:51.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.392+0000 7f3160f90700 1 -- 172.21.15.98:0/3136620060 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f315c0b1500 con 0x7f315c0b0760 2023-10-06T05:07:51.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.392+0000 7f3160f90700 1 --2- 172.21.15.98:0/3136620060 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f315c0b0760 0x7f315c0b1bc0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f3154009f20 tx=0x7f3154009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b91eb124b787eec6 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:51.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.392+0000 7f315affd700 1 -- 172.21.15.98:0/3136620060 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f315400e070 con 0x7f315c0b0760 2023-10-06T05:07:51.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.392+0000 7f315affd700 1 -- 172.21.15.98:0/3136620060 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f315400ba40 con 0x7f315c0b0760 2023-10-06T05:07:51.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.392+0000 7f315affd700 1 -- 172.21.15.98:0/3136620060 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31540092e0 con 0x7f315c0b0760 2023-10-06T05:07:51.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.393+0000 7f31629f3700 1 -- 172.21.15.98:0/3136620060 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f315c0b0760 msgr2=0x7f315c0b1bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:51.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.393+0000 7f31629f3700 1 --2- 172.21.15.98:0/3136620060 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f315c0b0760 0x7f315c0b1bc0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f3154009f20 tx=0x7f3154009840 comp rx=0 tx=0).stop 2023-10-06T05:07:51.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.393+0000 7f31629f3700 1 -- 172.21.15.98:0/3136620060 shutdown_connections 2023-10-06T05:07:51.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.393+0000 7f31629f3700 1 --2- 172.21.15.98:0/3136620060 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f315c0b0760 0x7f315c0b1bc0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:51.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.393+0000 7f31629f3700 1 --2- 172.21.15.98:0/3136620060 >> v2:172.21.15.174:3300/0 conn(0x7f315c159560 0x7f315c0aa530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:51.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.393+0000 7f31629f3700 1 -- 172.21.15.98:0/3136620060 >> 172.21.15.98:0/3136620060 conn(0x7f315c0997f0 msgr2=0x7f315c06f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:51.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.393+0000 7f31629f3700 1 -- 172.21.15.98:0/3136620060 shutdown_connections 2023-10-06T05:07:51.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.393+0000 7f31629f3700 1 -- 172.21.15.98:0/3136620060 wait complete. 2023-10-06T05:07:51.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.394+0000 7f31629f3700 1 Processor -- start 2023-10-06T05:07:51.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.394+0000 7f31629f3700 1 -- start start 2023-10-06T05:07:51.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.395+0000 7f31629f3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f315c0b0760 0x7f315c158760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:51.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.395+0000 7f31629f3700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f315c0b49a0 0x7f315c169af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:51.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.395+0000 7f31629f3700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f315c159560 0x7f315c09aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:51.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.395+0000 7f31629f3700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f315c159bf0 con 0x7f315c0b0760 2023-10-06T05:07:51.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.395+0000 7f31629f3700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f315c159e40 con 0x7f315c0b49a0 2023-10-06T05:07:51.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.395+0000 7f31629f3700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f315c15a120 con 0x7f315c159560 2023-10-06T05:07:51.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.395+0000 7f315bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f315c0b0760 0x7f315c158760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:51.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.395+0000 7f315bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f315c0b0760 0x7f315c158760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47424/0 (socket says 172.21.15.98:47424) 2023-10-06T05:07:51.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.395+0000 7f315bfff700 1 -- 172.21.15.98:0/962978504 learned_addr learned my addr 172.21.15.98:0/962978504 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:51.404 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.395+0000 7f315b7fe700 1 --2- 172.21.15.98:0/962978504 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f315c0b49a0 0x7f315c169af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:51.404 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.395+0000 7f3160f90700 1 --2- 172.21.15.98:0/962978504 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f315c159560 0x7f315c09aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:51.404 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.395+0000 7f315bfff700 1 -- 172.21.15.98:0/962978504 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f315c159560 msgr2=0x7f315c09aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:51.404 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.395+0000 7f315bfff700 1 --2- 172.21.15.98:0/962978504 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f315c159560 0x7f315c09aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:51.404 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.395+0000 7f315bfff700 1 -- 172.21.15.98:0/962978504 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f315c0b49a0 msgr2=0x7f315c169af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:51.404 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.395+0000 7f315bfff700 1 --2- 172.21.15.98:0/962978504 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f315c0b49a0 0x7f315c169af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:51.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.395+0000 7f315bfff700 1 -- 172.21.15.98:0/962978504 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3154009ae0 con 0x7f315c0b0760 2023-10-06T05:07:51.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.395+0000 7f315b7fe700 1 --2- 172.21.15.98:0/962978504 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f315c0b49a0 0x7f315c169af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:07:51.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.395+0000 7f3160f90700 1 --2- 172.21.15.98:0/962978504 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f315c159560 0x7f315c09aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:07:51.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.395+0000 7f315bfff700 1 --2- 172.21.15.98:0/962978504 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f315c0b0760 0x7f315c158760 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f314c00d890 tx=0x7f314c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:51.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.395+0000 7f3143fff700 1 -- 172.21.15.98:0/962978504 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f314c018070 con 0x7f315c0b0760 2023-10-06T05:07:51.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.395+0000 7f3143fff700 1 -- 172.21.15.98:0/962978504 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f314c005a00 con 0x7f315c0b0760 2023-10-06T05:07:51.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.395+0000 7f31629f3700 1 -- 172.21.15.98:0/962978504 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f315c15a400 con 0x7f315c0b0760 2023-10-06T05:07:51.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.395+0000 7f3143fff700 1 -- 172.21.15.98:0/962978504 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f314c005dc0 con 0x7f315c0b0760 2023-10-06T05:07:51.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.395+0000 7f31629f3700 1 -- 172.21.15.98:0/962978504 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f315c158d20 con 0x7f315c0b0760 2023-10-06T05:07:51.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.396+0000 7f31629f3700 1 -- 172.21.15.98:0/962978504 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f315c15a920 con 0x7f315c0b0760 2023-10-06T05:07:51.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.397+0000 7f3143fff700 1 -- 172.21.15.98:0/962978504 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f314c012800 con 0x7f315c0b0760 2023-10-06T05:07:51.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.397+0000 7f3143fff700 1 --2- 172.21.15.98:0/962978504 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f3148066f30 0x7f31480693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:51.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.397+0000 7f3143fff700 1 -- 172.21.15.98:0/962978504 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5526+0+0 (secure 0 0 0) 0x7f314c0845b0 con 0x7f315c0b0760 2023-10-06T05:07:51.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.397+0000 7f315b7fe700 1 --2- 172.21.15.98:0/962978504 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f3148066f30 0x7f31480693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:51.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.397+0000 7f315b7fe700 1 --2- 172.21.15.98:0/962978504 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f3148066f30 0x7f31480693e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f3150007fd0 tx=0x7f3150009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:51.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.401+0000 7f3143fff700 1 -- 172.21.15.98:0/962978504 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f315c15a920 con 0x7f315c0b0760 2023-10-06T05:07:51.660 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.658+0000 7f31629f3700 1 -- 172.21.15.98:0/962978504 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7f315c0b5660 con 0x7f315c0b0760 2023-10-06T05:07:51.665 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.663+0000 7f3143fff700 1 -- 172.21.15.98:0/962978504 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v8) v1 ==== 90+0+0 (secure 0 0 0) 0x7f314c055360 con 0x7f315c0b0760 2023-10-06T05:07:51.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.665+0000 7f31629f3700 1 -- 172.21.15.98:0/962978504 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f3148066f30 msgr2=0x7f31480693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:51.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.665+0000 7f31629f3700 1 --2- 172.21.15.98:0/962978504 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f3148066f30 0x7f31480693e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f3150007fd0 tx=0x7f3150009040 comp rx=0 tx=0).stop 2023-10-06T05:07:51.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.665+0000 7f31629f3700 1 -- 172.21.15.98:0/962978504 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f315c0b0760 msgr2=0x7f315c158760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:51.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.665+0000 7f31629f3700 1 --2- 172.21.15.98:0/962978504 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f315c0b0760 0x7f315c158760 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f314c00d890 tx=0x7f314c008040 comp rx=0 tx=0).stop 2023-10-06T05:07:51.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.665+0000 7f31629f3700 1 -- 172.21.15.98:0/962978504 shutdown_connections 2023-10-06T05:07:51.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.665+0000 7f31629f3700 1 --2- 172.21.15.98:0/962978504 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f315c159560 0x7f315c09aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:51.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.665+0000 7f31629f3700 1 --2- 172.21.15.98:0/962978504 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f3148066f30 0x7f31480693e0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:51.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.665+0000 7f31629f3700 1 --2- 172.21.15.98:0/962978504 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f315c0b0760 0x7f315c158760 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:51.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.665+0000 7f31629f3700 1 --2- 172.21.15.98:0/962978504 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f315c0b49a0 0x7f315c169af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:51.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.665+0000 7f31629f3700 1 -- 172.21.15.98:0/962978504 >> 172.21.15.98:0/962978504 conn(0x7f315c0997f0 msgr2=0x7f315c0b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:51.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.666+0000 7f31629f3700 1 -- 172.21.15.98:0/962978504 shutdown_connections 2023-10-06T05:07:51.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.666+0000 7f31629f3700 1 -- 172.21.15.98:0/962978504 wait complete. 2023-10-06T05:07:51.680 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2023-10-06T05:07:51.794 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.792+0000 7f44cfe4d700 1 Processor -- start 2023-10-06T05:07:51.794 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.792+0000 7f44cfe4d700 1 -- start start 2023-10-06T05:07:51.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.793+0000 7f44cfe4d700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f44c8159560 0x7f44c80aa530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:51.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.793+0000 7f44cfe4d700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f44c80b49a0 0x7f44c80aaa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:51.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.793+0000 7f44cfe4d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44c80b0c60 con 0x7f44c80b0760 2023-10-06T05:07:51.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.793+0000 7f44cfe4d700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44c80b0f40 con 0x7f44c80b49a0 2023-10-06T05:07:51.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.793+0000 7f44cfe4d700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44c80b1220 con 0x7f44c8159560 2023-10-06T05:07:51.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.793+0000 7f44ce3ea700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f44c80b0760 0x7f44c80b3890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:55714/0 (socket says 172.21.15.98:55714) 2023-10-06T05:07:51.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.793+0000 7f44ce3ea700 1 -- 172.21.15.98:0/2833583245 learned_addr learned my addr 172.21.15.98:0/2833583245 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:07:51.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.793+0000 7f44cdbe9700 1 --2- 172.21.15.98:0/2833583245 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f44c8159560 0x7f44c80aa530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:51.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.793+0000 7f44cd3e8700 1 --2- 172.21.15.98:0/2833583245 >> v2:172.21.15.174:3300/0 conn(0x7f44c80b49a0 0x7f44c80aaa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:51.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.793+0000 7f44ccbe7700 1 -- 172.21.15.98:0/2833583245 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4192055944 0 0) 0x7f44c80b0c60 con 0x7f44c80b0760 2023-10-06T05:07:51.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.794+0000 7f44ccbe7700 1 -- 172.21.15.98:0/2833583245 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44b4003570 con 0x7f44c80b0760 2023-10-06T05:07:51.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.794+0000 7f44ccbe7700 1 -- 172.21.15.98:0/2833583245 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1042114631 0 0) 0x7f44b4003570 con 0x7f44c80b0760 2023-10-06T05:07:51.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.794+0000 7f44ccbe7700 1 -- 172.21.15.98:0/2833583245 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f44c8159560 msgr2=0x7f44c80aa530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:51.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.794+0000 7f44ccbe7700 1 --2- 172.21.15.98:0/2833583245 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f44c8159560 0x7f44c80aa530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:51.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.794+0000 7f44ccbe7700 1 -- 172.21.15.98:0/2833583245 >> v2:172.21.15.174:3300/0 conn(0x7f44c80b49a0 msgr2=0x7f44c80aaa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:51.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.794+0000 7f44ccbe7700 1 --2- 172.21.15.98:0/2833583245 >> v2:172.21.15.174:3300/0 conn(0x7f44c80b49a0 0x7f44c80aaa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:51.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.794+0000 7f44ccbe7700 1 -- 172.21.15.98:0/2833583245 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44c80b1500 con 0x7f44c80b0760 2023-10-06T05:07:51.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.794+0000 7f44ccbe7700 1 -- 172.21.15.98:0/2833583245 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1543991876 0 0) 0x7f44c0001470 con 0x7f44c80b0760 2023-10-06T05:07:51.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.794+0000 7f44ccbe7700 1 -- 172.21.15.98:0/2833583245 >> v1:172.21.15.98:6789/0 conn(0x7f44c80b0760 legacy=0x7f44c80b3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:51.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.794+0000 7f44ccbe7700 1 --2- 172.21.15.98:0/2833583245 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f44c8159560 0x7f44b4003a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:51.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.794+0000 7f44ccbe7700 1 --2- 172.21.15.98:0/2833583245 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f44b4003fa0 0x7f44b4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:51.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.794+0000 7f44ccbe7700 1 --2- 172.21.15.98:0/2833583245 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44b4004d50 0x7f44b40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:51.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.794+0000 7f44ccbe7700 1 -- 172.21.15.98:0/2833583245 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c80b0f40 con 0x7f44b4004d50 2023-10-06T05:07:51.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.794+0000 7f44ccbe7700 1 -- 172.21.15.98:0/2833583245 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c80b1220 con 0x7f44b4003fa0 2023-10-06T05:07:51.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.794+0000 7f44ccbe7700 1 -- 172.21.15.98:0/2833583245 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44b400ba60 con 0x7f44c8159560 2023-10-06T05:07:51.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.794+0000 7f44ce3ea700 1 --2- 172.21.15.98:0/2833583245 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44b4004d50 0x7f44b40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:51.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.794+0000 7f44cdbe9700 1 --2- 172.21.15.98:0/2833583245 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f44c8159560 0x7f44b4003a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:51.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.794+0000 7f44cd3e8700 1 --2- 172.21.15.98:0/2833583245 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f44b4003fa0 0x7f44b4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:51.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.794+0000 7f44ce3ea700 1 -- 172.21.15.98:0/2833583245 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f44c8159560 msgr2=0x7f44b4003a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:51.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.794+0000 7f44ce3ea700 1 --2- 172.21.15.98:0/2833583245 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f44c8159560 0x7f44b4003a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:51.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.794+0000 7f44ce3ea700 1 -- 172.21.15.98:0/2833583245 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f44b4003fa0 msgr2=0x7f44b4004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:51.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.794+0000 7f44ce3ea700 1 --2- 172.21.15.98:0/2833583245 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f44b4003fa0 0x7f44b4004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:51.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.794+0000 7f44ce3ea700 1 -- 172.21.15.98:0/2833583245 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44b4003570 con 0x7f44b4004d50 2023-10-06T05:07:51.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.794+0000 7f44ce3ea700 1 --2- 172.21.15.98:0/2833583245 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44b4004d50 0x7f44b40051c0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f44c00051e0 tx=0x7f44c0008c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cb81f3a8536e61 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:51.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.795+0000 7f44ccbe7700 1 -- 172.21.15.98:0/2833583245 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44c0005920 con 0x7f44b4004d50 2023-10-06T05:07:51.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.795+0000 7f44ccbe7700 1 -- 172.21.15.98:0/2833583245 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44c0005a80 con 0x7f44b4004d50 2023-10-06T05:07:51.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.795+0000 7f44ccbe7700 1 -- 172.21.15.98:0/2833583245 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44c0005e20 con 0x7f44b4004d50 2023-10-06T05:07:51.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.795+0000 7f44cfe4d700 1 -- 172.21.15.98:0/2833583245 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44b4004d50 msgr2=0x7f44b40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:51.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.795+0000 7f44cfe4d700 1 --2- 172.21.15.98:0/2833583245 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44b4004d50 0x7f44b40051c0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f44c00051e0 tx=0x7f44c0008c40 comp rx=0 tx=0).stop 2023-10-06T05:07:51.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.795+0000 7f44ce3ea700 1 -- 172.21.15.98:0/2833583245 reap_dead start 2023-10-06T05:07:51.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.795+0000 7f44cfe4d700 1 -- 172.21.15.98:0/2833583245 shutdown_connections 2023-10-06T05:07:51.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.795+0000 7f44cfe4d700 1 -- 172.21.15.98:0/2833583245 >> 172.21.15.98:0/2833583245 conn(0x7f44c80997f0 msgr2=0x7f44c806f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:51.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.795+0000 7f44cfe4d700 1 -- 172.21.15.98:0/2833583245 shutdown_connections 2023-10-06T05:07:51.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.795+0000 7f44cfe4d700 1 -- 172.21.15.98:0/2833583245 wait complete. 2023-10-06T05:07:51.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.796+0000 7f44cfe4d700 1 Processor -- start 2023-10-06T05:07:51.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.796+0000 7f44cfe4d700 1 -- start start 2023-10-06T05:07:51.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.797+0000 7f44cfe4d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44b4004d50 0x7f44c8155a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:51.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.797+0000 7f44cfe4d700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f44c815a2f0 0x7f44c8154900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:51.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.797+0000 7f44cfe4d700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f44c815f0a0 0x7f44c815f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:51.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.797+0000 7f44cfe4d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c815fa30 con 0x7f44b4004d50 2023-10-06T05:07:51.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.797+0000 7f44cfe4d700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c81694c0 con 0x7f44c815f0a0 2023-10-06T05:07:51.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.797+0000 7f44cfe4d700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44c81560d0 con 0x7f44c815a2f0 2023-10-06T05:07:51.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.797+0000 7f44cdbe9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44b4004d50 0x7f44c8155a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:51.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.797+0000 7f44cdbe9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44b4004d50 0x7f44c8155a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47440/0 (socket says 172.21.15.98:47440) 2023-10-06T05:07:51.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.797+0000 7f44cdbe9700 1 -- 172.21.15.98:0/2633240722 learned_addr learned my addr 172.21.15.98:0/2633240722 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:51.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.797+0000 7f44cd3e8700 1 --2- 172.21.15.98:0/2633240722 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f44c815a2f0 0x7f44c8154900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:51.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.797+0000 7f44ce3ea700 1 --2- 172.21.15.98:0/2633240722 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f44c815f0a0 0x7f44c815f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:51.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.797+0000 7f44cdbe9700 1 -- 172.21.15.98:0/2633240722 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f44c815a2f0 msgr2=0x7f44c8154900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:51.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.797+0000 7f44cdbe9700 1 --2- 172.21.15.98:0/2633240722 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f44c815a2f0 0x7f44c8154900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:51.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.797+0000 7f44cdbe9700 1 -- 172.21.15.98:0/2633240722 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f44c815f0a0 msgr2=0x7f44c815f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:51.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.797+0000 7f44cdbe9700 1 --2- 172.21.15.98:0/2633240722 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f44c815f0a0 0x7f44c815f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:51.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.797+0000 7f44cdbe9700 1 -- 172.21.15.98:0/2633240722 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44c81563b0 con 0x7f44b4004d50 2023-10-06T05:07:51.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.797+0000 7f44cd3e8700 1 --2- 172.21.15.98:0/2633240722 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f44c815a2f0 0x7f44c8154900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:07:51.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.797+0000 7f44ce3ea700 1 --2- 172.21.15.98:0/2633240722 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f44c815f0a0 0x7f44c815f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:07:51.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.798+0000 7f44cdbe9700 1 --2- 172.21.15.98:0/2633240722 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44b4004d50 0x7f44c8155a80 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f44c4011f50 tx=0x7f44c400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:51.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.798+0000 7f44bdffb700 1 -- 172.21.15.98:0/2633240722 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44c400e040 con 0x7f44b4004d50 2023-10-06T05:07:51.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.798+0000 7f44bdffb700 1 -- 172.21.15.98:0/2633240722 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44c400f040 con 0x7f44b4004d50 2023-10-06T05:07:51.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.798+0000 7f44cfe4d700 1 -- 172.21.15.98:0/2633240722 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44c8156690 con 0x7f44b4004d50 2023-10-06T05:07:51.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.798+0000 7f44bdffb700 1 -- 172.21.15.98:0/2633240722 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44c4015430 con 0x7f44b4004d50 2023-10-06T05:07:51.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.798+0000 7f44cfe4d700 1 -- 172.21.15.98:0/2633240722 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44c8154e40 con 0x7f44b4004d50 2023-10-06T05:07:51.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.799+0000 7f44cfe4d700 1 -- 172.21.15.98:0/2633240722 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44c8064a90 con 0x7f44b4004d50 2023-10-06T05:07:51.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.799+0000 7f44bdffb700 1 -- 172.21.15.98:0/2633240722 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f44c4015820 con 0x7f44b4004d50 2023-10-06T05:07:51.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.800+0000 7f44bdffb700 1 --2- 172.21.15.98:0/2633240722 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f44ac066f30 0x7f44ac0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:51.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.800+0000 7f44bdffb700 1 -- 172.21.15.98:0/2633240722 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5526+0+0 (secure 0 0 0) 0x7f44c4085200 con 0x7f44b4004d50 2023-10-06T05:07:51.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.800+0000 7f44cd3e8700 1 --2- 172.21.15.98:0/2633240722 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f44ac066f30 0x7f44ac0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:51.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.800+0000 7f44cd3e8700 1 --2- 172.21.15.98:0/2633240722 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f44ac066f30 0x7f44ac0693e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f44c80af7b0 tx=0x7f44b8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:51.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:51.804+0000 7f44bdffb700 1 -- 172.21.15.98:0/2633240722 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f44c4055fb0 con 0x7f44b4004d50 2023-10-06T05:07:52.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.061+0000 7f44cfe4d700 1 -- 172.21.15.98:0/2633240722 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7f44c81697b0 con 0x7f44b4004d50 2023-10-06T05:07:52.068 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.066+0000 7f44bdffb700 1 -- 172.21.15.98:0/2633240722 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v9) v1 ==== 97+0+0 (secure 0 0 0) 0x7f44c405b030 con 0x7f44b4004d50 2023-10-06T05:07:52.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.068+0000 7f44cfe4d700 1 -- 172.21.15.98:0/2633240722 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f44ac066f30 msgr2=0x7f44ac0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:52.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.068+0000 7f44cfe4d700 1 --2- 172.21.15.98:0/2633240722 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f44ac066f30 0x7f44ac0693e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f44c80af7b0 tx=0x7f44b8008040 comp rx=0 tx=0).stop 2023-10-06T05:07:52.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.068+0000 7f44cfe4d700 1 -- 172.21.15.98:0/2633240722 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44b4004d50 msgr2=0x7f44c8155a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:52.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.068+0000 7f44cfe4d700 1 --2- 172.21.15.98:0/2633240722 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44b4004d50 0x7f44c8155a80 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f44c4011f50 tx=0x7f44c400b040 comp rx=0 tx=0).stop 2023-10-06T05:07:52.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.069+0000 7f44cfe4d700 1 -- 172.21.15.98:0/2633240722 shutdown_connections 2023-10-06T05:07:52.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.069+0000 7f44cfe4d700 1 --2- 172.21.15.98:0/2633240722 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f44c815f0a0 0x7f44c815f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:52.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.069+0000 7f44cfe4d700 1 --2- 172.21.15.98:0/2633240722 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f44ac066f30 0x7f44ac0693e0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:52.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.069+0000 7f44cfe4d700 1 --2- 172.21.15.98:0/2633240722 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44b4004d50 0x7f44c8155a80 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:52.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.069+0000 7f44cfe4d700 1 --2- 172.21.15.98:0/2633240722 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f44c815a2f0 0x7f44c8154900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:52.072 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.069+0000 7f44cfe4d700 1 -- 172.21.15.98:0/2633240722 >> 172.21.15.98:0/2633240722 conn(0x7f44c80997f0 msgr2=0x7f44c8155240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:52.072 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.069+0000 7f44cfe4d700 1 -- 172.21.15.98:0/2633240722 shutdown_connections 2023-10-06T05:07:52.072 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.069+0000 7f44cfe4d700 1 -- 172.21.15.98:0/2633240722 wait complete. 2023-10-06T05:07:52.082 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2023-10-06T05:07:52.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.195+0000 7f73b1829700 1 Processor -- start 2023-10-06T05:07:52.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.195+0000 7f73b1829700 1 -- start start 2023-10-06T05:07:52.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.196+0000 7f73b1829700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f73ac159440 0x7f73ac0b3090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:52.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.196+0000 7f73b1829700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f73ac0b35d0 0x7f73ac0cbc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:52.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.196+0000 7f73b1829700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73ac0cb7a0 con 0x7f73ac0cc170 2023-10-06T05:07:52.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.196+0000 7f73b1829700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73ac0b0860 con 0x7f73ac159440 2023-10-06T05:07:52.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.196+0000 7f73b1829700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73ac0cb150 con 0x7f73ac0b35d0 2023-10-06T05:07:52.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.196+0000 7f73a3fff700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f73ac0b35d0 0x7f73ac0cbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:52.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.196+0000 7f73b0827700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f73ac159440 0x7f73ac0b3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:52.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.196+0000 7f73a3fff700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f73ac0b35d0 0x7f73ac0cbc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.98:45670/0 (socket says 172.21.15.98:45670) 2023-10-06T05:07:52.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.196+0000 7f73b0827700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f73ac159440 0x7f73ac0b3090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.98:54280/0 (socket says 172.21.15.98:54280) 2023-10-06T05:07:52.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.196+0000 7f73a3fff700 1 -- 172.21.15.98:0/1231051538 learned_addr learned my addr 172.21.15.98:0/1231051538 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:52.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.196+0000 7f73b1028700 1 --1- 172.21.15.98:0/1231051538 >> v1:172.21.15.174:6789/0 conn(0x7f73ac0cc170 0x7f73ac0b2100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.98:41158/0 (socket says 172.21.15.98:41158) 2023-10-06T05:07:52.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.197+0000 7f73b0827700 1 -- 172.21.15.98:0/1231051538 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f73ac0b35d0 msgr2=0x7f73ac0cbc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:52.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.197+0000 7f73b0827700 1 --2- 172.21.15.98:0/1231051538 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f73ac0b35d0 0x7f73ac0cbc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:52.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.197+0000 7f73b0827700 1 -- 172.21.15.98:0/1231051538 >> v1:172.21.15.174:6789/0 conn(0x7f73ac0cc170 legacy=0x7f73ac0b2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:52.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.197+0000 7f73b0827700 1 -- 172.21.15.98:0/1231051538 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73ac0cb430 con 0x7f73ac159440 2023-10-06T05:07:52.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.197+0000 7f73a3fff700 1 --2- 172.21.15.98:0/1231051538 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f73ac0b35d0 0x7f73ac0cbc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T05:07:52.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.197+0000 7f73b0827700 1 --2- 172.21.15.98:0/1231051538 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f73ac159440 0x7f73ac0b3090 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f73a8009e60 tx=0x7f73a8009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b523caf82d389c54 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:52.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.197+0000 7f73a37fe700 1 -- 172.21.15.98:0/1231051538 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a800e070 con 0x7f73ac159440 2023-10-06T05:07:52.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.197+0000 7f73a37fe700 1 -- 172.21.15.98:0/1231051538 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73a800bb60 con 0x7f73ac159440 2023-10-06T05:07:52.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.197+0000 7f73a37fe700 1 -- 172.21.15.98:0/1231051538 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a8012430 con 0x7f73ac159440 2023-10-06T05:07:52.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.198+0000 7f73b1829700 1 -- 172.21.15.98:0/1231051538 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f73ac159440 msgr2=0x7f73ac0b3090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:52.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.198+0000 7f73b1829700 1 --2- 172.21.15.98:0/1231051538 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f73ac159440 0x7f73ac0b3090 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f73a8009e60 tx=0x7f73a8009780 comp rx=0 tx=0).stop 2023-10-06T05:07:52.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.198+0000 7f73b1829700 1 -- 172.21.15.98:0/1231051538 shutdown_connections 2023-10-06T05:07:52.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.198+0000 7f73b1829700 1 --2- 172.21.15.98:0/1231051538 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f73ac159440 0x7f73ac0b3090 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:52.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.198+0000 7f73b1829700 1 --2- 172.21.15.98:0/1231051538 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f73ac0b35d0 0x7f73ac0cbc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:52.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.198+0000 7f73b1829700 1 -- 172.21.15.98:0/1231051538 >> 172.21.15.98:0/1231051538 conn(0x7f73ac06ece0 msgr2=0x7f73ac06f0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:52.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.198+0000 7f73b1829700 1 -- 172.21.15.98:0/1231051538 shutdown_connections 2023-10-06T05:07:52.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.198+0000 7f73b1829700 1 -- 172.21.15.98:0/1231051538 wait complete. 2023-10-06T05:07:52.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.199+0000 7f73b1829700 1 Processor -- start 2023-10-06T05:07:52.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.199+0000 7f73b1829700 1 -- start start 2023-10-06T05:07:52.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.200+0000 7f73b1829700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f73ac0b35d0 0x7f73ac1549b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:52.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.200+0000 7f73b1829700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f73ac0cc170 0x7f73ac0c94f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:52.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.200+0000 7f73b1829700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73ac159440 0x7f73ac0c9a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:52.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.200+0000 7f73b1829700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73ac16a8a0 con 0x7f73ac159440 2023-10-06T05:07:52.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.200+0000 7f73b1829700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73ac169c90 con 0x7f73ac0cc170 2023-10-06T05:07:52.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.200+0000 7f73b1829700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73ac169f70 con 0x7f73ac0b35d0 2023-10-06T05:07:52.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.200+0000 7f73b1028700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73ac159440 0x7f73ac0c9a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:52.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.200+0000 7f73b1028700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73ac159440 0x7f73ac0c9a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47446/0 (socket says 172.21.15.98:47446) 2023-10-06T05:07:52.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.200+0000 7f73a3fff700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f73ac0cc170 0x7f73ac0c94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:52.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.200+0000 7f73b1028700 1 -- 172.21.15.98:0/3216867259 learned_addr learned my addr 172.21.15.98:0/3216867259 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:52.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.200+0000 7f73b0827700 1 --2- 172.21.15.98:0/3216867259 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f73ac0b35d0 0x7f73ac1549b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:52.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.200+0000 7f73b1028700 1 -- 172.21.15.98:0/3216867259 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f73ac0b35d0 msgr2=0x7f73ac1549b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:52.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.200+0000 7f73b1028700 1 --2- 172.21.15.98:0/3216867259 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f73ac0b35d0 0x7f73ac1549b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:52.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.200+0000 7f73b1028700 1 -- 172.21.15.98:0/3216867259 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f73ac0cc170 msgr2=0x7f73ac0c94f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:52.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.200+0000 7f73b1028700 1 --2- 172.21.15.98:0/3216867259 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f73ac0cc170 0x7f73ac0c94f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:52.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.200+0000 7f73b1028700 1 -- 172.21.15.98:0/3216867259 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f739c0094f0 con 0x7f73ac159440 2023-10-06T05:07:52.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.200+0000 7f73b0827700 1 --2- 172.21.15.98:0/3216867259 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f73ac0b35d0 0x7f73ac1549b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T05:07:52.206 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.200+0000 7f73b1028700 1 --2- 172.21.15.98:0/3216867259 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73ac159440 0x7f73ac0c9a30 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f73a400c590 tx=0x7f73a4007ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:52.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.200+0000 7f73a0ff9700 1 -- 172.21.15.98:0/3216867259 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a4005a70 con 0x7f73ac159440 2023-10-06T05:07:52.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.200+0000 7f73a0ff9700 1 -- 172.21.15.98:0/3216867259 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73a4005bf0 con 0x7f73ac159440 2023-10-06T05:07:52.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.200+0000 7f73a0ff9700 1 -- 172.21.15.98:0/3216867259 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a40070d0 con 0x7f73ac159440 2023-10-06T05:07:52.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.200+0000 7f73b1829700 1 -- 172.21.15.98:0/3216867259 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73a8009a20 con 0x7f73ac159440 2023-10-06T05:07:52.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.200+0000 7f73b1829700 1 -- 172.21.15.98:0/3216867259 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73ac16a530 con 0x7f73ac159440 2023-10-06T05:07:52.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.201+0000 7f73b1829700 1 -- 172.21.15.98:0/3216867259 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73ac05f230 con 0x7f73ac159440 2023-10-06T05:07:52.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.202+0000 7f73a0ff9700 1 -- 172.21.15.98:0/3216867259 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f73a40074c0 con 0x7f73ac159440 2023-10-06T05:07:52.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.202+0000 7f73a0ff9700 1 --2- 172.21.15.98:0/3216867259 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f7398066f30 0x7f73980693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:52.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.202+0000 7f73a0ff9700 1 -- 172.21.15.98:0/3216867259 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5526+0+0 (secure 0 0 0) 0x7f73a40837b0 con 0x7f73ac159440 2023-10-06T05:07:52.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.202+0000 7f73b0827700 1 --2- 172.21.15.98:0/3216867259 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f7398066f30 0x7f73980693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:52.207 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.203+0000 7f73b0827700 1 --2- 172.21.15.98:0/3216867259 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f7398066f30 0x7f73980693e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f73a80097c0 tx=0x7f73a8019000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:52.208 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.206+0000 7f73a0ff9700 1 -- 172.21.15.98:0/3216867259 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f73a4050ad0 con 0x7f73ac159440 2023-10-06T05:07:52.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.468+0000 7f73b1829700 1 -- 172.21.15.98:0/3216867259 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7f73ac159810 con 0x7f73ac159440 2023-10-06T05:07:52.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.473+0000 7f73a0ff9700 1 -- 172.21.15.98:0/3216867259 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v10) v1 ==== 90+0+0 (secure 0 0 0) 0x7f73a40170a0 con 0x7f73ac159440 2023-10-06T05:07:52.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.475+0000 7f73b1829700 1 -- 172.21.15.98:0/3216867259 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f7398066f30 msgr2=0x7f73980693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:52.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.475+0000 7f73b1829700 1 --2- 172.21.15.98:0/3216867259 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f7398066f30 0x7f73980693e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f73a80097c0 tx=0x7f73a8019000 comp rx=0 tx=0).stop 2023-10-06T05:07:52.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.475+0000 7f73b1829700 1 -- 172.21.15.98:0/3216867259 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73ac159440 msgr2=0x7f73ac0c9a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:52.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.475+0000 7f73b1829700 1 --2- 172.21.15.98:0/3216867259 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73ac159440 0x7f73ac0c9a30 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f73a400c590 tx=0x7f73a4007ef0 comp rx=0 tx=0).stop 2023-10-06T05:07:52.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.475+0000 7f73b1829700 1 -- 172.21.15.98:0/3216867259 shutdown_connections 2023-10-06T05:07:52.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.475+0000 7f73b1829700 1 --2- 172.21.15.98:0/3216867259 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f7398066f30 0x7f73980693e0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:52.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.475+0000 7f73b1829700 1 --2- 172.21.15.98:0/3216867259 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73ac159440 0x7f73ac0c9a30 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:52.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.475+0000 7f73b1829700 1 --2- 172.21.15.98:0/3216867259 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f73ac0b35d0 0x7f73ac1549b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:52.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.475+0000 7f73b1829700 1 --2- 172.21.15.98:0/3216867259 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f73ac0cc170 0x7f73ac0c94f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:52.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.476+0000 7f73b1829700 1 -- 172.21.15.98:0/3216867259 >> 172.21.15.98:0/3216867259 conn(0x7f73ac06ece0 msgr2=0x7f73ac0b4ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:52.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.476+0000 7f73b1829700 1 -- 172.21.15.98:0/3216867259 shutdown_connections 2023-10-06T05:07:52.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.476+0000 7f73b1829700 1 -- 172.21.15.98:0/3216867259 wait complete. 2023-10-06T05:07:52.490 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T05:07:52.605 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.602+0000 7f3bea123700 1 Processor -- start 2023-10-06T05:07:52.605 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.602+0000 7f3bea123700 1 -- start start 2023-10-06T05:07:52.616 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.603+0000 7f3bea123700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3be40cad80 0x7f3be4159f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:52.616 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.603+0000 7f3bea123700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3be4068d10 con 0x7f3be4159ad0 2023-10-06T05:07:52.616 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.603+0000 7f3bea123700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3be415a670 con 0x7f3be40cb5f0 2023-10-06T05:07:52.617 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.603+0000 7f3bea123700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3be415a490 con 0x7f3be40cad80 2023-10-06T05:07:52.617 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.603+0000 7f3be2ffd700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f3be4159ad0 0x7f3be40ca670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:55718/0 (socket says 172.21.15.98:55718) 2023-10-06T05:07:52.617 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.603+0000 7f3be2ffd700 1 -- 172.21.15.98:0/4200012389 learned_addr learned my addr 172.21.15.98:0/4200012389 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:07:52.617 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.604+0000 7f3be3fff700 1 --2- 172.21.15.98:0/4200012389 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3be40cad80 0x7f3be4159f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:52.617 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.604+0000 7f3be27fc700 1 -- 172.21.15.98:0/4200012389 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2514122616 0 0) 0x7f3be4068d10 con 0x7f3be4159ad0 2023-10-06T05:07:52.617 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.604+0000 7f3be27fc700 1 -- 172.21.15.98:0/4200012389 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bcc003570 con 0x7f3be4159ad0 2023-10-06T05:07:52.617 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.604+0000 7f3be27fc700 1 -- 172.21.15.98:0/4200012389 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2142295241 0 0) 0x7f3bcc003570 con 0x7f3be4159ad0 2023-10-06T05:07:52.618 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.604+0000 7f3be27fc700 1 -- 172.21.15.98:0/4200012389 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3be40cad80 msgr2=0x7f3be4159f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:52.618 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.604+0000 7f3be27fc700 1 --2- 172.21.15.98:0/4200012389 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3be40cad80 0x7f3be4159f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:52.618 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.604+0000 7f3be27fc700 1 -- 172.21.15.98:0/4200012389 >> v1:172.21.15.174:6789/0 conn(0x7f3be40cb5f0 legacy=0x7f3be40691a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:52.618 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.604+0000 7f3be27fc700 1 -- 172.21.15.98:0/4200012389 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3be415a950 con 0x7f3be4159ad0 2023-10-06T05:07:52.618 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.604+0000 7f3be27fc700 1 -- 172.21.15.98:0/4200012389 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1543991876 0 0) 0x7f3bd8001470 con 0x7f3be4159ad0 2023-10-06T05:07:52.618 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.604+0000 7f3be27fc700 1 -- 172.21.15.98:0/4200012389 >> v1:172.21.15.98:6789/0 conn(0x7f3be4159ad0 legacy=0x7f3be40ca670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:52.618 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.604+0000 7f3be27fc700 1 --2- 172.21.15.98:0/4200012389 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3bcc003a90 0x7f3bcc003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:52.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.604+0000 7f3be27fc700 1 --2- 172.21.15.98:0/4200012389 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3bcc004810 0x7f3bcc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:52.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.604+0000 7f3be27fc700 1 --2- 172.21.15.98:0/4200012389 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3be40cad80 0x7f3bcc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:52.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.604+0000 7f3be27fc700 1 -- 172.21.15.98:0/4200012389 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3be415a490 con 0x7f3bcc004810 2023-10-06T05:07:52.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.604+0000 7f3be27fc700 1 -- 172.21.15.98:0/4200012389 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bcc00bb10 con 0x7f3bcc003a90 2023-10-06T05:07:52.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.604+0000 7f3be27fc700 1 -- 172.21.15.98:0/4200012389 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bcc00bcd0 con 0x7f3be40cad80 2023-10-06T05:07:52.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.604+0000 7f3be2ffd700 1 --2- 172.21.15.98:0/4200012389 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3bcc004810 0x7f3bcc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:52.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.604+0000 7f3be3fff700 1 --2- 172.21.15.98:0/4200012389 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3be40cad80 0x7f3bcc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:52.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.604+0000 7f3be37fe700 1 --2- 172.21.15.98:0/4200012389 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3bcc003a90 0x7f3bcc003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:52.620 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.604+0000 7f3be2ffd700 1 -- 172.21.15.98:0/4200012389 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3be40cad80 msgr2=0x7f3bcc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:52.620 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.604+0000 7f3be2ffd700 1 --2- 172.21.15.98:0/4200012389 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3be40cad80 0x7f3bcc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:52.620 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.604+0000 7f3be2ffd700 1 -- 172.21.15.98:0/4200012389 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3bcc003a90 msgr2=0x7f3bcc003e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:52.620 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.604+0000 7f3be2ffd700 1 --2- 172.21.15.98:0/4200012389 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3bcc003a90 0x7f3bcc003e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:52.620 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.604+0000 7f3be2ffd700 1 -- 172.21.15.98:0/4200012389 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bcc003570 con 0x7f3bcc004810 2023-10-06T05:07:52.620 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.604+0000 7f3be37fe700 1 --2- 172.21.15.98:0/4200012389 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3bcc003a90 0x7f3bcc003e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:07:52.620 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.605+0000 7f3be2ffd700 1 --2- 172.21.15.98:0/4200012389 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3bcc004810 0x7f3bcc004c80 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f3bd800c2d0 tx=0x7f3bd8001df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff658ac1495ef88b server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:52.620 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.605+0000 7f3be27fc700 1 -- 172.21.15.98:0/4200012389 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bd800cbc0 con 0x7f3bcc004810 2023-10-06T05:07:52.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.605+0000 7f3be27fc700 1 -- 172.21.15.98:0/4200012389 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bd800cd20 con 0x7f3bcc004810 2023-10-06T05:07:52.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.605+0000 7f3be27fc700 1 -- 172.21.15.98:0/4200012389 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bd8005b70 con 0x7f3bcc004810 2023-10-06T05:07:52.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.605+0000 7f3bea123700 1 -- 172.21.15.98:0/4200012389 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3bcc004810 msgr2=0x7f3bcc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:52.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.605+0000 7f3bea123700 1 --2- 172.21.15.98:0/4200012389 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3bcc004810 0x7f3bcc004c80 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f3bd800c2d0 tx=0x7f3bd8001df0 comp rx=0 tx=0).stop 2023-10-06T05:07:52.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.605+0000 7f3be3fff700 1 -- 172.21.15.98:0/4200012389 reap_dead start 2023-10-06T05:07:52.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.605+0000 7f3bea123700 1 -- 172.21.15.98:0/4200012389 shutdown_connections 2023-10-06T05:07:52.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.605+0000 7f3bea123700 1 -- 172.21.15.98:0/4200012389 >> 172.21.15.98:0/4200012389 conn(0x7f3be40707d0 msgr2=0x7f3be4070bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:52.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.605+0000 7f3bea123700 1 -- 172.21.15.98:0/4200012389 shutdown_connections 2023-10-06T05:07:52.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.605+0000 7f3bea123700 1 -- 172.21.15.98:0/4200012389 wait complete. 2023-10-06T05:07:52.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.606+0000 7f3bea123700 1 Processor -- start 2023-10-06T05:07:52.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.606+0000 7f3bea123700 1 -- start start 2023-10-06T05:07:52.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.607+0000 7f3bea123700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3bcc004810 0x7f3be4152a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:52.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.607+0000 7f3bea123700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3be40c6630 0x7f3be40c7550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:52.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.607+0000 7f3bea123700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3be40b2ff0 0x7f3be40b3460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:52.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.607+0000 7f3bea123700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3be40b2b60 con 0x7f3be40b2ff0 2023-10-06T05:07:52.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.607+0000 7f3bea123700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3be4152fb0 con 0x7f3be40c6630 2023-10-06T05:07:52.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.607+0000 7f3bea123700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3be40b2230 con 0x7f3bcc004810 2023-10-06T05:07:52.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.607+0000 7f3be3fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3be40b2ff0 0x7f3be40b3460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:52.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.607+0000 7f3be3fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3be40b2ff0 0x7f3be40b3460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47468/0 (socket says 172.21.15.98:47468) 2023-10-06T05:07:52.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.607+0000 7f3be3fff700 1 -- 172.21.15.98:0/2462434164 learned_addr learned my addr 172.21.15.98:0/2462434164 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:52.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.608+0000 7f3be37fe700 1 --2- 172.21.15.98:0/2462434164 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3bcc004810 0x7f3be4152a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:52.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.608+0000 7f3be2ffd700 1 --2- 172.21.15.98:0/2462434164 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3be40c6630 0x7f3be40c7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:52.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.608+0000 7f3be3fff700 1 -- 172.21.15.98:0/2462434164 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3bcc004810 msgr2=0x7f3be4152a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:52.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.608+0000 7f3be3fff700 1 --2- 172.21.15.98:0/2462434164 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3bcc004810 0x7f3be4152a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:52.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.608+0000 7f3be3fff700 1 -- 172.21.15.98:0/2462434164 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3be40c6630 msgr2=0x7f3be40c7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:52.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.608+0000 7f3be3fff700 1 --2- 172.21.15.98:0/2462434164 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3be40c6630 0x7f3be40c7550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:52.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.608+0000 7f3be3fff700 1 -- 172.21.15.98:0/2462434164 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3be40b2510 con 0x7f3be40b2ff0 2023-10-06T05:07:52.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.608+0000 7f3be3fff700 1 --2- 172.21.15.98:0/2462434164 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3be40b2ff0 0x7f3be40b3460 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f3be406c0b0 tx=0x7f3bdc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:52.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.608+0000 7f3bcb7fe700 1 -- 172.21.15.98:0/2462434164 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bdc00f910 con 0x7f3be40b2ff0 2023-10-06T05:07:52.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.608+0000 7f3bcb7fe700 1 -- 172.21.15.98:0/2462434164 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bdc00fa70 con 0x7f3be40b2ff0 2023-10-06T05:07:52.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.608+0000 7f3bea123700 1 -- 172.21.15.98:0/2462434164 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3be40b27f0 con 0x7f3be40b2ff0 2023-10-06T05:07:52.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.608+0000 7f3bcb7fe700 1 -- 172.21.15.98:0/2462434164 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bdc00fe10 con 0x7f3be40b2ff0 2023-10-06T05:07:52.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.608+0000 7f3bea123700 1 -- 172.21.15.98:0/2462434164 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3be40b1ee0 con 0x7f3be40b2ff0 2023-10-06T05:07:52.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.609+0000 7f3bea123700 1 -- 172.21.15.98:0/2462434164 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3be40b2980 con 0x7f3be40b2ff0 2023-10-06T05:07:52.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.610+0000 7f3bcb7fe700 1 -- 172.21.15.98:0/2462434164 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f3bdc014650 con 0x7f3be40b2ff0 2023-10-06T05:07:52.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.610+0000 7f3bcb7fe700 1 --2- 172.21.15.98:0/2462434164 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f3bd0066f30 0x7f3bd00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:52.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.610+0000 7f3bcb7fe700 1 -- 172.21.15.98:0/2462434164 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5526+0+0 (secure 0 0 0) 0x7f3bdc084ec0 con 0x7f3be40b2ff0 2023-10-06T05:07:52.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.610+0000 7f3be37fe700 1 --2- 172.21.15.98:0/2462434164 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f3bd0066f30 0x7f3bd00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:52.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.611+0000 7f3be37fe700 1 --2- 172.21.15.98:0/2462434164 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f3bd0066f30 0x7f3bd00693e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3bd4007f40 tx=0x7f3bd4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:52.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.614+0000 7f3bcb7fe700 1 -- 172.21.15.98:0/2462434164 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3bdc052340 con 0x7f3be40b2ff0 2023-10-06T05:07:52.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.799+0000 7f3bea123700 1 -- 172.21.15.98:0/2462434164 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3be4064a90 con 0x7f3be40b2ff0 2023-10-06T05:07:52.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.800+0000 7f3bcb7fe700 1 -- 172.21.15.98:0/2462434164 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) v1 ==== 74+0+13855 (secure 0 0 0) 0x7f3bdc055960 con 0x7f3be40b2ff0 2023-10-06T05:07:52.802 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:07:52.802 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":39,"fsid":"f50c5df5-cad5-47a4-9bb1-8c791d4311ec","created":"2023-10-06T05:06:32.971175+0000","modified":"2023-10-06T05:07:49.261064+0000","last_up_change":"2023-10-06T05:06:44.016489+0000","last_in_change":"2023-10-06T05:06:35.544511+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-06T05:06:44.559671+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-06T05:06:51.432161+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d9495d95-da75-4c3b-8927-e9bae6a8b093","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":90627},{"type":"v1","addr":"172.21.15.98:6801","nonce":90627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":90627},{"type":"v1","addr":"172.21.15.98:6803","nonce":90627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":90627},{"type":"v1","addr":"172.21.15.98:6807","nonce":90627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":90627},{"type":"v1","addr":"172.21.15.98:6805","nonce":90627}]},"public_addr":"172.21.15.98:6801/90627","cluster_addr":"172.21.15.98:6803/90627","heartbeat_back_addr":"172.21.15.98:6807/90627","heartbeat_front_addr":"172.21.15.98:6805/90627","state":["exists","up"]},{"osd":1,"uuid":"f25ba5be-d65e-4707-a134-154092deb754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":90628},{"type":"v1","addr":"172.21.15.98:6809","nonce":90628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":90628},{"type":"v1","addr":"172.21.15.98:6811","nonce":90628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":90628},{"type":"v1","addr":"172.21.15.98:6815","nonce":90628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":90628},{"type":"v1","addr":"172.21.15.98:6813","nonce":90628}]},"public_addr":"172.21.15.98:6809/90628","cluster_addr":"172.21.15.98:6811/90628","heartbeat_back_addr":"172.21.15.98:6815/90628","heartbeat_front_addr":"172.21.15.98:6813/90628","state":["exists","up"]},{"osd":2,"uuid":"5ab9e028-945c-43e2-8c95-3b0b4cf38e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":90662},{"type":"v1","addr":"172.21.15.98:6818","nonce":90662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":90662},{"type":"v1","addr":"172.21.15.98:6822","nonce":90662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":90662},{"type":"v1","addr":"172.21.15.98:6830","nonce":90662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":90662},{"type":"v1","addr":"172.21.15.98:6826","nonce":90662}]},"public_addr":"172.21.15.98:6818/90662","cluster_addr":"172.21.15.98:6822/90662","heartbeat_back_addr":"172.21.15.98:6830/90662","heartbeat_front_addr":"172.21.15.98:6826/90662","state":["exists","up"]},{"osd":3,"uuid":"3bb803ff-1738-46bf-be62-2267bf378822","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6817","nonce":90663},{"type":"v1","addr":"172.21.15.98:6819","nonce":90663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6821","nonce":90663},{"type":"v1","addr":"172.21.15.98:6823","nonce":90663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6829","nonce":90663},{"type":"v1","addr":"172.21.15.98:6831","nonce":90663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6825","nonce":90663},{"type":"v1","addr":"172.21.15.98:6827","nonce":90663}]},"public_addr":"172.21.15.98:6819/90663","cluster_addr":"172.21.15.98:6823/90663","heartbeat_back_addr":"172.21.15.98:6831/90663","heartbeat_front_addr":"172.21.15.98:6827/90663","state":["exists","up"]},{"osd":4,"uuid":"7d57a4c9-6462-43f6-b64a-7f9e9b4353d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":90847},{"type":"v1","addr":"172.21.15.174:6801","nonce":90847}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":90847},{"type":"v1","addr":"172.21.15.174:6803","nonce":90847}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":90847},{"type":"v1","addr":"172.21.15.174:6807","nonce":90847}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":90847},{"type":"v1","addr":"172.21.15.174:6805","nonce":90847}]},"public_addr":"172.21.15.174:6801/90847","cluster_addr":"172.21.15.174:6803/90847","heartbeat_back_addr":"172.21.15.174:6807/90847","heartbeat_front_addr":"172.21.15.174:6805/90847","state":["exists","up"]},{"osd":5,"uuid":"83287ea4-e44d-4af9-bc77-36d47c60d4a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":90848},{"type":"v1","addr":"172.21.15.174:6809","nonce":90848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":90848},{"type":"v1","addr":"172.21.15.174:6811","nonce":90848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":90848},{"type":"v1","addr":"172.21.15.174:6815","nonce":90848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":90848},{"type":"v1","addr":"172.21.15.174:6813","nonce":90848}]},"public_addr":"172.21.15.174:6809/90848","cluster_addr":"172.21.15.174:6811/90848","heartbeat_back_addr":"172.21.15.174:6815/90848","heartbeat_front_addr":"172.21.15.174:6813/90848","state":["exists","up"]},{"osd":6,"uuid":"496a6272-892e-43c4-a145-206623c22f99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":90877},{"type":"v1","addr":"172.21.15.174:6817","nonce":90877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":90877},{"type":"v1","addr":"172.21.15.174:6819","nonce":90877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":90877},{"type":"v1","addr":"172.21.15.174:6823","nonce":90877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":90877},{"type":"v1","addr":"172.21.15.174:6821","nonce":90877}]},"public_addr":"172.21.15.174:6817/90877","cluster_addr":"172.21.15.174:6819/90877","heartbeat_back_addr":"172.21.15.174:6823/90877","heartbeat_front_addr":"172.21.15.174:6821/90877","state":["exists","up"]},{"osd":7,"uuid":"49967e34-8b07-4bab-a13e-0c16d6a0f3a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":90878},{"type":"v1","addr":"172.21.15.174:6825","nonce":90878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":90878},{"type":"v1","addr":"172.21.15.174:6827","nonce":90878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":90878},{"type":"v1","addr":"172.21.15.174:6831","nonce":90878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":90878},{"type":"v1","addr":"172.21.15.174:6829","nonce":90878}]},"public_addr":"172.21.15.174:6825/90878","cluster_addr":"172.21.15.174:6827/90878","heartbeat_back_addr":"172.21.15.174:6831/90878","heartbeat_front_addr":"172.21.15.174:6829/90878","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.446479+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.514266+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.794108+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.810894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.731919+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.717756+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.737435+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.508682+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T05:07:52.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.802+0000 7f3bea123700 1 -- 172.21.15.98:0/2462434164 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f3bd0066f30 msgr2=0x7f3bd00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:52.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.802+0000 7f3bea123700 1 --2- 172.21.15.98:0/2462434164 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f3bd0066f30 0x7f3bd00693e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3bd4007f40 tx=0x7f3bd4009040 comp rx=0 tx=0).stop 2023-10-06T05:07:52.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.802+0000 7f3bea123700 1 -- 172.21.15.98:0/2462434164 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3be40b2ff0 msgr2=0x7f3be40b3460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:52.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.802+0000 7f3bea123700 1 --2- 172.21.15.98:0/2462434164 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3be40b2ff0 0x7f3be40b3460 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f3be406c0b0 tx=0x7f3bdc0091a0 comp rx=0 tx=0).stop 2023-10-06T05:07:52.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.802+0000 7f3bea123700 1 -- 172.21.15.98:0/2462434164 shutdown_connections 2023-10-06T05:07:52.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.802+0000 7f3bea123700 1 --2- 172.21.15.98:0/2462434164 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f3bd0066f30 0x7f3bd00693e0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:52.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.802+0000 7f3bea123700 1 --2- 172.21.15.98:0/2462434164 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3be40b2ff0 0x7f3be40b3460 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:52.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.802+0000 7f3bea123700 1 --2- 172.21.15.98:0/2462434164 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3bcc004810 0x7f3be4152a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:52.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.802+0000 7f3bea123700 1 --2- 172.21.15.98:0/2462434164 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3be40c6630 0x7f3be40c7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:52.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.802+0000 7f3bea123700 1 -- 172.21.15.98:0/2462434164 >> 172.21.15.98:0/2462434164 conn(0x7f3be40707d0 msgr2=0x7f3be4153830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:52.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.802+0000 7f3bea123700 1 -- 172.21.15.98:0/2462434164 shutdown_connections 2023-10-06T05:07:52.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.802+0000 7f3bea123700 1 -- 172.21.15.98:0/2462434164 wait complete. 2023-10-06T05:07:52.816 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-06T05:06:44.559671+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-10-06T05:06:51.432161+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2023-10-06T05:07:52.816 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-06T05:07:52.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.932+0000 7f803a281700 1 Processor -- start 2023-10-06T05:07:52.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.932+0000 7f803a281700 1 -- start start 2023-10-06T05:07:52.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.933+0000 7f803a281700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f80340b2620 0x7f80340cccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:52.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.933+0000 7f803a281700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f80340b29f0 0x7f80340cd1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:52.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.933+0000 7f803a281700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f80340ceaa0 0x7f80340cef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:52.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.933+0000 7f803a281700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80340cc970 con 0x7f80340ceaa0 2023-10-06T05:07:52.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.933+0000 7f803a281700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80340cf480 con 0x7f80340b29f0 2023-10-06T05:07:52.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.933+0000 7f803a281700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80340cbd60 con 0x7f80340b2620 2023-10-06T05:07:52.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.933+0000 7f8039a80700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f80340ceaa0 0x7f80340cef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:52.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.933+0000 7f8039a80700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f80340ceaa0 0x7f80340cef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47480/0 (socket says 172.21.15.98:47480) 2023-10-06T05:07:52.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.933+0000 7f8039a80700 1 -- 172.21.15.98:0/3800379190 learned_addr learned my addr 172.21.15.98:0/3800379190 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:52.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.933+0000 7f803927f700 1 --2- 172.21.15.98:0/3800379190 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f80340b2620 0x7f80340cccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:52.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.933+0000 7f8038a7e700 1 --2- 172.21.15.98:0/3800379190 >> v2:172.21.15.174:3300/0 conn(0x7f80340b29f0 0x7f80340cd1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:52.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.933+0000 7f8039a80700 1 -- 172.21.15.98:0/3800379190 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f80340b2620 msgr2=0x7f80340cccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:52.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.933+0000 7f8039a80700 1 --2- 172.21.15.98:0/3800379190 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f80340b2620 0x7f80340cccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:52.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.933+0000 7f8039a80700 1 -- 172.21.15.98:0/3800379190 >> v2:172.21.15.174:3300/0 conn(0x7f80340b29f0 msgr2=0x7f80340cd1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:52.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.933+0000 7f8039a80700 1 --2- 172.21.15.98:0/3800379190 >> v2:172.21.15.174:3300/0 conn(0x7f80340b29f0 0x7f80340cd1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:52.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.933+0000 7f8039a80700 1 -- 172.21.15.98:0/3800379190 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80340cc040 con 0x7f80340ceaa0 2023-10-06T05:07:52.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.934+0000 7f8039a80700 1 --2- 172.21.15.98:0/3800379190 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80340ceaa0 0x7f80340cef40 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f80340893b0 tx=0x7f802c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b7bd6b850addf45 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:52.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.934+0000 7f802bfff700 1 -- 172.21.15.98:0/3800379190 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f802c00e070 con 0x7f80340ceaa0 2023-10-06T05:07:52.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.934+0000 7f802bfff700 1 -- 172.21.15.98:0/3800379190 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f802c00ba50 con 0x7f80340ceaa0 2023-10-06T05:07:52.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.934+0000 7f802bfff700 1 -- 172.21.15.98:0/3800379190 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f802c0092e0 con 0x7f80340ceaa0 2023-10-06T05:07:52.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.934+0000 7f803a281700 1 -- 172.21.15.98:0/3800379190 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80340ceaa0 msgr2=0x7f80340cef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:52.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.934+0000 7f803a281700 1 --2- 172.21.15.98:0/3800379190 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80340ceaa0 0x7f80340cef40 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f80340893b0 tx=0x7f802c009840 comp rx=0 tx=0).stop 2023-10-06T05:07:52.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.934+0000 7f803a281700 1 -- 172.21.15.98:0/3800379190 shutdown_connections 2023-10-06T05:07:52.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.934+0000 7f803a281700 1 --2- 172.21.15.98:0/3800379190 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80340ceaa0 0x7f80340cef40 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:52.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.934+0000 7f803a281700 1 --2- 172.21.15.98:0/3800379190 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f80340b2620 0x7f80340cccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:52.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.934+0000 7f803a281700 1 --2- 172.21.15.98:0/3800379190 >> v2:172.21.15.174:3300/0 conn(0x7f80340b29f0 0x7f80340cd1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:52.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.934+0000 7f803a281700 1 -- 172.21.15.98:0/3800379190 >> 172.21.15.98:0/3800379190 conn(0x7f8034099520 msgr2=0x7f803406f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:52.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.934+0000 7f803a281700 1 -- 172.21.15.98:0/3800379190 shutdown_connections 2023-10-06T05:07:52.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.934+0000 7f803a281700 1 -- 172.21.15.98:0/3800379190 wait complete. 2023-10-06T05:07:52.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.935+0000 7f803a281700 1 Processor -- start 2023-10-06T05:07:52.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.935+0000 7f803a281700 1 -- start start 2023-10-06T05:07:52.942 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.936+0000 7f803a281700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80340b2620 0x7f80340c7d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:52.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.936+0000 7f803a281700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f80340b29f0 0x7f80340c8dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:52.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.936+0000 7f803a281700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f80340ceaa0 0x7f80340c9310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:52.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.936+0000 7f803a281700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80340b3dd0 con 0x7f80340b2620 2023-10-06T05:07:52.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.936+0000 7f803a281700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80340b31c0 con 0x7f80340ceaa0 2023-10-06T05:07:52.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.936+0000 7f803a281700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80340b34a0 con 0x7f80340b29f0 2023-10-06T05:07:52.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.936+0000 7f803927f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80340b2620 0x7f80340c7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:52.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.936+0000 7f803927f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80340b2620 0x7f80340c7d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47494/0 (socket says 172.21.15.98:47494) 2023-10-06T05:07:52.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.936+0000 7f803927f700 1 -- 172.21.15.98:0/2172374831 learned_addr learned my addr 172.21.15.98:0/2172374831 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:52.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.937+0000 7f8038a7e700 1 --2- 172.21.15.98:0/2172374831 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f80340b29f0 0x7f80340c8dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:52.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.937+0000 7f8039a80700 1 --2- 172.21.15.98:0/2172374831 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f80340ceaa0 0x7f80340c9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:52.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.937+0000 7f803927f700 1 -- 172.21.15.98:0/2172374831 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f80340b29f0 msgr2=0x7f80340c8dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:52.944 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.937+0000 7f803927f700 1 --2- 172.21.15.98:0/2172374831 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f80340b29f0 0x7f80340c8dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:52.944 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.937+0000 7f803927f700 1 -- 172.21.15.98:0/2172374831 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f80340ceaa0 msgr2=0x7f80340c9310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:52.944 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.937+0000 7f803927f700 1 --2- 172.21.15.98:0/2172374831 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f80340ceaa0 0x7f80340c9310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:52.944 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.937+0000 7f803927f700 1 -- 172.21.15.98:0/2172374831 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f802c009ae0 con 0x7f80340b2620 2023-10-06T05:07:52.944 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.937+0000 7f803927f700 1 --2- 172.21.15.98:0/2172374831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80340b2620 0x7f80340c7d40 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f803000d840 tx=0x7f8030008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:52.944 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.937+0000 7f80297fa700 1 -- 172.21.15.98:0/2172374831 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8030014620 con 0x7f80340b2620 2023-10-06T05:07:52.944 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.937+0000 7f80297fa700 1 -- 172.21.15.98:0/2172374831 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8030014780 con 0x7f80340b2620 2023-10-06T05:07:52.944 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.937+0000 7f803a281700 1 -- 172.21.15.98:0/2172374831 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80340b3780 con 0x7f80340b2620 2023-10-06T05:07:52.944 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.937+0000 7f80297fa700 1 -- 172.21.15.98:0/2172374831 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8030014b40 con 0x7f80340b2620 2023-10-06T05:07:52.944 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.937+0000 7f803a281700 1 -- 172.21.15.98:0/2172374831 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80340c8280 con 0x7f80340b2620 2023-10-06T05:07:52.944 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.938+0000 7f803a281700 1 -- 172.21.15.98:0/2172374831 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f803405f230 con 0x7f80340b2620 2023-10-06T05:07:52.945 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.938+0000 7f80297fa700 1 -- 172.21.15.98:0/2172374831 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f803001d400 con 0x7f80340b2620 2023-10-06T05:07:52.945 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.939+0000 7f80297fa700 1 --2- 172.21.15.98:0/2172374831 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f8018066f30 0x7f80180693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:52.945 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.939+0000 7f8038a7e700 1 --2- 172.21.15.98:0/2172374831 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f8018066f30 0x7f80180693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:52.945 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.939+0000 7f80297fa700 1 -- 172.21.15.98:0/2172374831 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5526+0+0 (secure 0 0 0) 0x7f8030084d70 con 0x7f80340b2620 2023-10-06T05:07:52.945 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.939+0000 7f8038a7e700 1 --2- 172.21.15.98:0/2172374831 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f8018066f30 0x7f80180693e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f8034158ca0 tx=0x7f8024008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:52.945 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:52.943+0000 7f80297fa700 1 -- 172.21.15.98:0/2172374831 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8030052090 con 0x7f80340b2620 2023-10-06T05:07:53.128 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.126+0000 7f803a281700 1 -- 172.21.15.98:0/2172374831 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f80340cc320 con 0x7f80340b2620 2023-10-06T05:07:53.128 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.126+0000 7f80297fa700 1 -- 172.21.15.98:0/2172374831 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v39) v1 ==== 93+0+10 (secure 0 0 0) 0x7f80300556b0 con 0x7f80340b2620 2023-10-06T05:07:53.129 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2023-10-06T05:07:53.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.128+0000 7f803a281700 1 -- 172.21.15.98:0/2172374831 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f8018066f30 msgr2=0x7f80180693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:53.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.128+0000 7f803a281700 1 --2- 172.21.15.98:0/2172374831 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f8018066f30 0x7f80180693e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f8034158ca0 tx=0x7f8024008040 comp rx=0 tx=0).stop 2023-10-06T05:07:53.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.128+0000 7f803a281700 1 -- 172.21.15.98:0/2172374831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80340b2620 msgr2=0x7f80340c7d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:53.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.128+0000 7f803a281700 1 --2- 172.21.15.98:0/2172374831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80340b2620 0x7f80340c7d40 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f803000d840 tx=0x7f8030008040 comp rx=0 tx=0).stop 2023-10-06T05:07:53.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.128+0000 7f803a281700 1 -- 172.21.15.98:0/2172374831 shutdown_connections 2023-10-06T05:07:53.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.128+0000 7f803a281700 1 --2- 172.21.15.98:0/2172374831 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f80340ceaa0 0x7f80340c9310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:53.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.128+0000 7f803a281700 1 --2- 172.21.15.98:0/2172374831 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f8018066f30 0x7f80180693e0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:53.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.128+0000 7f803a281700 1 --2- 172.21.15.98:0/2172374831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80340b2620 0x7f80340c7d40 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:53.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.128+0000 7f803a281700 1 --2- 172.21.15.98:0/2172374831 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f80340b29f0 0x7f80340c8dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:53.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.128+0000 7f803a281700 1 -- 172.21.15.98:0/2172374831 >> 172.21.15.98:0/2172374831 conn(0x7f8034099520 msgr2=0x7f80340682c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:53.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.128+0000 7f803a281700 1 -- 172.21.15.98:0/2172374831 shutdown_connections 2023-10-06T05:07:53.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.129+0000 7f803a281700 1 -- 172.21.15.98:0/2172374831 wait complete. 2023-10-06T05:07:53.142 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-10-06T05:07:53.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.257+0000 7f1ad4789700 1 Processor -- start 2023-10-06T05:07:53.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.257+0000 7f1ad4789700 1 -- start start 2023-10-06T05:07:53.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.258+0000 7f1ad4789700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f1acc1598a0 0x7f1acc0ce7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:53.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.258+0000 7f1ad4789700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1acc0cf5d0 con 0x7f1acc1594d0 2023-10-06T05:07:53.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.258+0000 7f1ad4789700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1acc0cf8b0 con 0x7f1acc0cbc70 2023-10-06T05:07:53.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.258+0000 7f1ad4789700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1acc0cfb90 con 0x7f1acc1598a0 2023-10-06T05:07:53.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.258+0000 7f1ad2525700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f1acc1594d0 0x7f1acc0cb560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:55728/0 (socket says 172.21.15.98:55728) 2023-10-06T05:07:53.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.258+0000 7f1ad2525700 1 -- 172.21.15.98:0/1112409127 learned_addr learned my addr 172.21.15.98:0/1112409127 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:07:53.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.259+0000 7f1ad1523700 1 -- 172.21.15.98:0/1112409127 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2628032634 0 0) 0x7f1acc0cf5d0 con 0x7f1acc1594d0 2023-10-06T05:07:53.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.259+0000 7f1ad1d24700 1 --2- 172.21.15.98:0/1112409127 >> v2:172.21.15.174:3300/0 conn(0x7f1acc1598a0 0x7f1acc0ce7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:53.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.259+0000 7f1ad1523700 1 -- 172.21.15.98:0/1112409127 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ab8003570 con 0x7f1acc1594d0 2023-10-06T05:07:53.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.259+0000 7f1ad1523700 1 -- 172.21.15.98:0/1112409127 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2139556294 0 0) 0x7f1ab8003570 con 0x7f1acc1594d0 2023-10-06T05:07:53.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.259+0000 7f1ad1523700 1 -- 172.21.15.98:0/1112409127 >> v2:172.21.15.174:3300/0 conn(0x7f1acc1598a0 msgr2=0x7f1acc0ce7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:53.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.259+0000 7f1ad1523700 1 --2- 172.21.15.98:0/1112409127 >> v2:172.21.15.174:3300/0 conn(0x7f1acc1598a0 0x7f1acc0ce7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:53.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.259+0000 7f1ad1523700 1 -- 172.21.15.98:0/1112409127 >> v1:172.21.15.174:6789/0 conn(0x7f1acc0cbc70 legacy=0x7f1acc0ced20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:53.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.259+0000 7f1ad1523700 1 -- 172.21.15.98:0/1112409127 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1acc0cfe70 con 0x7f1acc1594d0 2023-10-06T05:07:53.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.259+0000 7f1ad1523700 1 -- 172.21.15.98:0/1112409127 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1543991876 0 0) 0x7f1ac8001470 con 0x7f1acc1594d0 2023-10-06T05:07:53.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.259+0000 7f1ad1523700 1 -- 172.21.15.98:0/1112409127 >> v1:172.21.15.98:6789/0 conn(0x7f1acc1594d0 legacy=0x7f1acc0cb560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:53.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.259+0000 7f1ad1523700 1 --2- 172.21.15.98:0/1112409127 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ab8003a60 0x7f1ab8003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:53.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.259+0000 7f1ad1523700 1 --2- 172.21.15.98:0/1112409127 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1ab8004810 0x7f1ab8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:53.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.259+0000 7f1ad1523700 1 --2- 172.21.15.98:0/1112409127 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1ab80051c0 0x7f1ab800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:53.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.259+0000 7f1ad1523700 1 -- 172.21.15.98:0/1112409127 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1acc0cfb90 con 0x7f1ab8003a60 2023-10-06T05:07:53.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.259+0000 7f1ad1523700 1 -- 172.21.15.98:0/1112409127 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab800c0d0 con 0x7f1ab8004810 2023-10-06T05:07:53.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.259+0000 7f1ad1523700 1 -- 172.21.15.98:0/1112409127 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab800c290 con 0x7f1ab80051c0 2023-10-06T05:07:53.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.259+0000 7f1ad2525700 1 --2- 172.21.15.98:0/1112409127 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ab8003a60 0x7f1ab8003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:53.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.259+0000 7f1ad1d24700 1 --2- 172.21.15.98:0/1112409127 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1ab8004810 0x7f1ab8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:53.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.259+0000 7f1ad2d26700 1 --2- 172.21.15.98:0/1112409127 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1ab80051c0 0x7f1ab800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:53.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.259+0000 7f1ad2525700 1 -- 172.21.15.98:0/1112409127 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1ab80051c0 msgr2=0x7f1ab800ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:53.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.259+0000 7f1ad2525700 1 --2- 172.21.15.98:0/1112409127 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1ab80051c0 0x7f1ab800ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:53.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.259+0000 7f1ad2525700 1 -- 172.21.15.98:0/1112409127 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1ab8004810 msgr2=0x7f1ab8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:53.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.259+0000 7f1ad2525700 1 --2- 172.21.15.98:0/1112409127 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1ab8004810 0x7f1ab8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:53.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.259+0000 7f1ad2525700 1 -- 172.21.15.98:0/1112409127 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ab8003570 con 0x7f1ab8003a60 2023-10-06T05:07:53.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.259+0000 7f1ad2d26700 1 -- 172.21.15.98:0/1112409127 reap_dead start 2023-10-06T05:07:53.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.259+0000 7f1ad2525700 1 --2- 172.21.15.98:0/1112409127 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ab8003a60 0x7f1ab8003e70 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f1ac800c3c0 tx=0x7f1ac8001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fbaaa9ed1456338 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:53.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.260+0000 7f1ad1523700 1 -- 172.21.15.98:0/1112409127 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ac800cc90 con 0x7f1ab8003a60 2023-10-06T05:07:53.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.260+0000 7f1ad1523700 1 -- 172.21.15.98:0/1112409127 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ac800ce10 con 0x7f1ab8003a60 2023-10-06T05:07:53.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.260+0000 7f1ad1523700 1 -- 172.21.15.98:0/1112409127 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ac8005b70 con 0x7f1ab8003a60 2023-10-06T05:07:53.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.260+0000 7f1ad4789700 1 -- 172.21.15.98:0/1112409127 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ab8003a60 msgr2=0x7f1ab8003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:53.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.260+0000 7f1ad4789700 1 --2- 172.21.15.98:0/1112409127 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ab8003a60 0x7f1ab8003e70 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f1ac800c3c0 tx=0x7f1ac8001e50 comp rx=0 tx=0).stop 2023-10-06T05:07:53.267 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.260+0000 7f1ad4789700 1 -- 172.21.15.98:0/1112409127 shutdown_connections 2023-10-06T05:07:53.267 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.260+0000 7f1ad4789700 1 --2- 172.21.15.98:0/1112409127 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ab8003a60 0x7f1ab8003e70 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:53.267 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.260+0000 7f1ad4789700 1 -- 172.21.15.98:0/1112409127 >> 172.21.15.98:0/1112409127 conn(0x7f1acc089e50 msgr2=0x7f1acc099770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:53.267 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.260+0000 7f1ad4789700 1 -- 172.21.15.98:0/1112409127 shutdown_connections 2023-10-06T05:07:53.267 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.260+0000 7f1ad4789700 1 -- 172.21.15.98:0/1112409127 wait complete. 2023-10-06T05:07:53.267 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.261+0000 7f1ad4789700 1 Processor -- start 2023-10-06T05:07:53.267 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.261+0000 7f1ad4789700 1 -- start start 2023-10-06T05:07:53.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.262+0000 7f1ad4789700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ab8003a60 0x7f1acc14a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:53.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.262+0000 7f1ad4789700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1acc0c9a80 0x7f1acc14e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:53.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.262+0000 7f1ad4789700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1acc0c8dd0 0x7f1acc0c9270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:53.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.262+0000 7f1ad4789700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1acc14a830 con 0x7f1ab8003a60 2023-10-06T05:07:53.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.262+0000 7f1ad4789700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1acc0c9f80 con 0x7f1acc0c9a80 2023-10-06T05:07:53.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.262+0000 7f1ad4789700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1acc0ca260 con 0x7f1acc0c8dd0 2023-10-06T05:07:53.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.262+0000 7f1ad2525700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ab8003a60 0x7f1acc14a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:53.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.262+0000 7f1ad2525700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ab8003a60 0x7f1acc14a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47516/0 (socket says 172.21.15.98:47516) 2023-10-06T05:07:53.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.262+0000 7f1ad2525700 1 -- 172.21.15.98:0/2188463525 learned_addr learned my addr 172.21.15.98:0/2188463525 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:53.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.262+0000 7f1ad2525700 1 -- 172.21.15.98:0/2188463525 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1acc0c8dd0 msgr2=0x7f1acc0c9270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:53.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.262+0000 7f1ad2525700 1 --2- 172.21.15.98:0/2188463525 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1acc0c8dd0 0x7f1acc0c9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:53.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.262+0000 7f1ad2525700 1 -- 172.21.15.98:0/2188463525 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1acc0c9a80 msgr2=0x7f1acc14e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:53.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.262+0000 7f1ad2525700 1 --2- 172.21.15.98:0/2188463525 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1acc0c9a80 0x7f1acc14e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:53.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.262+0000 7f1ad2525700 1 -- 172.21.15.98:0/2188463525 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1acc0ca540 con 0x7f1ab8003a60 2023-10-06T05:07:53.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.263+0000 7f1ad2525700 1 --2- 172.21.15.98:0/2188463525 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ab8003a60 0x7f1acc14a2f0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f1ac8007cd0 tx=0x7f1ac8015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:53.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.263+0000 7f1ac27fc700 1 -- 172.21.15.98:0/2188463525 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ac8022070 con 0x7f1ab8003a60 2023-10-06T05:07:53.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.263+0000 7f1ac27fc700 1 -- 172.21.15.98:0/2188463525 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ac80070d0 con 0x7f1ab8003a60 2023-10-06T05:07:53.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.263+0000 7f1ac27fc700 1 -- 172.21.15.98:0/2188463525 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ac8007490 con 0x7f1ab8003a60 2023-10-06T05:07:53.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.263+0000 7f1ad4789700 1 -- 172.21.15.98:0/2188463525 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1acc0c88b0 con 0x7f1ab8003a60 2023-10-06T05:07:53.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.263+0000 7f1ad4789700 1 -- 172.21.15.98:0/2188463525 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1acc14e810 con 0x7f1ab8003a60 2023-10-06T05:07:53.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.264+0000 7f1ad4789700 1 -- 172.21.15.98:0/2188463525 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1acc0c8ac0 con 0x7f1ab8003a60 2023-10-06T05:07:53.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.264+0000 7f1ac27fc700 1 -- 172.21.15.98:0/2188463525 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f1ac801e430 con 0x7f1ab8003a60 2023-10-06T05:07:53.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.264+0000 7f1ac27fc700 1 --2- 172.21.15.98:0/2188463525 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f1ab0066f60 0x7f1ab0069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:53.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.264+0000 7f1ac27fc700 1 -- 172.21.15.98:0/2188463525 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5526+0+0 (secure 0 0 0) 0x7f1ac80857c0 con 0x7f1ab8003a60 2023-10-06T05:07:53.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.264+0000 7f1ad1d24700 1 --2- 172.21.15.98:0/2188463525 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f1ab0066f60 0x7f1ab0069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:53.272 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.265+0000 7f1ad1d24700 1 --2- 172.21.15.98:0/2188463525 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f1ab0066f60 0x7f1ab0069410 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f1acc16cf70 tx=0x7f1abc0091a0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:53.272 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.269+0000 7f1ac27fc700 1 -- 172.21.15.98:0/2188463525 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1ac8052b90 con 0x7f1ab8003a60 2023-10-06T05:07:53.454 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.452+0000 7f1ad4789700 1 -- 172.21.15.98:0/2188463525 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f1acc14e810 con 0x7f1ab8003a60 2023-10-06T05:07:53.454 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.452+0000 7f1ac27fc700 1 -- 172.21.15.98:0/2188463525 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v39) v1 ==== 92+0+10 (secure 0 0 0) 0x7f1acc14e810 con 0x7f1ab8003a60 2023-10-06T05:07:53.454 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 8 2023-10-06T05:07:53.457 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.454+0000 7f1ad4789700 1 -- 172.21.15.98:0/2188463525 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f1ab0066f60 msgr2=0x7f1ab0069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:53.457 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.454+0000 7f1ad4789700 1 --2- 172.21.15.98:0/2188463525 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f1ab0066f60 0x7f1ab0069410 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f1acc16cf70 tx=0x7f1abc0091a0 comp rx=0 tx=0).stop 2023-10-06T05:07:53.457 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.454+0000 7f1ad4789700 1 -- 172.21.15.98:0/2188463525 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ab8003a60 msgr2=0x7f1acc14a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:53.457 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.454+0000 7f1ad4789700 1 --2- 172.21.15.98:0/2188463525 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ab8003a60 0x7f1acc14a2f0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f1ac8007cd0 tx=0x7f1ac8015000 comp rx=0 tx=0).stop 2023-10-06T05:07:53.457 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.454+0000 7f1ad4789700 1 -- 172.21.15.98:0/2188463525 shutdown_connections 2023-10-06T05:07:53.457 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.454+0000 7f1ad4789700 1 --2- 172.21.15.98:0/2188463525 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1acc0c8dd0 0x7f1acc0c9270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:53.457 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.454+0000 7f1ad4789700 1 --2- 172.21.15.98:0/2188463525 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f1ab0066f60 0x7f1ab0069410 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:53.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.454+0000 7f1ad4789700 1 --2- 172.21.15.98:0/2188463525 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ab8003a60 0x7f1acc14a2f0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:53.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.454+0000 7f1ad4789700 1 --2- 172.21.15.98:0/2188463525 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1acc0c9a80 0x7f1acc14e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:53.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.454+0000 7f1ad4789700 1 -- 172.21.15.98:0/2188463525 >> 172.21.15.98:0/2188463525 conn(0x7f1acc089e50 msgr2=0x7f1acc14d500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:53.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.454+0000 7f1ad4789700 1 -- 172.21.15.98:0/2188463525 shutdown_connections 2023-10-06T05:07:53.458 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.454+0000 7f1ad4789700 1 -- 172.21.15.98:0/2188463525 wait complete. 2023-10-06T05:07:53.468 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-06T05:07:53.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.581+0000 7fb8ab59e700 1 Processor -- start 2023-10-06T05:07:53.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.581+0000 7fb8ab59e700 1 -- start start 2023-10-06T05:07:53.588 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.582+0000 7fb8ab59e700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb8ac0cbc30 0x7fb8ac0cc080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:53.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.582+0000 7fb8ab59e700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fb8ac0b2100 0x7fb8ac0b2520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:53.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.582+0000 7fb8ab59e700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8ac0cb7a0 con 0x7fb8ac159440 2023-10-06T05:07:53.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.582+0000 7fb8ab59e700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8ac0b0860 con 0x7fb8ac0cbc30 2023-10-06T05:07:53.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.582+0000 7fb8ab59e700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8ac0cb150 con 0x7fb8ac0b2100 2023-10-06T05:07:53.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.582+0000 7fb8a1d9b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb8ac0cbc30 0x7fb8ac0cc080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:53.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.582+0000 7fb8aa59c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fb8ac159440 0x7fb8ac0b3090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:55742/0 (socket says 172.21.15.98:55742) 2023-10-06T05:07:53.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.582+0000 7fb8aa59c700 1 -- 172.21.15.98:0/2188427540 learned_addr learned my addr 172.21.15.98:0/2188427540 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:07:53.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.582+0000 7fb8a1d9b700 1 -- 172.21.15.98:0/2188427540 >> v2:172.21.15.174:3300/0 conn(0x7fb8ac0b2100 msgr2=0x7fb8ac0b2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:53.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.582+0000 7fb8aad9d700 1 --2- 172.21.15.98:0/2188427540 >> v2:172.21.15.174:3300/0 conn(0x7fb8ac0b2100 0x7fb8ac0b2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:53.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.582+0000 7fb8a1d9b700 1 --2- 172.21.15.98:0/2188427540 >> v2:172.21.15.174:3300/0 conn(0x7fb8ac0b2100 0x7fb8ac0b2520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:53.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.582+0000 7fb8a1d9b700 1 -- 172.21.15.98:0/2188427540 >> v1:172.21.15.98:6789/0 conn(0x7fb8ac159440 legacy=0x7fb8ac0b3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:53.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.582+0000 7fb8a9d9b700 1 -- 172.21.15.98:0/2188427540 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1509018326 0 0) 0x7fb8ac0cb7a0 con 0x7fb8ac159440 2023-10-06T05:07:53.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.582+0000 7fb8a1d9b700 1 -- 172.21.15.98:0/2188427540 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8ac0cb430 con 0x7fb8ac0cbc30 2023-10-06T05:07:53.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.582+0000 7fb8aad9d700 1 --2- 172.21.15.98:0/2188427540 >> v2:172.21.15.174:3300/0 conn(0x7fb8ac0b2100 0x7fb8ac0b2520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:07:53.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.582+0000 7fb8a1d9b700 1 --2- 172.21.15.98:0/2188427540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8ac0cbc30 0x7fb8ac0cc080 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac0ca7f0 tx=0x7fb8940091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44819fe2224c377 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:53.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.583+0000 7fb8a9d9b700 1 -- 172.21.15.98:0/2188427540 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb89400b6b0 con 0x7fb8ac0cbc30 2023-10-06T05:07:53.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.583+0000 7fb8a9d9b700 1 -- 172.21.15.98:0/2188427540 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb89400b810 con 0x7fb8ac0cbc30 2023-10-06T05:07:53.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.583+0000 7fb8a9d9b700 1 -- 172.21.15.98:0/2188427540 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb89400bbb0 con 0x7fb8ac0cbc30 2023-10-06T05:07:53.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.583+0000 7fb8ab59e700 1 -- 172.21.15.98:0/2188427540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8ac0cbc30 msgr2=0x7fb8ac0cc080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:53.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.583+0000 7fb8ab59e700 1 --2- 172.21.15.98:0/2188427540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8ac0cbc30 0x7fb8ac0cc080 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac0ca7f0 tx=0x7fb8940091a0 comp rx=0 tx=0).stop 2023-10-06T05:07:53.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.583+0000 7fb8ab59e700 1 -- 172.21.15.98:0/2188427540 shutdown_connections 2023-10-06T05:07:53.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.583+0000 7fb8ab59e700 1 --2- 172.21.15.98:0/2188427540 >> v2:172.21.15.174:3300/0 conn(0x7fb8ac0b2100 0x7fb8ac0b2520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:53.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.583+0000 7fb8ab59e700 1 --2- 172.21.15.98:0/2188427540 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8ac0cbc30 0x7fb8ac0cc080 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:53.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.583+0000 7fb8ab59e700 1 -- 172.21.15.98:0/2188427540 >> 172.21.15.98:0/2188427540 conn(0x7fb8ac06ece0 msgr2=0x7fb8ac06f0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:53.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.583+0000 7fb8ab59e700 1 -- 172.21.15.98:0/2188427540 shutdown_connections 2023-10-06T05:07:53.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.583+0000 7fb8ab59e700 1 -- 172.21.15.98:0/2188427540 wait complete. 2023-10-06T05:07:53.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.584+0000 7fb8ab59e700 1 Processor -- start 2023-10-06T05:07:53.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.584+0000 7fb8ab59e700 1 -- start start 2023-10-06T05:07:53.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.585+0000 7fb8ab59e700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb8ac0b2100 0x7fb8ac0c7f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:53.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.585+0000 7fb8ab59e700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb8ac0cbc30 0x7fb8ac14a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:53.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.585+0000 7fb8ab59e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8ac159440 0x7fb8ac15bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:53.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.585+0000 7fb8ab59e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8ac159a90 con 0x7fb8ac159440 2023-10-06T05:07:53.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.585+0000 7fb8ab59e700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8ac15a330 con 0x7fb8ac0cbc30 2023-10-06T05:07:53.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.585+0000 7fb8ab59e700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8ac0c8440 con 0x7fb8ac0b2100 2023-10-06T05:07:53.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.585+0000 7fb8aad9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8ac159440 0x7fb8ac15bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:53.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.585+0000 7fb8aad9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8ac159440 0x7fb8ac15bd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47540/0 (socket says 172.21.15.98:47540) 2023-10-06T05:07:53.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.585+0000 7fb8aad9d700 1 -- 172.21.15.98:0/2422585187 learned_addr learned my addr 172.21.15.98:0/2422585187 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:53.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.585+0000 7fb8a1d9b700 1 --2- 172.21.15.98:0/2422585187 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb8ac0cbc30 0x7fb8ac14a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:53.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.585+0000 7fb8aad9d700 1 -- 172.21.15.98:0/2422585187 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb8ac0b2100 msgr2=0x7fb8ac0c7f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:53.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.585+0000 7fb8aa59c700 1 --2- 172.21.15.98:0/2422585187 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb8ac0b2100 0x7fb8ac0c7f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:53.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.585+0000 7fb8aad9d700 1 --2- 172.21.15.98:0/2422585187 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb8ac0b2100 0x7fb8ac0c7f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:53.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.585+0000 7fb8aad9d700 1 -- 172.21.15.98:0/2422585187 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb8ac0cbc30 msgr2=0x7fb8ac14a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:53.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.585+0000 7fb8aad9d700 1 --2- 172.21.15.98:0/2422585187 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb8ac0cbc30 0x7fb8ac14a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:53.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.585+0000 7fb8aad9d700 1 -- 172.21.15.98:0/2422585187 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb894009420 con 0x7fb8ac159440 2023-10-06T05:07:53.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.585+0000 7fb8a1d9b700 1 --2- 172.21.15.98:0/2422585187 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb8ac0cbc30 0x7fb8ac14a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:07:53.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.586+0000 7fb8aad9d700 1 --2- 172.21.15.98:0/2422585187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8ac159440 0x7fb8ac15bd40 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fb8a400a380 tx=0x7fb8a4007900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:53.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.586+0000 7fb8a2ffd700 1 -- 172.21.15.98:0/2422585187 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8a400ae10 con 0x7fb8ac159440 2023-10-06T05:07:53.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.586+0000 7fb8a2ffd700 1 -- 172.21.15.98:0/2422585187 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8a400d650 con 0x7fb8ac159440 2023-10-06T05:07:53.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.586+0000 7fb8ab59e700 1 -- 172.21.15.98:0/2422585187 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8ac15c720 con 0x7fb8ac159440 2023-10-06T05:07:53.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.586+0000 7fb8a2ffd700 1 -- 172.21.15.98:0/2422585187 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8a400da10 con 0x7fb8ac159440 2023-10-06T05:07:53.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.586+0000 7fb8ab59e700 1 -- 172.21.15.98:0/2422585187 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8ac159fc0 con 0x7fb8ac159440 2023-10-06T05:07:53.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.587+0000 7fb8ab59e700 1 -- 172.21.15.98:0/2422585187 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8ac15a4c0 con 0x7fb8ac159440 2023-10-06T05:07:53.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.587+0000 7fb8a2ffd700 1 -- 172.21.15.98:0/2422585187 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fb8a401e3b0 con 0x7fb8ac159440 2023-10-06T05:07:53.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.587+0000 7fb8a2ffd700 1 --2- 172.21.15.98:0/2422585187 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fb88c066f30 0x7fb88c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:53.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.587+0000 7fb8a2ffd700 1 -- 172.21.15.98:0/2422585187 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5526+0+0 (secure 0 0 0) 0x7fb8a4084f50 con 0x7fb8ac159440 2023-10-06T05:07:53.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.588+0000 7fb8aa59c700 1 --2- 172.21.15.98:0/2422585187 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fb88c066f30 0x7fb88c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:53.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.588+0000 7fb8aa59c700 1 --2- 172.21.15.98:0/2422585187 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fb88c066f30 0x7fb88c0693e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb89c001750 tx=0x7fb89c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:53.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.592+0000 7fb8a2ffd700 1 -- 172.21.15.98:0/2422585187 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb8a4052270 con 0x7fb8ac159440 2023-10-06T05:07:53.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.779+0000 7fb8ab59e700 1 -- 172.21.15.98:0/2422585187 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fb8ac05f230 con 0x7fb8ac159440 2023-10-06T05:07:53.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.780+0000 7fb8a2ffd700 1 -- 172.21.15.98:0/2422585187 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v39) v1 ==== 84+0+126 (secure 0 0 0) 0x7fb8a401a020 con 0x7fb8ac159440 2023-10-06T05:07:53.782 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:07:53.782 INFO:teuthology.orchestra.run.smithi098.stdout:[ 2023-10-06T05:07:53.782 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-10-06T05:07:53.782 INFO:teuthology.orchestra.run.smithi098.stdout: "poolnum": 1, 2023-10-06T05:07:53.783 INFO:teuthology.orchestra.run.smithi098.stdout: "poolname": ".mgr" 2023-10-06T05:07:53.783 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2023-10-06T05:07:53.783 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-10-06T05:07:53.783 INFO:teuthology.orchestra.run.smithi098.stdout: "poolnum": 2, 2023-10-06T05:07:53.783 INFO:teuthology.orchestra.run.smithi098.stdout: "poolname": "rbd" 2023-10-06T05:07:53.783 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-10-06T05:07:53.783 INFO:teuthology.orchestra.run.smithi098.stdout:] 2023-10-06T05:07:53.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.782+0000 7fb8ab59e700 1 -- 172.21.15.98:0/2422585187 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fb88c066f30 msgr2=0x7fb88c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:53.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.782+0000 7fb8ab59e700 1 --2- 172.21.15.98:0/2422585187 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fb88c066f30 0x7fb88c0693e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb89c001750 tx=0x7fb89c009040 comp rx=0 tx=0).stop 2023-10-06T05:07:53.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.782+0000 7fb8ab59e700 1 -- 172.21.15.98:0/2422585187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8ac159440 msgr2=0x7fb8ac15bd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:53.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.782+0000 7fb8ab59e700 1 --2- 172.21.15.98:0/2422585187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8ac159440 0x7fb8ac15bd40 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fb8a400a380 tx=0x7fb8a4007900 comp rx=0 tx=0).stop 2023-10-06T05:07:53.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.782+0000 7fb8ab59e700 1 -- 172.21.15.98:0/2422585187 shutdown_connections 2023-10-06T05:07:53.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.782+0000 7fb8ab59e700 1 --2- 172.21.15.98:0/2422585187 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fb88c066f30 0x7fb88c0693e0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:53.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.782+0000 7fb8ab59e700 1 --2- 172.21.15.98:0/2422585187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8ac159440 0x7fb8ac15bd40 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:53.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.782+0000 7fb8ab59e700 1 --2- 172.21.15.98:0/2422585187 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb8ac0b2100 0x7fb8ac0c7f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:53.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.782+0000 7fb8ab59e700 1 --2- 172.21.15.98:0/2422585187 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb8ac0cbc30 0x7fb8ac14a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:53.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.782+0000 7fb8ab59e700 1 -- 172.21.15.98:0/2422585187 >> 172.21.15.98:0/2422585187 conn(0x7fb8ac06ece0 msgr2=0x7fb8ac155270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:53.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.782+0000 7fb8ab59e700 1 -- 172.21.15.98:0/2422585187 shutdown_connections 2023-10-06T05:07:53.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.782+0000 7fb8ab59e700 1 -- 172.21.15.98:0/2422585187 wait complete. 2023-10-06T05:07:53.796 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-06T05:07:53.796 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-06T05:07:53.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.912+0000 7f2f095fc700 1 Processor -- start 2023-10-06T05:07:53.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.912+0000 7f2f095fc700 1 -- start start 2023-10-06T05:07:53.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.913+0000 7f2f095fc700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2f0406ece0 0x7f2f040b2360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:53.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.913+0000 7f2f095fc700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f2f0406f0b0 0x7f2f040b08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:53.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.913+0000 7f2f095fc700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f040b2ce0 con 0x7f2f040aede0 2023-10-06T05:07:53.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.913+0000 7f2f095fc700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f0406fb40 con 0x7f2f0406ece0 2023-10-06T05:07:53.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.913+0000 7f2f095fc700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f040b0410 con 0x7f2f0406f0b0 2023-10-06T05:07:53.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.913+0000 7f2f02ffd700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2f0406ece0 0x7f2f040b2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:53.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.913+0000 7f2f037fe700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f2f040aede0 0x7f2f040af240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:55746/0 (socket says 172.21.15.98:55746) 2023-10-06T05:07:53.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.913+0000 7f2f037fe700 1 -- 172.21.15.98:0/367121713 learned_addr learned my addr 172.21.15.98:0/367121713 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:07:53.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.913+0000 7f2efa7fc700 1 --2- 172.21.15.98:0/367121713 >> v2:172.21.15.174:3300/0 conn(0x7f2f0406f0b0 0x7f2f040b08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:53.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.913+0000 7f2f02ffd700 1 -- 172.21.15.98:0/367121713 >> v2:172.21.15.174:3300/0 conn(0x7f2f0406f0b0 msgr2=0x7f2f040b08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:53.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.913+0000 7f2f02ffd700 1 --2- 172.21.15.98:0/367121713 >> v2:172.21.15.174:3300/0 conn(0x7f2f0406f0b0 0x7f2f040b08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:53.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.913+0000 7f2f02ffd700 1 -- 172.21.15.98:0/367121713 >> v1:172.21.15.98:6789/0 conn(0x7f2f040aede0 legacy=0x7f2f040af240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:53.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.913+0000 7f2f027fc700 1 -- 172.21.15.98:0/367121713 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1148064848 0 0) 0x7f2f040b2ce0 con 0x7f2f040aede0 2023-10-06T05:07:53.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.913+0000 7f2f02ffd700 1 -- 172.21.15.98:0/367121713 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f040b2a00 con 0x7f2f0406ece0 2023-10-06T05:07:53.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.914+0000 7f2f02ffd700 1 --2- 172.21.15.98:0/367121713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f0406ece0 0x7f2f040b2360 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f2ef40098e0 tx=0x7f2ef40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f10135d748eaa024 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:53.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.914+0000 7f2f027fc700 1 -- 172.21.15.98:0/367121713 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef4011630 con 0x7f2f0406ece0 2023-10-06T05:07:53.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.914+0000 7f2f027fc700 1 -- 172.21.15.98:0/367121713 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ef4011790 con 0x7f2f0406ece0 2023-10-06T05:07:53.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.914+0000 7f2f027fc700 1 -- 172.21.15.98:0/367121713 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef4011b30 con 0x7f2f0406ece0 2023-10-06T05:07:53.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.914+0000 7f2f095fc700 1 -- 172.21.15.98:0/367121713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f0406ece0 msgr2=0x7f2f040b2360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:53.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.914+0000 7f2f095fc700 1 --2- 172.21.15.98:0/367121713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f0406ece0 0x7f2f040b2360 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f2ef40098e0 tx=0x7f2ef40091a0 comp rx=0 tx=0).stop 2023-10-06T05:07:53.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.914+0000 7f2f095fc700 1 -- 172.21.15.98:0/367121713 shutdown_connections 2023-10-06T05:07:53.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.914+0000 7f2f095fc700 1 --2- 172.21.15.98:0/367121713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f0406ece0 0x7f2f040b2360 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:53.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.914+0000 7f2f095fc700 1 --2- 172.21.15.98:0/367121713 >> v2:172.21.15.174:3300/0 conn(0x7f2f0406f0b0 0x7f2f040b08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:53.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.914+0000 7f2f095fc700 1 -- 172.21.15.98:0/367121713 >> 172.21.15.98:0/367121713 conn(0x7f2f040682f0 msgr2=0x7f2f040686f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:53.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.914+0000 7f2f095fc700 1 -- 172.21.15.98:0/367121713 shutdown_connections 2023-10-06T05:07:53.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.914+0000 7f2f095fc700 1 -- 172.21.15.98:0/367121713 wait complete. 2023-10-06T05:07:53.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.916+0000 7f2f095fc700 1 Processor -- start 2023-10-06T05:07:53.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.916+0000 7f2f095fc700 1 -- start start 2023-10-06T05:07:53.922 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.916+0000 7f2f095fc700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2f0406ece0 0x7f2f0414f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:53.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.916+0000 7f2f095fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f0406f0b0 0x7f2f040a7d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:53.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.916+0000 7f2f095fc700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2f040aede0 0x7f2f040a8270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:53.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.916+0000 7f2f095fc700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f04150900 con 0x7f2f0406f0b0 2023-10-06T05:07:53.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.916+0000 7f2f095fc700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0414fcf0 con 0x7f2f0406ece0 2023-10-06T05:07:53.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.916+0000 7f2f095fc700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0414ffd0 con 0x7f2f040aede0 2023-10-06T05:07:53.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.916+0000 7f2efa7fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f0406f0b0 0x7f2f040a7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:53.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.916+0000 7f2efa7fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f0406f0b0 0x7f2f040a7d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47558/0 (socket says 172.21.15.98:47558) 2023-10-06T05:07:53.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.916+0000 7f2efa7fc700 1 -- 172.21.15.98:0/1358132721 learned_addr learned my addr 172.21.15.98:0/1358132721 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:53.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.917+0000 7f2f037fe700 1 --2- 172.21.15.98:0/1358132721 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2f040aede0 0x7f2f040a8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:53.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.917+0000 7f2f02ffd700 1 --2- 172.21.15.98:0/1358132721 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2f0406ece0 0x7f2f0414f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:53.923 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.917+0000 7f2efa7fc700 1 -- 172.21.15.98:0/1358132721 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2f040aede0 msgr2=0x7f2f040a8270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:53.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.917+0000 7f2efa7fc700 1 --2- 172.21.15.98:0/1358132721 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2f040aede0 0x7f2f040a8270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:53.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.917+0000 7f2efa7fc700 1 -- 172.21.15.98:0/1358132721 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2f0406ece0 msgr2=0x7f2f0414f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:53.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.917+0000 7f2efa7fc700 1 --2- 172.21.15.98:0/1358132721 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2f0406ece0 0x7f2f0414f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:53.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.917+0000 7f2efa7fc700 1 -- 172.21.15.98:0/1358132721 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ef4009420 con 0x7f2f0406f0b0 2023-10-06T05:07:53.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.917+0000 7f2f037fe700 1 --2- 172.21.15.98:0/1358132721 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2f040aede0 0x7f2f040a8270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:07:53.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.917+0000 7f2f02ffd700 1 --2- 172.21.15.98:0/1358132721 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2f0406ece0 0x7f2f0414f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:07:53.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.917+0000 7f2efa7fc700 1 --2- 172.21.15.98:0/1358132721 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f0406f0b0 0x7f2f040a7d30 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f2eec00e950 tx=0x7f2eec0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:53.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.917+0000 7f2efb7fe700 1 -- 172.21.15.98:0/1358132721 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eec00c850 con 0x7f2f0406f0b0 2023-10-06T05:07:53.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.917+0000 7f2efb7fe700 1 -- 172.21.15.98:0/1358132721 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2eec00c9b0 con 0x7f2f0406f0b0 2023-10-06T05:07:53.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.917+0000 7f2efb7fe700 1 -- 172.21.15.98:0/1358132721 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eec00cdb0 con 0x7f2f0406f0b0 2023-10-06T05:07:53.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.917+0000 7f2f095fc700 1 -- 172.21.15.98:0/1358132721 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f041502b0 con 0x7f2f0406f0b0 2023-10-06T05:07:53.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.917+0000 7f2f095fc700 1 -- 172.21.15.98:0/1358132721 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f04156200 con 0x7f2f0406f0b0 2023-10-06T05:07:53.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.918+0000 7f2f095fc700 1 -- 172.21.15.98:0/1358132721 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f04150a70 con 0x7f2f0406f0b0 2023-10-06T05:07:53.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.918+0000 7f2efb7fe700 1 -- 172.21.15.98:0/1358132721 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f2eec013680 con 0x7f2f0406f0b0 2023-10-06T05:07:53.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.919+0000 7f2efb7fe700 1 --2- 172.21.15.98:0/1358132721 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f2ee4066f30 0x7f2ee40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:53.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.919+0000 7f2efb7fe700 1 -- 172.21.15.98:0/1358132721 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5526+0+0 (secure 0 0 0) 0x7f2eec084750 con 0x7f2f0406f0b0 2023-10-06T05:07:53.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.919+0000 7f2f02ffd700 1 --2- 172.21.15.98:0/1358132721 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f2ee4066f30 0x7f2ee40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:53.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.919+0000 7f2f02ffd700 1 --2- 172.21.15.98:0/1358132721 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f2ee4066f30 0x7f2ee40693e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f2f040d08b0 tx=0x7f2ef4017000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:53.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:53.923+0000 7f2efb7fe700 1 -- 172.21.15.98:0/1358132721 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2eec0519d0 con 0x7f2f0406f0b0 2023-10-06T05:07:54.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.107+0000 7f2f095fc700 1 -- 172.21.15.98:0/1358132721 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f2f04064a90 con 0x7f2f0406f0b0 2023-10-06T05:07:54.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.476+0000 7f2efb7fe700 1 -- 172.21.15.98:0/1358132721 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v40) v1 ==== 138+0+0 (secure 0 0 0) 0x7f2eec054ff0 con 0x7f2f0406f0b0 2023-10-06T05:07:54.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.479+0000 7f2f095fc700 1 -- 172.21.15.98:0/1358132721 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f2ee4066f30 msgr2=0x7f2ee40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:54.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.479+0000 7f2f095fc700 1 --2- 172.21.15.98:0/1358132721 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f2ee4066f30 0x7f2ee40693e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f2f040d08b0 tx=0x7f2ef4017000 comp rx=0 tx=0).stop 2023-10-06T05:07:54.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.479+0000 7f2f095fc700 1 -- 172.21.15.98:0/1358132721 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f0406f0b0 msgr2=0x7f2f040a7d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:54.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.479+0000 7f2f095fc700 1 --2- 172.21.15.98:0/1358132721 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f0406f0b0 0x7f2f040a7d30 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f2eec00e950 tx=0x7f2eec0091a0 comp rx=0 tx=0).stop 2023-10-06T05:07:54.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.480+0000 7f2f095fc700 1 -- 172.21.15.98:0/1358132721 shutdown_connections 2023-10-06T05:07:54.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.480+0000 7f2f095fc700 1 --2- 172.21.15.98:0/1358132721 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2f040aede0 0x7f2f040a8270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:54.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.480+0000 7f2f095fc700 1 --2- 172.21.15.98:0/1358132721 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2f0406ece0 0x7f2f0414f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:54.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.480+0000 7f2f095fc700 1 --2- 172.21.15.98:0/1358132721 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f2ee4066f30 0x7f2ee40693e0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:54.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.480+0000 7f2f095fc700 1 --2- 172.21.15.98:0/1358132721 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2f0406f0b0 0x7f2f040a7d30 secure :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f2eec00e950 tx=0x7f2eec0091a0 comp rx=0 tx=0).stop 2023-10-06T05:07:54.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.480+0000 7f2f095fc700 1 -- 172.21.15.98:0/1358132721 >> 172.21.15.98:0/1358132721 conn(0x7f2f040682f0 msgr2=0x7f2f040c94b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:54.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.480+0000 7f2f095fc700 1 -- 172.21.15.98:0/1358132721 shutdown_connections 2023-10-06T05:07:54.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.481+0000 7f2f095fc700 1 -- 172.21.15.98:0/1358132721 wait complete. 2023-10-06T05:07:54.485 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'cephfs_metadata' created 2023-10-06T05:07:54.497 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-06T05:07:54.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.617+0000 7f7d9f3a8700 1 Processor -- start 2023-10-06T05:07:54.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.617+0000 7f7d9f3a8700 1 -- start start 2023-10-06T05:07:54.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.618+0000 7f7d9f3a8700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7d981598a0 0x7f7d980ce7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:54.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.618+0000 7f7d9f3a8700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f7d980cbc70 0x7f7d980ced20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:54.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.618+0000 7f7d9f3a8700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d980cf5d0 con 0x7f7d981594d0 2023-10-06T05:07:54.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.618+0000 7f7d9f3a8700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d980cf8b0 con 0x7f7d980cbc70 2023-10-06T05:07:54.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.618+0000 7f7d9f3a8700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d980cfb90 con 0x7f7d981598a0 2023-10-06T05:07:54.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.618+0000 7f7d9d945700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f7d980cbc70 0x7f7d980ced20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:54.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.618+0000 7f7d9c943700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7d981598a0 0x7f7d980ce7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:54.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.618+0000 7f7d9d144700 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f7d981594d0 0x7f7d980cb560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.98:41174/0 (socket says 172.21.15.98:41174) 2023-10-06T05:07:54.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.618+0000 7f7d9d945700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f7d980cbc70 0x7f7d980ced20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.98:54410/0 (socket says 172.21.15.98:54410) 2023-10-06T05:07:54.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.618+0000 7f7d9d144700 1 -- 172.21.15.98:0/2401746547 learned_addr learned my addr 172.21.15.98:0/2401746547 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:07:54.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.618+0000 7f7d9c943700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7d981598a0 0x7f7d980ce7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.98:45790/0 (socket says 172.21.15.98:45790) 2023-10-06T05:07:54.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.618+0000 7f7d9d945700 1 -- 172.21.15.98:0/2401746547 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7d981598a0 msgr2=0x7f7d980ce7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:54.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.618+0000 7f7d9d945700 1 --2- 172.21.15.98:0/2401746547 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7d981598a0 0x7f7d980ce7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:54.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.618+0000 7f7d9d945700 1 -- 172.21.15.98:0/2401746547 >> v1:172.21.15.174:6789/0 conn(0x7f7d981594d0 legacy=0x7f7d980cb560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:54.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.618+0000 7f7d9d945700 1 -- 172.21.15.98:0/2401746547 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d980cfe70 con 0x7f7d980cbc70 2023-10-06T05:07:54.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.618+0000 7f7d8ffff700 1 -- 172.21.15.98:0/2401746547 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1683519923 0 0) 0x7f7d980cf5d0 con 0x7f7d981594d0 2023-10-06T05:07:54.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.619+0000 7f7d9d945700 1 --2- 172.21.15.98:0/2401746547 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d980cbc70 0x7f7d980ced20 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f7d90009e60 tx=0x7f7d90009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f6cbac5310121847 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:54.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.619+0000 7f7d8ffff700 1 -- 172.21.15.98:0/2401746547 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d9000e070 con 0x7f7d980cbc70 2023-10-06T05:07:54.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.619+0000 7f7d8ffff700 1 -- 172.21.15.98:0/2401746547 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d9000bb60 con 0x7f7d980cbc70 2023-10-06T05:07:54.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.619+0000 7f7d8ffff700 1 -- 172.21.15.98:0/2401746547 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d90012430 con 0x7f7d980cbc70 2023-10-06T05:07:54.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.619+0000 7f7d9f3a8700 1 -- 172.21.15.98:0/2401746547 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d980cbc70 msgr2=0x7f7d980ced20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:54.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.619+0000 7f7d9f3a8700 1 --2- 172.21.15.98:0/2401746547 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d980cbc70 0x7f7d980ced20 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f7d90009e60 tx=0x7f7d90009780 comp rx=0 tx=0).stop 2023-10-06T05:07:54.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.619+0000 7f7d9f3a8700 1 -- 172.21.15.98:0/2401746547 shutdown_connections 2023-10-06T05:07:54.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.619+0000 7f7d9f3a8700 1 --2- 172.21.15.98:0/2401746547 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d980cbc70 0x7f7d980ced20 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:54.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.619+0000 7f7d9f3a8700 1 --2- 172.21.15.98:0/2401746547 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7d981598a0 0x7f7d980ce7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:54.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.619+0000 7f7d9f3a8700 1 -- 172.21.15.98:0/2401746547 >> 172.21.15.98:0/2401746547 conn(0x7f7d98089e50 msgr2=0x7f7d98099770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:54.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.619+0000 7f7d9f3a8700 1 -- 172.21.15.98:0/2401746547 shutdown_connections 2023-10-06T05:07:54.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.619+0000 7f7d9f3a8700 1 -- 172.21.15.98:0/2401746547 wait complete. 2023-10-06T05:07:54.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.620+0000 7f7d9f3a8700 1 Processor -- start 2023-10-06T05:07:54.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.620+0000 7f7d9f3a8700 1 -- start start 2023-10-06T05:07:54.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.621+0000 7f7d9f3a8700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d980cbc70 0x7f7d9815b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:54.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.621+0000 7f7d9f3a8700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7d981594d0 0x7f7d9816d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:54.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.621+0000 7f7d9f3a8700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d981598a0 0x7f7d9816d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:54.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.621+0000 7f7d9f3a8700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d980b3f30 con 0x7f7d980cbc70 2023-10-06T05:07:54.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.621+0000 7f7d9f3a8700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d9815c080 con 0x7f7d981598a0 2023-10-06T05:07:54.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.621+0000 7f7d9f3a8700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d9815c360 con 0x7f7d981594d0 2023-10-06T05:07:54.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.621+0000 7f7d9d144700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d980cbc70 0x7f7d9815b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:54.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.621+0000 7f7d9d144700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d980cbc70 0x7f7d9815b650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47560/0 (socket says 172.21.15.98:47560) 2023-10-06T05:07:54.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.622+0000 7f7d9d144700 1 -- 172.21.15.98:0/2073466055 learned_addr learned my addr 172.21.15.98:0/2073466055 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:54.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.622+0000 7f7d9d945700 1 --2- 172.21.15.98:0/2073466055 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d981598a0 0x7f7d9816d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:54.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.622+0000 7f7d9c943700 1 --2- 172.21.15.98:0/2073466055 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7d981594d0 0x7f7d9816d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:54.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.622+0000 7f7d9d144700 1 -- 172.21.15.98:0/2073466055 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7d981594d0 msgr2=0x7f7d9816d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:54.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.622+0000 7f7d9d144700 1 --2- 172.21.15.98:0/2073466055 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7d981594d0 0x7f7d9816d2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:54.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.622+0000 7f7d9d144700 1 -- 172.21.15.98:0/2073466055 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d981598a0 msgr2=0x7f7d9816d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:54.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.622+0000 7f7d9d144700 1 --2- 172.21.15.98:0/2073466055 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d981598a0 0x7f7d9816d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:54.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.622+0000 7f7d9d144700 1 -- 172.21.15.98:0/2073466055 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d90009a20 con 0x7f7d980cbc70 2023-10-06T05:07:54.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.622+0000 7f7d9d144700 1 --2- 172.21.15.98:0/2073466055 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d980cbc70 0x7f7d9815b650 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f7d9400c7e0 tx=0x7f7d94007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:54.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.622+0000 7f7d8d7fa700 1 -- 172.21.15.98:0/2073466055 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d94005b30 con 0x7f7d980cbc70 2023-10-06T05:07:54.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.622+0000 7f7d8d7fa700 1 -- 172.21.15.98:0/2073466055 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d94005c90 con 0x7f7d980cbc70 2023-10-06T05:07:54.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.622+0000 7f7d9f3a8700 1 -- 172.21.15.98:0/2073466055 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d9815c640 con 0x7f7d980cbc70 2023-10-06T05:07:54.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.622+0000 7f7d8d7fa700 1 -- 172.21.15.98:0/2073466055 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d94007300 con 0x7f7d980cbc70 2023-10-06T05:07:54.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.622+0000 7f7d9f3a8700 1 -- 172.21.15.98:0/2073466055 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d980b60e0 con 0x7f7d980cbc70 2023-10-06T05:07:54.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.623+0000 7f7d9f3a8700 1 -- 172.21.15.98:0/2073466055 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d98064a90 con 0x7f7d980cbc70 2023-10-06T05:07:54.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.624+0000 7f7d8d7fa700 1 -- 172.21.15.98:0/2073466055 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f7d9401b400 con 0x7f7d980cbc70 2023-10-06T05:07:54.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.624+0000 7f7d8d7fa700 1 --2- 172.21.15.98:0/2073466055 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f7d7c066f30 0x7f7d7c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:54.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.624+0000 7f7d9c943700 1 --2- 172.21.15.98:0/2073466055 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f7d7c066f30 0x7f7d7c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:54.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.624+0000 7f7d8d7fa700 1 -- 172.21.15.98:0/2073466055 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5896+0+0 (secure 0 0 0) 0x7f7d94084170 con 0x7f7d980cbc70 2023-10-06T05:07:54.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.624+0000 7f7d9c943700 1 --2- 172.21.15.98:0/2073466055 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f7d7c066f30 0x7f7d7c0693e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f7d980b38d0 tx=0x7f7d88008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:54.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.629+0000 7f7d8d7fa700 1 -- 172.21.15.98:0/2073466055 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7d940513a0 con 0x7f7d980cbc70 2023-10-06T05:07:54.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:54.812+0000 7f7d9f3a8700 1 -- 172.21.15.98:0/2073466055 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f7d980cf260 con 0x7f7d980cbc70 2023-10-06T05:07:55.485 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:55.483+0000 7f7d8d7fa700 1 -- 172.21.15.98:0/2073466055 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v41) v1 ==== 170+0+0 (secure 0 0 0) 0x7f7d940549c0 con 0x7f7d980cbc70 2023-10-06T05:07:55.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:55.491+0000 7f7d9f3a8700 1 -- 172.21.15.98:0/2073466055 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f7d7c066f30 msgr2=0x7f7d7c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:55.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:55.491+0000 7f7d9f3a8700 1 --2- 172.21.15.98:0/2073466055 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f7d7c066f30 0x7f7d7c0693e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f7d980b38d0 tx=0x7f7d88008040 comp rx=0 tx=0).stop 2023-10-06T05:07:55.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:55.491+0000 7f7d9f3a8700 1 -- 172.21.15.98:0/2073466055 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d980cbc70 msgr2=0x7f7d9815b650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:55.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:55.491+0000 7f7d9f3a8700 1 --2- 172.21.15.98:0/2073466055 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d980cbc70 0x7f7d9815b650 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f7d9400c7e0 tx=0x7f7d94007df0 comp rx=0 tx=0).stop 2023-10-06T05:07:55.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:55.491+0000 7f7d9f3a8700 1 -- 172.21.15.98:0/2073466055 shutdown_connections 2023-10-06T05:07:55.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:55.491+0000 7f7d9f3a8700 1 --2- 172.21.15.98:0/2073466055 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d981598a0 0x7f7d9816d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:55.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:55.491+0000 7f7d9f3a8700 1 --2- 172.21.15.98:0/2073466055 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f7d7c066f30 0x7f7d7c0693e0 secure :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f7d980b38d0 tx=0x7f7d88008040 comp rx=0 tx=0).stop 2023-10-06T05:07:55.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:55.491+0000 7f7d9f3a8700 1 --2- 172.21.15.98:0/2073466055 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7d980cbc70 0x7f7d9815b650 secure :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f7d9400c7e0 tx=0x7f7d94007df0 comp rx=0 tx=0).stop 2023-10-06T05:07:55.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:55.491+0000 7f7d9f3a8700 1 --2- 172.21.15.98:0/2073466055 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7d981594d0 0x7f7d9816d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:55.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:55.491+0000 7f7d9f3a8700 1 -- 172.21.15.98:0/2073466055 >> 172.21.15.98:0/2073466055 conn(0x7f7d98089e50 msgr2=0x7f7d9814b3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:55.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:55.492+0000 7f7d9f3a8700 1 -- 172.21.15.98:0/2073466055 shutdown_connections 2023-10-06T05:07:55.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:55.494+0000 7f7d9f3a8700 1 -- 172.21.15.98:0/2073466055 wait complete. 2023-10-06T05:07:55.500 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'cephfs_data' created 2023-10-06T05:07:55.516 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-06T05:07:55.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:55.634+0000 7f83d621b700 1 Processor -- start 2023-10-06T05:07:55.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:55.634+0000 7f83d621b700 1 -- start start 2023-10-06T05:07:55.638 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:55.635+0000 7f83d621b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f83d0067bd0 0x7f83d00cad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:55.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:55.635+0000 7f83d621b700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f83d015a1f0 0x7f83d00cb2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:55.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:55.635+0000 7f83d621b700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83d015d860 con 0x7f83d015dcf0 2023-10-06T05:07:55.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:55.635+0000 7f83d621b700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83d015aad0 con 0x7f83d0067bd0 2023-10-06T05:07:55.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:55.635+0000 7f83d621b700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83d015a7f0 con 0x7f83d015a1f0 2023-10-06T05:07:55.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:55.635+0000 7f83cf7fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f83d0067bd0 0x7f83d00cad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:55.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:55.635+0000 7f83ceffd700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f83d015dcf0 0x7f83d0159ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:55750/0 (socket says 172.21.15.98:55750) 2023-10-06T05:07:55.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:55.635+0000 7f83ceffd700 1 -- 172.21.15.98:0/3240602469 learned_addr learned my addr 172.21.15.98:0/3240602469 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:07:55.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:55.635+0000 7f83cffff700 1 --2- 172.21.15.98:0/3240602469 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f83d015a1f0 0x7f83d00cb2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:55.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:55.635+0000 7f83cf7fe700 1 -- 172.21.15.98:0/3240602469 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f83d015a1f0 msgr2=0x7f83d00cb2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:55.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:55.635+0000 7f83cf7fe700 1 --2- 172.21.15.98:0/3240602469 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f83d015a1f0 0x7f83d00cb2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:55.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:55.635+0000 7f83cf7fe700 1 -- 172.21.15.98:0/3240602469 >> v1:172.21.15.98:6789/0 conn(0x7f83d015dcf0 legacy=0x7f83d0159ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:55.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:55.635+0000 7f83cf7fe700 1 -- 172.21.15.98:0/3240602469 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83d015d210 con 0x7f83d0067bd0 2023-10-06T05:07:55.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:55.635+0000 7f83ce7fc700 1 -- 172.21.15.98:0/3240602469 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2677241394 0 0) 0x7f83d015d860 con 0x7f83d015dcf0 2023-10-06T05:07:55.641 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:55.635+0000 7f83cf7fe700 1 --2- 172.21.15.98:0/3240602469 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83d0067bd0 0x7f83d00cad70 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f83c0009e60 tx=0x7f83c0009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcd50488d6e8dd9c server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:55.641 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:55.636+0000 7f83ce7fc700 1 -- 172.21.15.98:0/3240602469 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83c000e070 con 0x7f83d0067bd0 2023-10-06T05:07:55.641 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:55.636+0000 7f83ce7fc700 1 -- 172.21.15.98:0/3240602469 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83c000bb60 con 0x7f83d0067bd0 2023-10-06T05:07:55.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:55.636+0000 7f83ce7fc700 1 -- 172.21.15.98:0/3240602469 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83c0012430 con 0x7f83d0067bd0 2023-10-06T05:07:55.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:55.636+0000 7f83d621b700 1 -- 172.21.15.98:0/3240602469 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83d0067bd0 msgr2=0x7f83d00cad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:55.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:55.636+0000 7f83d621b700 1 --2- 172.21.15.98:0/3240602469 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83d0067bd0 0x7f83d00cad70 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f83c0009e60 tx=0x7f83c0009780 comp rx=0 tx=0).stop 2023-10-06T05:07:55.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:55.636+0000 7f83d621b700 1 -- 172.21.15.98:0/3240602469 shutdown_connections 2023-10-06T05:07:55.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:55.636+0000 7f83d621b700 1 --2- 172.21.15.98:0/3240602469 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f83d015a1f0 0x7f83d00cb2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:55.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:55.636+0000 7f83d621b700 1 --2- 172.21.15.98:0/3240602469 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83d0067bd0 0x7f83d00cad70 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:55.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:55.636+0000 7f83d621b700 1 -- 172.21.15.98:0/3240602469 >> 172.21.15.98:0/3240602469 conn(0x7f83d00cd1a0 msgr2=0x7f83d00cd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:55.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:55.636+0000 7f83d621b700 1 -- 172.21.15.98:0/3240602469 shutdown_connections 2023-10-06T05:07:55.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:55.636+0000 7f83d621b700 1 -- 172.21.15.98:0/3240602469 wait complete. 2023-10-06T05:07:55.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:55.637+0000 7f83d621b700 1 Processor -- start 2023-10-06T05:07:55.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:55.637+0000 7f83d621b700 1 -- start start 2023-10-06T05:07:55.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:55.638+0000 7f83d621b700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f83d0067bd0 0x7f83d00b19f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:55.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:55.638+0000 7f83d621b700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f83d015a1f0 0x7f83d00c8280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:55.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:55.638+0000 7f83d621b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83d015dcf0 0x7f83d014e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:55.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:55.638+0000 7f83d621b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83d00c87c0 con 0x7f83d015dcf0 2023-10-06T05:07:55.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:55.638+0000 7f83d621b700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83d00b0070 con 0x7f83d0067bd0 2023-10-06T05:07:55.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:55.638+0000 7f83d621b700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83d00b20f0 con 0x7f83d015a1f0 2023-10-06T05:07:55.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:55.638+0000 7f83cffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83d015dcf0 0x7f83d014e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:55.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:55.638+0000 7f83cffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83d015dcf0 0x7f83d014e1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47572/0 (socket says 172.21.15.98:47572) 2023-10-06T05:07:55.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:55.638+0000 7f83cffff700 1 -- 172.21.15.98:0/822715239 learned_addr learned my addr 172.21.15.98:0/822715239 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:55.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:55.638+0000 7f83ceffd700 1 --2- 172.21.15.98:0/822715239 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f83d015a1f0 0x7f83d00c8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:55.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:55.639+0000 7f83cffff700 1 -- 172.21.15.98:0/822715239 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f83d015a1f0 msgr2=0x7f83d00c8280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:55.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:55.639+0000 7f83cffff700 1 --2- 172.21.15.98:0/822715239 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f83d015a1f0 0x7f83d00c8280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:55.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:55.639+0000 7f83cffff700 1 -- 172.21.15.98:0/822715239 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f83d0067bd0 msgr2=0x7f83d00b19f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:55.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:55.639+0000 7f83cf7fe700 1 --2- 172.21.15.98:0/822715239 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f83d0067bd0 0x7f83d00b19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:55.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:55.639+0000 7f83cffff700 1 --2- 172.21.15.98:0/822715239 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f83d0067bd0 0x7f83d00b19f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:55.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:55.639+0000 7f83cffff700 1 -- 172.21.15.98:0/822715239 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83c0009a20 con 0x7f83d015dcf0 2023-10-06T05:07:55.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:55.639+0000 7f83ceffd700 1 --2- 172.21.15.98:0/822715239 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f83d015a1f0 0x7f83d00c8280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:07:55.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:55.639+0000 7f83cffff700 1 --2- 172.21.15.98:0/822715239 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83d015dcf0 0x7f83d014e1c0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f83c800ea50 tx=0x7f83c80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:55.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:55.639+0000 7f83b77fe700 1 -- 172.21.15.98:0/822715239 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83c800c920 con 0x7f83d015dcf0 2023-10-06T05:07:55.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:55.639+0000 7f83b77fe700 1 -- 172.21.15.98:0/822715239 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83c800caa0 con 0x7f83d015dcf0 2023-10-06T05:07:55.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:55.639+0000 7f83d621b700 1 -- 172.21.15.98:0/822715239 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83d00b23d0 con 0x7f83d015dcf0 2023-10-06T05:07:55.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:55.639+0000 7f83b77fe700 1 -- 172.21.15.98:0/822715239 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83c800ce60 con 0x7f83d015dcf0 2023-10-06T05:07:55.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:55.639+0000 7f83d621b700 1 -- 172.21.15.98:0/822715239 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83d014e700 con 0x7f83d015dcf0 2023-10-06T05:07:55.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:55.640+0000 7f83d621b700 1 -- 172.21.15.98:0/822715239 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83d00b01e0 con 0x7f83d015dcf0 2023-10-06T05:07:55.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:55.640+0000 7f83b77fe700 1 -- 172.21.15.98:0/822715239 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f83c8013680 con 0x7f83d015dcf0 2023-10-06T05:07:55.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:55.641+0000 7f83b77fe700 1 --2- 172.21.15.98:0/822715239 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f83bc066f30 0x7f83bc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:55.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:55.641+0000 7f83b77fe700 1 -- 172.21.15.98:0/822715239 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6278+0+0 (secure 0 0 0) 0x7f83c8084b30 con 0x7f83d015dcf0 2023-10-06T05:07:55.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:55.641+0000 7f83cf7fe700 1 --2- 172.21.15.98:0/822715239 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f83bc066f30 0x7f83bc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:55.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:55.641+0000 7f83cf7fe700 1 --2- 172.21.15.98:0/822715239 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f83bc066f30 0x7f83bc0693e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f83c00097c0 tx=0x7f83c0019000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:55.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:55.645+0000 7f83b77fe700 1 -- 172.21.15.98:0/822715239 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f83c8051b60 con 0x7f83d015dcf0 2023-10-06T05:07:55.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:55.871+0000 7f83d621b700 1 -- 172.21.15.98:0/822715239 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f83d0064a90 con 0x7f83d015dcf0 2023-10-06T05:07:56.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.486+0000 7f83b77fe700 1 -- 172.21.15.98:0/822715239 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 6 and data pool 7 v13) v1 ==== 172+0+0 (secure 0 0 0) 0x7f83c8055180 con 0x7f83d015dcf0 2023-10-06T05:07:56.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.492+0000 7f83d621b700 1 -- 172.21.15.98:0/822715239 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f83bc066f30 msgr2=0x7f83bc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:56.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.492+0000 7f83d621b700 1 --2- 172.21.15.98:0/822715239 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f83bc066f30 0x7f83bc0693e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f83c00097c0 tx=0x7f83c0019000 comp rx=0 tx=0).stop 2023-10-06T05:07:56.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.492+0000 7f83d621b700 1 -- 172.21.15.98:0/822715239 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83d015dcf0 msgr2=0x7f83d014e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:56.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.492+0000 7f83d621b700 1 --2- 172.21.15.98:0/822715239 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83d015dcf0 0x7f83d014e1c0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f83c800ea50 tx=0x7f83c80091a0 comp rx=0 tx=0).stop 2023-10-06T05:07:56.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.494+0000 7f83d621b700 1 -- 172.21.15.98:0/822715239 shutdown_connections 2023-10-06T05:07:56.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.495+0000 7f83d621b700 1 --2- 172.21.15.98:0/822715239 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f83bc066f30 0x7f83bc0693e0 secure :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f83c00097c0 tx=0x7f83c0019000 comp rx=0 tx=0).stop 2023-10-06T05:07:56.497 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.495+0000 7f83d621b700 1 --2- 172.21.15.98:0/822715239 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83d015dcf0 0x7f83d014e1c0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:56.497 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.495+0000 7f83d621b700 1 --2- 172.21.15.98:0/822715239 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f83d0067bd0 0x7f83d00b19f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:56.497 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.495+0000 7f83d621b700 1 --2- 172.21.15.98:0/822715239 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f83d015a1f0 0x7f83d00c8280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:56.497 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.495+0000 7f83d621b700 1 -- 172.21.15.98:0/822715239 >> 172.21.15.98:0/822715239 conn(0x7f83d00cd1a0 msgr2=0x7f83d00cd570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:56.500 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.498+0000 7f83d621b700 1 -- 172.21.15.98:0/822715239 shutdown_connections 2023-10-06T05:07:56.500 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.498+0000 7f83d621b700 1 -- 172.21.15.98:0/822715239 wait complete. 2023-10-06T05:07:56.510 INFO:teuthology.orchestra.run.smithi098.stderr:new fs with metadata pool 6 and data pool 7 2023-10-06T05:07:56.525 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T05:07:56.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.638+0000 7f888bfff700 1 Processor -- start 2023-10-06T05:07:56.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.638+0000 7f888bfff700 1 -- start start 2023-10-06T05:07:56.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.639+0000 7f888bfff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f888c0cbc30 0x7f888c0cc080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:56.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.639+0000 7f888bfff700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f888c0b2100 0x7f888c0b2520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:56.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.639+0000 7f888bfff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f888c0cb7a0 con 0x7f888c159440 2023-10-06T05:07:56.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.639+0000 7f888bfff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f888c0b0860 con 0x7f888c0cbc30 2023-10-06T05:07:56.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.639+0000 7f888bfff700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f888c0cb150 con 0x7f888c0b2100 2023-10-06T05:07:56.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.639+0000 7f88827fc700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f888c0cbc30 0x7f888c0cc080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:56.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.639+0000 7f888affd700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f888c159440 0x7f888c0b3090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:55756/0 (socket says 172.21.15.98:55756) 2023-10-06T05:07:56.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.639+0000 7f888affd700 1 -- 172.21.15.98:0/776923897 learned_addr learned my addr 172.21.15.98:0/776923897 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:07:56.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.639+0000 7f888b7fe700 1 --2- 172.21.15.98:0/776923897 >> v2:172.21.15.174:3300/0 conn(0x7f888c0b2100 0x7f888c0b2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:56.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.639+0000 7f88827fc700 1 -- 172.21.15.98:0/776923897 >> v2:172.21.15.174:3300/0 conn(0x7f888c0b2100 msgr2=0x7f888c0b2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:56.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.639+0000 7f88827fc700 1 --2- 172.21.15.98:0/776923897 >> v2:172.21.15.174:3300/0 conn(0x7f888c0b2100 0x7f888c0b2520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:56.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.639+0000 7f88827fc700 1 -- 172.21.15.98:0/776923897 >> v1:172.21.15.98:6789/0 conn(0x7f888c159440 legacy=0x7f888c0b3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:56.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.639+0000 7f88827fc700 1 -- 172.21.15.98:0/776923897 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f888c0cb430 con 0x7f888c0cbc30 2023-10-06T05:07:56.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.639+0000 7f888a7fc700 1 -- 172.21.15.98:0/776923897 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 787353317 0 0) 0x7f888c0cb7a0 con 0x7f888c159440 2023-10-06T05:07:56.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.640+0000 7f88827fc700 1 --2- 172.21.15.98:0/776923897 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f888c0cbc30 0x7f888c0cc080 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f888c0ca7f0 tx=0x7f88740091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=936871a00e581aff server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:56.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.640+0000 7f888a7fc700 1 -- 172.21.15.98:0/776923897 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f887400b6b0 con 0x7f888c0cbc30 2023-10-06T05:07:56.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.640+0000 7f888a7fc700 1 -- 172.21.15.98:0/776923897 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f887400b810 con 0x7f888c0cbc30 2023-10-06T05:07:56.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.640+0000 7f888a7fc700 1 -- 172.21.15.98:0/776923897 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f887400bbb0 con 0x7f888c0cbc30 2023-10-06T05:07:56.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.640+0000 7f888bfff700 1 -- 172.21.15.98:0/776923897 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f888c0cbc30 msgr2=0x7f888c0cc080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:56.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.640+0000 7f888bfff700 1 --2- 172.21.15.98:0/776923897 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f888c0cbc30 0x7f888c0cc080 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f888c0ca7f0 tx=0x7f88740091a0 comp rx=0 tx=0).stop 2023-10-06T05:07:56.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.640+0000 7f888bfff700 1 -- 172.21.15.98:0/776923897 shutdown_connections 2023-10-06T05:07:56.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.640+0000 7f888bfff700 1 --2- 172.21.15.98:0/776923897 >> v2:172.21.15.174:3300/0 conn(0x7f888c0b2100 0x7f888c0b2520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:56.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.640+0000 7f888bfff700 1 --2- 172.21.15.98:0/776923897 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f888c0cbc30 0x7f888c0cc080 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:56.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.640+0000 7f888bfff700 1 -- 172.21.15.98:0/776923897 >> 172.21.15.98:0/776923897 conn(0x7f888c06ece0 msgr2=0x7f888c06f0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:56.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.640+0000 7f888bfff700 1 -- 172.21.15.98:0/776923897 shutdown_connections 2023-10-06T05:07:56.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.640+0000 7f888bfff700 1 -- 172.21.15.98:0/776923897 wait complete. 2023-10-06T05:07:56.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.641+0000 7f888bfff700 1 Processor -- start 2023-10-06T05:07:56.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.641+0000 7f888bfff700 1 -- start start 2023-10-06T05:07:56.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.642+0000 7f888bfff700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f888c0b2100 0x7f888c153c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:56.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.642+0000 7f888bfff700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f888c0cbc30 0x7f888c0c72d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:56.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.642+0000 7f888bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f888c159440 0x7f888c0c7810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:56.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.642+0000 7f888bfff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f888c1541c0 con 0x7f888c159440 2023-10-06T05:07:56.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.642+0000 7f888bfff700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f888c0ce440 con 0x7f888c0b2100 2023-10-06T05:07:56.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.642+0000 7f888bfff700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f888c0c9730 con 0x7f888c0cbc30 2023-10-06T05:07:56.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.642+0000 7f888b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f888c159440 0x7f888c0c7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:56.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.642+0000 7f888b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f888c159440 0x7f888c0c7810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47602/0 (socket says 172.21.15.98:47602) 2023-10-06T05:07:56.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.642+0000 7f888b7fe700 1 -- 172.21.15.98:0/433893211 learned_addr learned my addr 172.21.15.98:0/433893211 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:56.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.642+0000 7f888affd700 1 --2- 172.21.15.98:0/433893211 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f888c0b2100 0x7f888c153c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:56.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.642+0000 7f88827fc700 1 --2- 172.21.15.98:0/433893211 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f888c0cbc30 0x7f888c0c72d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:56.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.643+0000 7f888b7fe700 1 -- 172.21.15.98:0/433893211 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f888c0cbc30 msgr2=0x7f888c0c72d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:56.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.643+0000 7f888b7fe700 1 --2- 172.21.15.98:0/433893211 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f888c0cbc30 0x7f888c0c72d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:56.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.643+0000 7f888b7fe700 1 -- 172.21.15.98:0/433893211 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f888c0b2100 msgr2=0x7f888c153c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:56.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.643+0000 7f888b7fe700 1 --2- 172.21.15.98:0/433893211 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f888c0b2100 0x7f888c153c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:56.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.643+0000 7f888b7fe700 1 -- 172.21.15.98:0/433893211 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8874009420 con 0x7f888c159440 2023-10-06T05:07:56.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.643+0000 7f888affd700 1 --2- 172.21.15.98:0/433893211 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f888c0b2100 0x7f888c153c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:07:56.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.643+0000 7f88827fc700 1 --2- 172.21.15.98:0/433893211 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f888c0cbc30 0x7f888c0c72d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:07:56.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.643+0000 7f888b7fe700 1 --2- 172.21.15.98:0/433893211 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f888c159440 0x7f888c0c7810 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f8884009fa0 tx=0x7f8884009900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:56.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.643+0000 7f88837fe700 1 -- 172.21.15.98:0/433893211 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f888400ed20 con 0x7f888c159440 2023-10-06T05:07:56.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.643+0000 7f88837fe700 1 -- 172.21.15.98:0/433893211 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f888400ee80 con 0x7f888c159440 2023-10-06T05:07:56.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.643+0000 7f88837fe700 1 -- 172.21.15.98:0/433893211 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8884014670 con 0x7f888c159440 2023-10-06T05:07:56.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.643+0000 7f888bfff700 1 -- 172.21.15.98:0/433893211 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f888c0c9a10 con 0x7f888c159440 2023-10-06T05:07:56.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.643+0000 7f888bfff700 1 -- 172.21.15.98:0/433893211 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f888c0ce0d0 con 0x7f888c159440 2023-10-06T05:07:56.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.644+0000 7f888bfff700 1 -- 172.21.15.98:0/433893211 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f888c05f230 con 0x7f888c159440 2023-10-06T05:07:56.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.644+0000 7f88837fe700 1 -- 172.21.15.98:0/433893211 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f88840149e0 con 0x7f888c159440 2023-10-06T05:07:56.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.645+0000 7f88837fe700 1 --2- 172.21.15.98:0/433893211 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f886c066f30 0x7f886c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:56.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.645+0000 7f88837fe700 1 -- 172.21.15.98:0/433893211 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6378+0+0 (secure 0 0 0) 0x7f8884084f70 con 0x7f888c159440 2023-10-06T05:07:56.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.645+0000 7f888affd700 1 --2- 172.21.15.98:0/433893211 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f886c066f30 0x7f886c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:56.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.645+0000 7f888affd700 1 --2- 172.21.15.98:0/433893211 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f886c066f30 0x7f886c0693e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f888c0cd6f0 tx=0x7f887c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:56.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.649+0000 7f88837fe700 1 -- 172.21.15.98:0/433893211 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8884051f40 con 0x7f888c159440 2023-10-06T05:07:56.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.834+0000 7f888bfff700 1 -- 172.21.15.98:0/433893211 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f888c159810 con 0x7f888c159440 2023-10-06T05:07:56.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.835+0000 7f88837fe700 1 -- 172.21.15.98:0/433893211 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) v1 ==== 74+0+17084 (secure 0 0 0) 0x7f8884055560 con 0x7f888c159440 2023-10-06T05:07:56.837 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:07:56.837 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":42,"fsid":"f50c5df5-cad5-47a4-9bb1-8c791d4311ec","created":"2023-10-06T05:06:32.971175+0000","modified":"2023-10-06T05:07:55.873162+0000","last_up_change":"2023-10-06T05:06:44.016489+0000","last_in_change":"2023-10-06T05:06:35.544511+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-06T05:06:44.559671+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-06T05:06:51.432161+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-10-06T05:07:54.125452+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-10-06T05:07:54.831073+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d9495d95-da75-4c3b-8927-e9bae6a8b093","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":90627},{"type":"v1","addr":"172.21.15.98:6801","nonce":90627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":90627},{"type":"v1","addr":"172.21.15.98:6803","nonce":90627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":90627},{"type":"v1","addr":"172.21.15.98:6807","nonce":90627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":90627},{"type":"v1","addr":"172.21.15.98:6805","nonce":90627}]},"public_addr":"172.21.15.98:6801/90627","cluster_addr":"172.21.15.98:6803/90627","heartbeat_back_addr":"172.21.15.98:6807/90627","heartbeat_front_addr":"172.21.15.98:6805/90627","state":["exists","up"]},{"osd":1,"uuid":"f25ba5be-d65e-4707-a134-154092deb754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":90628},{"type":"v1","addr":"172.21.15.98:6809","nonce":90628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":90628},{"type":"v1","addr":"172.21.15.98:6811","nonce":90628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":90628},{"type":"v1","addr":"172.21.15.98:6815","nonce":90628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":90628},{"type":"v1","addr":"172.21.15.98:6813","nonce":90628}]},"public_addr":"172.21.15.98:6809/90628","cluster_addr":"172.21.15.98:6811/90628","heartbeat_back_addr":"172.21.15.98:6815/90628","heartbeat_front_addr":"172.21.15.98:6813/90628","state":["exists","up"]},{"osd":2,"uuid":"5ab9e028-945c-43e2-8c95-3b0b4cf38e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":90662},{"type":"v1","addr":"172.21.15.98:6818","nonce":90662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":90662},{"type":"v1","addr":"172.21.15.98:6822","nonce":90662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":90662},{"type":"v1","addr":"172.21.15.98:6830","nonce":90662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":90662},{"type":"v1","addr":"172.21.15.98:6826","nonce":90662}]},"public_addr":"172.21.15.98:6818/90662","cluster_addr":"172.21.15.98:6822/90662","heartbeat_back_addr":"172.21.15.98:6830/90662","heartbeat_front_addr":"172.21.15.98:6826/90662","state":["exists","up"]},{"osd":3,"uuid":"3bb803ff-1738-46bf-be62-2267bf378822","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6817","nonce":90663},{"type":"v1","addr":"172.21.15.98:6819","nonce":90663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6821","nonce":90663},{"type":"v1","addr":"172.21.15.98:6823","nonce":90663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6829","nonce":90663},{"type":"v1","addr":"172.21.15.98:6831","nonce":90663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6825","nonce":90663},{"type":"v1","addr":"172.21.15.98:6827","nonce":90663}]},"public_addr":"172.21.15.98:6819/90663","cluster_addr":"172.21.15.98:6823/90663","heartbeat_back_addr":"172.21.15.98:6831/90663","heartbeat_front_addr":"172.21.15.98:6827/90663","state":["exists","up"]},{"osd":4,"uuid":"7d57a4c9-6462-43f6-b64a-7f9e9b4353d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":90847},{"type":"v1","addr":"172.21.15.174:6801","nonce":90847}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":90847},{"type":"v1","addr":"172.21.15.174:6803","nonce":90847}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":90847},{"type":"v1","addr":"172.21.15.174:6807","nonce":90847}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":90847},{"type":"v1","addr":"172.21.15.174:6805","nonce":90847}]},"public_addr":"172.21.15.174:6801/90847","cluster_addr":"172.21.15.174:6803/90847","heartbeat_back_addr":"172.21.15.174:6807/90847","heartbeat_front_addr":"172.21.15.174:6805/90847","state":["exists","up"]},{"osd":5,"uuid":"83287ea4-e44d-4af9-bc77-36d47c60d4a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":90848},{"type":"v1","addr":"172.21.15.174:6809","nonce":90848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":90848},{"type":"v1","addr":"172.21.15.174:6811","nonce":90848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":90848},{"type":"v1","addr":"172.21.15.174:6815","nonce":90848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":90848},{"type":"v1","addr":"172.21.15.174:6813","nonce":90848}]},"public_addr":"172.21.15.174:6809/90848","cluster_addr":"172.21.15.174:6811/90848","heartbeat_back_addr":"172.21.15.174:6815/90848","heartbeat_front_addr":"172.21.15.174:6813/90848","state":["exists","up"]},{"osd":6,"uuid":"496a6272-892e-43c4-a145-206623c22f99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":90877},{"type":"v1","addr":"172.21.15.174:6817","nonce":90877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":90877},{"type":"v1","addr":"172.21.15.174:6819","nonce":90877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":90877},{"type":"v1","addr":"172.21.15.174:6823","nonce":90877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":90877},{"type":"v1","addr":"172.21.15.174:6821","nonce":90877}]},"public_addr":"172.21.15.174:6817/90877","cluster_addr":"172.21.15.174:6819/90877","heartbeat_back_addr":"172.21.15.174:6823/90877","heartbeat_front_addr":"172.21.15.174:6821/90877","state":["exists","up"]},{"osd":7,"uuid":"49967e34-8b07-4bab-a13e-0c16d6a0f3a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":90878},{"type":"v1","addr":"172.21.15.174:6825","nonce":90878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":90878},{"type":"v1","addr":"172.21.15.174:6827","nonce":90878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":90878},{"type":"v1","addr":"172.21.15.174:6831","nonce":90878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":90878},{"type":"v1","addr":"172.21.15.174:6829","nonce":90878}]},"public_addr":"172.21.15.174:6825/90878","cluster_addr":"172.21.15.174:6827/90878","heartbeat_back_addr":"172.21.15.174:6831/90878","heartbeat_front_addr":"172.21.15.174:6829/90878","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.446479+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.514266+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.794108+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.810894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.731919+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.717756+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.737435+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.508682+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T05:07:56.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.836+0000 7f888bfff700 1 -- 172.21.15.98:0/433893211 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f886c066f30 msgr2=0x7f886c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:56.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.836+0000 7f888bfff700 1 --2- 172.21.15.98:0/433893211 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f886c066f30 0x7f886c0693e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f888c0cd6f0 tx=0x7f887c009040 comp rx=0 tx=0).stop 2023-10-06T05:07:56.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.836+0000 7f888bfff700 1 -- 172.21.15.98:0/433893211 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f888c159440 msgr2=0x7f888c0c7810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:56.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.836+0000 7f888bfff700 1 --2- 172.21.15.98:0/433893211 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f888c159440 0x7f888c0c7810 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f8884009fa0 tx=0x7f8884009900 comp rx=0 tx=0).stop 2023-10-06T05:07:56.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.836+0000 7f888bfff700 1 -- 172.21.15.98:0/433893211 shutdown_connections 2023-10-06T05:07:56.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.836+0000 7f888bfff700 1 --2- 172.21.15.98:0/433893211 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f886c066f30 0x7f886c0693e0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:56.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.836+0000 7f888bfff700 1 --2- 172.21.15.98:0/433893211 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f888c159440 0x7f888c0c7810 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:56.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.836+0000 7f888bfff700 1 --2- 172.21.15.98:0/433893211 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f888c0b2100 0x7f888c153c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:56.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.836+0000 7f888bfff700 1 --2- 172.21.15.98:0/433893211 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f888c0cbc30 0x7f888c0c72d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:56.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.836+0000 7f888bfff700 1 -- 172.21.15.98:0/433893211 >> 172.21.15.98:0/433893211 conn(0x7f888c06ece0 msgr2=0x7f888c158570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:56.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.837+0000 7f888bfff700 1 -- 172.21.15.98:0/433893211 shutdown_connections 2023-10-06T05:07:56.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.837+0000 7f888bfff700 1 -- 172.21.15.98:0/433893211 wait complete. 2023-10-06T05:07:56.851 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T05:07:56.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.964+0000 7f17ef11c700 1 Processor -- start 2023-10-06T05:07:56.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.964+0000 7f17ef11c700 1 -- start start 2023-10-06T05:07:56.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.965+0000 7f17ef11c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f17e8159560 0x7f17e80aa530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:56.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.965+0000 7f17ef11c700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f17e80b49a0 0x7f17e80aaa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:56.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.965+0000 7f17ef11c700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f17e80b0760 0x7f17e80b3890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:56.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.965+0000 7f17ef11c700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17e80b0c60 con 0x7f17e8159560 2023-10-06T05:07:56.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.965+0000 7f17ef11c700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17e80b0f40 con 0x7f17e80b0760 2023-10-06T05:07:56.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.965+0000 7f17ef11c700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17e80b1220 con 0x7f17e80b49a0 2023-10-06T05:07:56.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.965+0000 7f17eceb8700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f17e8159560 0x7f17e80aa530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:56.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.965+0000 7f17eceb8700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f17e8159560 0x7f17e80aa530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47614/0 (socket says 172.21.15.98:47614) 2023-10-06T05:07:56.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.965+0000 7f17eceb8700 1 -- 172.21.15.98:0/2458273495 learned_addr learned my addr 172.21.15.98:0/2458273495 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:56.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.965+0000 7f17eceb8700 1 -- 172.21.15.98:0/2458273495 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f17e80b49a0 msgr2=0x7f17e80aaa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:56.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.965+0000 7f17ed6b9700 1 --2- 172.21.15.98:0/2458273495 >> v2:172.21.15.174:3300/0 conn(0x7f17e80b0760 0x7f17e80b3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:56.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.965+0000 7f17dffff700 1 --2- 172.21.15.98:0/2458273495 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f17e80b49a0 0x7f17e80aaa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:56.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.966+0000 7f17eceb8700 1 --2- 172.21.15.98:0/2458273495 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f17e80b49a0 0x7f17e80aaa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:56.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.966+0000 7f17eceb8700 1 -- 172.21.15.98:0/2458273495 >> v2:172.21.15.174:3300/0 conn(0x7f17e80b0760 msgr2=0x7f17e80b3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:56.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.966+0000 7f17eceb8700 1 --2- 172.21.15.98:0/2458273495 >> v2:172.21.15.174:3300/0 conn(0x7f17e80b0760 0x7f17e80b3890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:56.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.966+0000 7f17eceb8700 1 -- 172.21.15.98:0/2458273495 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17e80b1500 con 0x7f17e8159560 2023-10-06T05:07:56.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.966+0000 7f17eceb8700 1 --2- 172.21.15.98:0/2458273495 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17e8159560 0x7f17e80aa530 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f17e80d0460 tx=0x7f17e4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a456a20255f49056 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:56.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.966+0000 7f17df7fe700 1 -- 172.21.15.98:0/2458273495 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17e400e070 con 0x7f17e8159560 2023-10-06T05:07:56.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.966+0000 7f17df7fe700 1 -- 172.21.15.98:0/2458273495 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17e400bad0 con 0x7f17e8159560 2023-10-06T05:07:56.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.966+0000 7f17df7fe700 1 -- 172.21.15.98:0/2458273495 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17e40092e0 con 0x7f17e8159560 2023-10-06T05:07:56.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.966+0000 7f17ef11c700 1 -- 172.21.15.98:0/2458273495 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17e8159560 msgr2=0x7f17e80aa530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:56.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.966+0000 7f17ef11c700 1 --2- 172.21.15.98:0/2458273495 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17e8159560 0x7f17e80aa530 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f17e80d0460 tx=0x7f17e4009840 comp rx=0 tx=0).stop 2023-10-06T05:07:56.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.966+0000 7f17ef11c700 1 -- 172.21.15.98:0/2458273495 shutdown_connections 2023-10-06T05:07:56.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.966+0000 7f17ef11c700 1 --2- 172.21.15.98:0/2458273495 >> v2:172.21.15.174:3300/0 conn(0x7f17e80b0760 0x7f17e80b3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:56.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.966+0000 7f17ef11c700 1 --2- 172.21.15.98:0/2458273495 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17e8159560 0x7f17e80aa530 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:56.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.966+0000 7f17ef11c700 1 --2- 172.21.15.98:0/2458273495 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f17e80b49a0 0x7f17e80aaa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:56.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.966+0000 7f17ef11c700 1 -- 172.21.15.98:0/2458273495 >> 172.21.15.98:0/2458273495 conn(0x7f17e80997f0 msgr2=0x7f17e806f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:56.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.966+0000 7f17ef11c700 1 -- 172.21.15.98:0/2458273495 shutdown_connections 2023-10-06T05:07:56.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.966+0000 7f17ef11c700 1 -- 172.21.15.98:0/2458273495 wait complete. 2023-10-06T05:07:56.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.967+0000 7f17ef11c700 1 Processor -- start 2023-10-06T05:07:56.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.968+0000 7f17ef11c700 1 -- start start 2023-10-06T05:07:56.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.968+0000 7f17ef11c700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f17e80b0760 0x7f17e815c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:56.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.968+0000 7f17ef11c700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f17e80b49a0 0x7f17e80ce8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:56.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.968+0000 7f17ef11c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17e8159560 0x7f17e8158390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:56.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.968+0000 7f17ef11c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17e81559e0 con 0x7f17e8159560 2023-10-06T05:07:56.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.968+0000 7f17ef11c700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17e8154dd0 con 0x7f17e80b49a0 2023-10-06T05:07:56.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.968+0000 7f17ef11c700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17e81550b0 con 0x7f17e80b0760 2023-10-06T05:07:56.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.969+0000 7f17ed6b9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17e8159560 0x7f17e8158390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:56.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.969+0000 7f17ed6b9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17e8159560 0x7f17e8158390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47622/0 (socket says 172.21.15.98:47622) 2023-10-06T05:07:56.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.969+0000 7f17ed6b9700 1 -- 172.21.15.98:0/592633895 learned_addr learned my addr 172.21.15.98:0/592633895 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:56.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.969+0000 7f17eceb8700 1 --2- 172.21.15.98:0/592633895 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f17e80b0760 0x7f17e815c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:56.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.969+0000 7f17dffff700 1 --2- 172.21.15.98:0/592633895 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f17e80b49a0 0x7f17e80ce8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:56.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.969+0000 7f17ed6b9700 1 -- 172.21.15.98:0/592633895 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f17e80b0760 msgr2=0x7f17e815c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:56.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.969+0000 7f17ed6b9700 1 --2- 172.21.15.98:0/592633895 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f17e80b0760 0x7f17e815c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:56.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.969+0000 7f17ed6b9700 1 -- 172.21.15.98:0/592633895 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f17e80b49a0 msgr2=0x7f17e80ce8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:56.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.969+0000 7f17ed6b9700 1 --2- 172.21.15.98:0/592633895 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f17e80b49a0 0x7f17e80ce8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:56.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.969+0000 7f17ed6b9700 1 -- 172.21.15.98:0/592633895 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17e4009ae0 con 0x7f17e8159560 2023-10-06T05:07:56.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.969+0000 7f17dffff700 1 --2- 172.21.15.98:0/592633895 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f17e80b49a0 0x7f17e80ce8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:07:56.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.969+0000 7f17ed6b9700 1 --2- 172.21.15.98:0/592633895 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17e8159560 0x7f17e8158390 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f17e000c880 tx=0x7f17e0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:56.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.969+0000 7f17dcff9700 1 -- 172.21.15.98:0/592633895 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17e0011600 con 0x7f17e8159560 2023-10-06T05:07:56.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.969+0000 7f17dcff9700 1 -- 172.21.15.98:0/592633895 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17e0011760 con 0x7f17e8159560 2023-10-06T05:07:56.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.969+0000 7f17ef11c700 1 -- 172.21.15.98:0/592633895 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17e8155390 con 0x7f17e8159560 2023-10-06T05:07:56.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.969+0000 7f17dcff9700 1 -- 172.21.15.98:0/592633895 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17e0011ae0 con 0x7f17e8159560 2023-10-06T05:07:56.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.969+0000 7f17ef11c700 1 -- 172.21.15.98:0/592633895 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17e81588d0 con 0x7f17e8159560 2023-10-06T05:07:56.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.970+0000 7f17ef11c700 1 -- 172.21.15.98:0/592633895 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17e8155d70 con 0x7f17e8159560 2023-10-06T05:07:56.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.970+0000 7f17dcff9700 1 -- 172.21.15.98:0/592633895 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f17e00193f0 con 0x7f17e8159560 2023-10-06T05:07:56.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.971+0000 7f17dcff9700 1 --2- 172.21.15.98:0/592633895 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f17d4066f30 0x7f17d40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:56.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.971+0000 7f17dcff9700 1 -- 172.21.15.98:0/592633895 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6378+0+0 (secure 0 0 0) 0x7f17e0084b10 con 0x7f17e8159560 2023-10-06T05:07:56.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.971+0000 7f17eceb8700 1 --2- 172.21.15.98:0/592633895 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f17d4066f30 0x7f17d40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:56.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.971+0000 7f17eceb8700 1 --2- 172.21.15.98:0/592633895 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f17d4066f30 0x7f17d40693e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f17e4009fd0 tx=0x7f17e4018000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:56.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:56.975+0000 7f17dcff9700 1 -- 172.21.15.98:0/592633895 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f17e8155d70 con 0x7f17e8159560 2023-10-06T05:07:57.000 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-06T05:07:57.002 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-06T05:07:57.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:57.164+0000 7f17ef11c700 1 -- 172.21.15.98:0/592633895 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f17e8155d70 con 0x7f17e8159560 2023-10-06T05:07:57.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:57.165+0000 7f17dcff9700 1 -- 172.21.15.98:0/592633895 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) v1 ==== 74+0+17084 (secure 0 0 0) 0x7f17e8155d70 con 0x7f17e8159560 2023-10-06T05:07:57.167 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:07:57.168 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":42,"fsid":"f50c5df5-cad5-47a4-9bb1-8c791d4311ec","created":"2023-10-06T05:06:32.971175+0000","modified":"2023-10-06T05:07:55.873162+0000","last_up_change":"2023-10-06T05:06:44.016489+0000","last_in_change":"2023-10-06T05:06:35.544511+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-06T05:06:44.559671+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-06T05:06:51.432161+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-10-06T05:07:54.125452+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-10-06T05:07:54.831073+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d9495d95-da75-4c3b-8927-e9bae6a8b093","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":90627},{"type":"v1","addr":"172.21.15.98:6801","nonce":90627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":90627},{"type":"v1","addr":"172.21.15.98:6803","nonce":90627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":90627},{"type":"v1","addr":"172.21.15.98:6807","nonce":90627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":90627},{"type":"v1","addr":"172.21.15.98:6805","nonce":90627}]},"public_addr":"172.21.15.98:6801/90627","cluster_addr":"172.21.15.98:6803/90627","heartbeat_back_addr":"172.21.15.98:6807/90627","heartbeat_front_addr":"172.21.15.98:6805/90627","state":["exists","up"]},{"osd":1,"uuid":"f25ba5be-d65e-4707-a134-154092deb754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":90628},{"type":"v1","addr":"172.21.15.98:6809","nonce":90628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":90628},{"type":"v1","addr":"172.21.15.98:6811","nonce":90628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":90628},{"type":"v1","addr":"172.21.15.98:6815","nonce":90628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":90628},{"type":"v1","addr":"172.21.15.98:6813","nonce":90628}]},"public_addr":"172.21.15.98:6809/90628","cluster_addr":"172.21.15.98:6811/90628","heartbeat_back_addr":"172.21.15.98:6815/90628","heartbeat_front_addr":"172.21.15.98:6813/90628","state":["exists","up"]},{"osd":2,"uuid":"5ab9e028-945c-43e2-8c95-3b0b4cf38e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":90662},{"type":"v1","addr":"172.21.15.98:6818","nonce":90662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":90662},{"type":"v1","addr":"172.21.15.98:6822","nonce":90662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":90662},{"type":"v1","addr":"172.21.15.98:6830","nonce":90662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":90662},{"type":"v1","addr":"172.21.15.98:6826","nonce":90662}]},"public_addr":"172.21.15.98:6818/90662","cluster_addr":"172.21.15.98:6822/90662","heartbeat_back_addr":"172.21.15.98:6830/90662","heartbeat_front_addr":"172.21.15.98:6826/90662","state":["exists","up"]},{"osd":3,"uuid":"3bb803ff-1738-46bf-be62-2267bf378822","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6817","nonce":90663},{"type":"v1","addr":"172.21.15.98:6819","nonce":90663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6821","nonce":90663},{"type":"v1","addr":"172.21.15.98:6823","nonce":90663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6829","nonce":90663},{"type":"v1","addr":"172.21.15.98:6831","nonce":90663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6825","nonce":90663},{"type":"v1","addr":"172.21.15.98:6827","nonce":90663}]},"public_addr":"172.21.15.98:6819/90663","cluster_addr":"172.21.15.98:6823/90663","heartbeat_back_addr":"172.21.15.98:6831/90663","heartbeat_front_addr":"172.21.15.98:6827/90663","state":["exists","up"]},{"osd":4,"uuid":"7d57a4c9-6462-43f6-b64a-7f9e9b4353d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":90847},{"type":"v1","addr":"172.21.15.174:6801","nonce":90847}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":90847},{"type":"v1","addr":"172.21.15.174:6803","nonce":90847}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":90847},{"type":"v1","addr":"172.21.15.174:6807","nonce":90847}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":90847},{"type":"v1","addr":"172.21.15.174:6805","nonce":90847}]},"public_addr":"172.21.15.174:6801/90847","cluster_addr":"172.21.15.174:6803/90847","heartbeat_back_addr":"172.21.15.174:6807/90847","heartbeat_front_addr":"172.21.15.174:6805/90847","state":["exists","up"]},{"osd":5,"uuid":"83287ea4-e44d-4af9-bc77-36d47c60d4a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":90848},{"type":"v1","addr":"172.21.15.174:6809","nonce":90848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":90848},{"type":"v1","addr":"172.21.15.174:6811","nonce":90848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":90848},{"type":"v1","addr":"172.21.15.174:6815","nonce":90848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":90848},{"type":"v1","addr":"172.21.15.174:6813","nonce":90848}]},"public_addr":"172.21.15.174:6809/90848","cluster_addr":"172.21.15.174:6811/90848","heartbeat_back_addr":"172.21.15.174:6815/90848","heartbeat_front_addr":"172.21.15.174:6813/90848","state":["exists","up"]},{"osd":6,"uuid":"496a6272-892e-43c4-a145-206623c22f99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":90877},{"type":"v1","addr":"172.21.15.174:6817","nonce":90877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":90877},{"type":"v1","addr":"172.21.15.174:6819","nonce":90877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":90877},{"type":"v1","addr":"172.21.15.174:6823","nonce":90877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":90877},{"type":"v1","addr":"172.21.15.174:6821","nonce":90877}]},"public_addr":"172.21.15.174:6817/90877","cluster_addr":"172.21.15.174:6819/90877","heartbeat_back_addr":"172.21.15.174:6823/90877","heartbeat_front_addr":"172.21.15.174:6821/90877","state":["exists","up"]},{"osd":7,"uuid":"49967e34-8b07-4bab-a13e-0c16d6a0f3a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":90878},{"type":"v1","addr":"172.21.15.174:6825","nonce":90878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":90878},{"type":"v1","addr":"172.21.15.174:6827","nonce":90878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":90878},{"type":"v1","addr":"172.21.15.174:6831","nonce":90878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":90878},{"type":"v1","addr":"172.21.15.174:6829","nonce":90878}]},"public_addr":"172.21.15.174:6825/90878","cluster_addr":"172.21.15.174:6827/90878","heartbeat_back_addr":"172.21.15.174:6831/90878","heartbeat_front_addr":"172.21.15.174:6829/90878","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.446479+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.514266+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.794108+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.810894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.731919+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.717756+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.737435+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.508682+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T05:07:57.169 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:57.167+0000 7f17ef11c700 1 -- 172.21.15.98:0/592633895 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f17d4066f30 msgr2=0x7f17d40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:57.169 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:57.167+0000 7f17ef11c700 1 --2- 172.21.15.98:0/592633895 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f17d4066f30 0x7f17d40693e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f17e4009fd0 tx=0x7f17e4018000 comp rx=0 tx=0).stop 2023-10-06T05:07:57.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:57.167+0000 7f17ef11c700 1 -- 172.21.15.98:0/592633895 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17e8159560 msgr2=0x7f17e8158390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:57.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:57.167+0000 7f17ef11c700 1 --2- 172.21.15.98:0/592633895 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17e8159560 0x7f17e8158390 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f17e000c880 tx=0x7f17e0008040 comp rx=0 tx=0).stop 2023-10-06T05:07:57.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:57.167+0000 7f17ef11c700 1 -- 172.21.15.98:0/592633895 shutdown_connections 2023-10-06T05:07:57.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:57.167+0000 7f17ef11c700 1 --2- 172.21.15.98:0/592633895 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f17d4066f30 0x7f17d40693e0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:57.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:57.167+0000 7f17ef11c700 1 --2- 172.21.15.98:0/592633895 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17e8159560 0x7f17e8158390 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:57.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:57.167+0000 7f17ef11c700 1 --2- 172.21.15.98:0/592633895 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f17e80b0760 0x7f17e815c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:57.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:57.167+0000 7f17ef11c700 1 --2- 172.21.15.98:0/592633895 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f17e80b49a0 0x7f17e80ce8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:57.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:57.167+0000 7f17ef11c700 1 -- 172.21.15.98:0/592633895 >> 172.21.15.98:0/592633895 conn(0x7f17e80997f0 msgr2=0x7f17e80cce90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:57.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:57.167+0000 7f17ef11c700 1 -- 172.21.15.98:0/592633895 shutdown_connections 2023-10-06T05:07:57.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:57.167+0000 7f17ef11c700 1 -- 172.21.15.98:0/592633895 wait complete. 2023-10-06T05:07:57.181 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-06T05:07:57.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:57.296+0000 7fa480fb2700 1 Processor -- start 2023-10-06T05:07:57.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:57.296+0000 7fa480fb2700 1 -- start start 2023-10-06T05:07:57.301 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:57.297+0000 7fa480fb2700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa47c0b0640 0x7fa47c0cecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:57.301 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:57.297+0000 7fa480fb2700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa47c0b0a10 0x7fa47c0cbee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:57.301 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:57.297+0000 7fa480fb2700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa47c0c53c0 con 0x7fa47c0cc420 2023-10-06T05:07:57.301 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:57.297+0000 7fa480fb2700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa47c0ca790 con 0x7fa47c0b0640 2023-10-06T05:07:57.301 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:57.297+0000 7fa480fb2700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa47c0ccd00 con 0x7fa47c0b0a10 2023-10-06T05:07:57.301 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:57.297+0000 7fa47a59c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa47c0b0640 0x7fa47c0cecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:57.301 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:57.297+0000 7fa47a59c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa47c0b0640 0x7fa47c0cecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47632/0 (socket says 172.21.15.98:47632) 2023-10-06T05:07:57.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:57.297+0000 7fa47a59c700 1 -- 172.21.15.98:0/4055742133 learned_addr learned my addr 172.21.15.98:0/4055742133 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:57.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:57.297+0000 7fa471d9b700 1 --2- 172.21.15.98:0/4055742133 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa47c0b0a10 0x7fa47c0cbee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:57.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:57.298+0000 7fa47a59c700 1 -- 172.21.15.98:0/4055742133 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa47c0b0a10 msgr2=0x7fa47c0cbee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:57.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:57.298+0000 7fa47a59c700 1 --2- 172.21.15.98:0/4055742133 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa47c0b0a10 0x7fa47c0cbee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:57.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:57.298+0000 7fa47a59c700 1 -- 172.21.15.98:0/4055742133 >> v1:172.21.15.174:6789/0 conn(0x7fa47c0cc420 legacy=0x7fa47c0c3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:57.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:57.298+0000 7fa47a59c700 1 -- 172.21.15.98:0/4055742133 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa47c0ccfe0 con 0x7fa47c0b0640 2023-10-06T05:07:57.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:57.298+0000 7fa47a59c700 1 --2- 172.21.15.98:0/4055742133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa47c0b0640 0x7fa47c0cecd0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fa46c009990 tx=0x7fa46c009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1566b319bd566b1f server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:57.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:57.298+0000 7fa479d9b700 1 -- 172.21.15.98:0/4055742133 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa46c005c10 con 0x7fa47c0b0640 2023-10-06T05:07:57.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:57.298+0000 7fa479d9b700 1 -- 172.21.15.98:0/4055742133 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa46c005d70 con 0x7fa47c0b0640 2023-10-06T05:07:57.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:57.298+0000 7fa479d9b700 1 -- 172.21.15.98:0/4055742133 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa46c00b880 con 0x7fa47c0b0640 2023-10-06T05:07:57.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:57.298+0000 7fa480fb2700 1 -- 172.21.15.98:0/4055742133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa47c0b0640 msgr2=0x7fa47c0cecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:57.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:57.298+0000 7fa480fb2700 1 --2- 172.21.15.98:0/4055742133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa47c0b0640 0x7fa47c0cecd0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fa46c009990 tx=0x7fa46c009220 comp rx=0 tx=0).stop 2023-10-06T05:07:57.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:57.298+0000 7fa480fb2700 1 -- 172.21.15.98:0/4055742133 shutdown_connections 2023-10-06T05:07:57.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:57.298+0000 7fa480fb2700 1 --2- 172.21.15.98:0/4055742133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa47c0b0640 0x7fa47c0cecd0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:57.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:57.298+0000 7fa480fb2700 1 --2- 172.21.15.98:0/4055742133 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa47c0b0a10 0x7fa47c0cbee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:57.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:57.298+0000 7fa480fb2700 1 -- 172.21.15.98:0/4055742133 >> 172.21.15.98:0/4055742133 conn(0x7fa47c087fc0 msgr2=0x7fa47c0ce310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:57.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:57.298+0000 7fa480fb2700 1 -- 172.21.15.98:0/4055742133 shutdown_connections 2023-10-06T05:07:57.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:57.298+0000 7fa480fb2700 1 -- 172.21.15.98:0/4055742133 wait complete. 2023-10-06T05:07:57.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:57.300+0000 7fa480fb2700 1 Processor -- start 2023-10-06T05:07:57.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:57.300+0000 7fa480fb2700 1 -- start start 2023-10-06T05:07:57.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:57.300+0000 7fa480fb2700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa47c0b0640 0x7fa47c155460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:57.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:57.300+0000 7fa480fb2700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa47c0b0a10 0x7fa47c159100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:57.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:57.300+0000 7fa480fb2700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa47c0cc420 0x7fa47c157ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:57.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:57.300+0000 7fa480fb2700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa47c098a00 con 0x7fa47c0cc420 2023-10-06T05:07:57.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:57.300+0000 7fa480fb2700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa47c09b110 con 0x7fa47c0b0a10 2023-10-06T05:07:57.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:57.300+0000 7fa480fb2700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa47c158880 con 0x7fa47c0b0640 2023-10-06T05:07:57.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:57.300+0000 7fa47ad9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa47c0cc420 0x7fa47c157ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:57.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:57.301+0000 7fa47ad9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa47c0cc420 0x7fa47c157ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47636/0 (socket says 172.21.15.98:47636) 2023-10-06T05:07:57.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:57.301+0000 7fa47ad9d700 1 -- 172.21.15.98:0/3701144362 learned_addr learned my addr 172.21.15.98:0/3701144362 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:57.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:57.301+0000 7fa47a59c700 1 --2- 172.21.15.98:0/3701144362 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa47c0b0640 0x7fa47c155460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:57.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:57.301+0000 7fa471d9b700 1 --2- 172.21.15.98:0/3701144362 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa47c0b0a10 0x7fa47c159100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:57.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:57.301+0000 7fa47ad9d700 1 -- 172.21.15.98:0/3701144362 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa47c0b0640 msgr2=0x7fa47c155460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:57.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:57.301+0000 7fa47ad9d700 1 --2- 172.21.15.98:0/3701144362 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa47c0b0640 0x7fa47c155460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:57.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:57.301+0000 7fa47ad9d700 1 -- 172.21.15.98:0/3701144362 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa47c0b0a10 msgr2=0x7fa47c159100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:57.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:57.301+0000 7fa47ad9d700 1 --2- 172.21.15.98:0/3701144362 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa47c0b0a10 0x7fa47c159100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:57.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:57.301+0000 7fa47ad9d700 1 -- 172.21.15.98:0/3701144362 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa46c009480 con 0x7fa47c0cc420 2023-10-06T05:07:57.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:57.301+0000 7fa471d9b700 1 --2- 172.21.15.98:0/3701144362 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa47c0b0a10 0x7fa47c159100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:07:57.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:57.301+0000 7fa47ad9d700 1 --2- 172.21.15.98:0/3701144362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa47c0cc420 0x7fa47c157ca0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fa47400c840 tx=0x7fa474001670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:57.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:57.301+0000 7fa472ffd700 1 -- 172.21.15.98:0/3701144362 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa474012630 con 0x7fa47c0cc420 2023-10-06T05:07:57.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:57.301+0000 7fa472ffd700 1 -- 172.21.15.98:0/3701144362 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4740127b0 con 0x7fa47c0cc420 2023-10-06T05:07:57.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:57.301+0000 7fa480fb2700 1 -- 172.21.15.98:0/3701144362 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa47c158b60 con 0x7fa47c0cc420 2023-10-06T05:07:57.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:57.301+0000 7fa472ffd700 1 -- 172.21.15.98:0/3701144362 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa474012b70 con 0x7fa47c0cc420 2023-10-06T05:07:57.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:57.301+0000 7fa480fb2700 1 -- 172.21.15.98:0/3701144362 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa47c09ada0 con 0x7fa47c0cc420 2023-10-06T05:07:57.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:57.302+0000 7fa480fb2700 1 -- 172.21.15.98:0/3701144362 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa47c09a920 con 0x7fa47c0cc420 2023-10-06T05:07:57.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:57.302+0000 7fa472ffd700 1 -- 172.21.15.98:0/3701144362 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fa474003030 con 0x7fa47c0cc420 2023-10-06T05:07:57.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:57.303+0000 7fa472ffd700 1 --2- 172.21.15.98:0/3701144362 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fa45c066fb0 0x7fa45c069460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:57.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:57.303+0000 7fa472ffd700 1 -- 172.21.15.98:0/3701144362 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6378+0+0 (secure 0 0 0) 0x7fa474084740 con 0x7fa47c0cc420 2023-10-06T05:07:57.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:57.303+0000 7fa47a59c700 1 --2- 172.21.15.98:0/3701144362 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fa45c066fb0 0x7fa45c069460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:57.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:57.303+0000 7fa47a59c700 1 --2- 172.21.15.98:0/3701144362 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fa45c066fb0 0x7fa45c069460 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fa46c009260 tx=0x7fa46c016000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:57.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:57.307+0000 7fa472ffd700 1 -- 172.21.15.98:0/3701144362 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa47c09a920 con 0x7fa47c0cc420 2023-10-06T05:07:57.538 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:57.536+0000 7fa480fb2700 1 -- 172.21.15.98:0/3701144362 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fa47c09a920 con 0x7fa47c0cc420 2023-10-06T05:07:58.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:58.511+0000 7fa472ffd700 1 -- 172.21.15.98:0/3701144362 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v16) v1 ==== 118+0+0 (secure 0 0 0) 0x7fa47c09a920 con 0x7fa47c0cc420 2023-10-06T05:07:58.517 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:58.513+0000 7fa480fb2700 1 -- 172.21.15.98:0/3701144362 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fa45c066fb0 msgr2=0x7fa45c069460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:58.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:58.513+0000 7fa480fb2700 1 --2- 172.21.15.98:0/3701144362 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fa45c066fb0 0x7fa45c069460 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fa46c009260 tx=0x7fa46c016000 comp rx=0 tx=0).stop 2023-10-06T05:07:58.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:58.513+0000 7fa480fb2700 1 -- 172.21.15.98:0/3701144362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa47c0cc420 msgr2=0x7fa47c157ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:58.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:58.513+0000 7fa480fb2700 1 --2- 172.21.15.98:0/3701144362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa47c0cc420 0x7fa47c157ca0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fa47400c840 tx=0x7fa474001670 comp rx=0 tx=0).stop 2023-10-06T05:07:58.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:58.514+0000 7fa480fb2700 1 -- 172.21.15.98:0/3701144362 shutdown_connections 2023-10-06T05:07:58.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:58.514+0000 7fa480fb2700 1 --2- 172.21.15.98:0/3701144362 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fa45c066fb0 0x7fa45c069460 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:58.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:58.514+0000 7fa480fb2700 1 --2- 172.21.15.98:0/3701144362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa47c0cc420 0x7fa47c157ca0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:58.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:58.514+0000 7fa480fb2700 1 --2- 172.21.15.98:0/3701144362 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa47c0b0640 0x7fa47c155460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:58.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:58.514+0000 7fa480fb2700 1 --2- 172.21.15.98:0/3701144362 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa47c0b0a10 0x7fa47c159100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:58.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:58.514+0000 7fa480fb2700 1 -- 172.21.15.98:0/3701144362 >> 172.21.15.98:0/3701144362 conn(0x7fa47c087fc0 msgr2=0x7fa47c0c9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:58.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:58.514+0000 7fa480fb2700 1 -- 172.21.15.98:0/3701144362 shutdown_connections 2023-10-06T05:07:58.519 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:58.514+0000 7fa480fb2700 1 -- 172.21.15.98:0/3701144362 wait complete. 2023-10-06T05:07:58.530 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-06T05:07:58.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:58.651+0000 7fd1bdbc0700 1 Processor -- start 2023-10-06T05:07:58.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:58.651+0000 7fd1bdbc0700 1 -- start start 2023-10-06T05:07:58.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:58.652+0000 7fd1bdbc0700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd1b80cd3c0 0x7fd1b80ceaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:58.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:58.652+0000 7fd1bdbc0700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fd1b80b29f0 0x7fd1b80cefe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:58.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:58.652+0000 7fd1bdbc0700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1b80cc970 con 0x7fd1b80b2620 2023-10-06T05:07:58.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:58.652+0000 7fd1bdbc0700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1b80cf520 con 0x7fd1b80cd3c0 2023-10-06T05:07:58.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:58.652+0000 7fd1bdbc0700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1b80cbd60 con 0x7fd1b80b29f0 2023-10-06T05:07:58.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:58.652+0000 7fd1bcbbe700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fd1b80b2620 0x7fd1b80cccb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:55772/0 (socket says 172.21.15.98:55772) 2023-10-06T05:07:58.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:58.652+0000 7fd1b7fff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd1b80cd3c0 0x7fd1b80ceaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:58.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:58.652+0000 7fd1bcbbe700 1 -- 172.21.15.98:0/1260852689 learned_addr learned my addr 172.21.15.98:0/1260852689 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:07:58.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:58.652+0000 7fd1bd3bf700 1 --2- 172.21.15.98:0/1260852689 >> v2:172.21.15.174:3300/0 conn(0x7fd1b80b29f0 0x7fd1b80cefe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:58.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:58.652+0000 7fd1b7fff700 1 -- 172.21.15.98:0/1260852689 >> v2:172.21.15.174:3300/0 conn(0x7fd1b80b29f0 msgr2=0x7fd1b80cefe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:58.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:58.652+0000 7fd1b7fff700 1 --2- 172.21.15.98:0/1260852689 >> v2:172.21.15.174:3300/0 conn(0x7fd1b80b29f0 0x7fd1b80cefe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:58.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:58.652+0000 7fd1b7fff700 1 -- 172.21.15.98:0/1260852689 >> v1:172.21.15.98:6789/0 conn(0x7fd1b80b2620 legacy=0x7fd1b80cccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:58.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:58.653+0000 7fd1b7fff700 1 -- 172.21.15.98:0/1260852689 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1b80cc040 con 0x7fd1b80cd3c0 2023-10-06T05:07:58.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:58.653+0000 7fd1b77fe700 1 -- 172.21.15.98:0/1260852689 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1955276982 0 0) 0x7fd1b80cc970 con 0x7fd1b80b2620 2023-10-06T05:07:58.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:58.653+0000 7fd1bd3bf700 1 --2- 172.21.15.98:0/1260852689 >> v2:172.21.15.174:3300/0 conn(0x7fd1b80b29f0 0x7fd1b80cefe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:07:58.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:58.653+0000 7fd1b7fff700 1 --2- 172.21.15.98:0/1260852689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1b80cd3c0 0x7fd1b80ceaa0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fd1b8099b90 tx=0x7fd1a40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53c596665315081 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:58.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:58.653+0000 7fd1b77fe700 1 -- 172.21.15.98:0/1260852689 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a400b6b0 con 0x7fd1b80cd3c0 2023-10-06T05:07:58.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:58.653+0000 7fd1b77fe700 1 -- 172.21.15.98:0/1260852689 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1a400b810 con 0x7fd1b80cd3c0 2023-10-06T05:07:58.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:58.653+0000 7fd1b77fe700 1 -- 172.21.15.98:0/1260852689 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a400bbb0 con 0x7fd1b80cd3c0 2023-10-06T05:07:58.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:58.653+0000 7fd1bdbc0700 1 -- 172.21.15.98:0/1260852689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1b80cd3c0 msgr2=0x7fd1b80ceaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:58.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:58.653+0000 7fd1bdbc0700 1 --2- 172.21.15.98:0/1260852689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1b80cd3c0 0x7fd1b80ceaa0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fd1b8099b90 tx=0x7fd1a40091a0 comp rx=0 tx=0).stop 2023-10-06T05:07:58.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:58.653+0000 7fd1bdbc0700 1 -- 172.21.15.98:0/1260852689 shutdown_connections 2023-10-06T05:07:58.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:58.653+0000 7fd1bdbc0700 1 --2- 172.21.15.98:0/1260852689 >> v2:172.21.15.174:3300/0 conn(0x7fd1b80b29f0 0x7fd1b80cefe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:58.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:58.653+0000 7fd1bdbc0700 1 --2- 172.21.15.98:0/1260852689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1b80cd3c0 0x7fd1b80ceaa0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:58.659 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:58.653+0000 7fd1bdbc0700 1 -- 172.21.15.98:0/1260852689 >> 172.21.15.98:0/1260852689 conn(0x7fd1b8099520 msgr2=0x7fd1b806f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:58.659 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:58.653+0000 7fd1bdbc0700 1 -- 172.21.15.98:0/1260852689 shutdown_connections 2023-10-06T05:07:58.659 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:58.653+0000 7fd1bdbc0700 1 -- 172.21.15.98:0/1260852689 wait complete. 2023-10-06T05:07:58.659 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:58.654+0000 7fd1bdbc0700 1 Processor -- start 2023-10-06T05:07:58.659 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:58.655+0000 7fd1bdbc0700 1 -- start start 2023-10-06T05:07:58.661 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:58.655+0000 7fd1bdbc0700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd1b80b2620 0x7fd1b8146e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:58.661 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:58.655+0000 7fd1bdbc0700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd1b80b29f0 0x7fd1b8169820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:58.661 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:58.655+0000 7fd1bdbc0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1b80cd3c0 0x7fd1b8159bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:58.661 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:58.655+0000 7fd1bdbc0700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b816a860 con 0x7fd1b80cd3c0 2023-10-06T05:07:58.661 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:58.655+0000 7fd1bdbc0700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b80b4570 con 0x7fd1b80b2620 2023-10-06T05:07:58.661 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:58.655+0000 7fd1bdbc0700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b81473b0 con 0x7fd1b80b29f0 2023-10-06T05:07:58.661 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:58.655+0000 7fd1bd3bf700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1b80cd3c0 0x7fd1b8159bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:58.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:58.655+0000 7fd1bd3bf700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1b80cd3c0 0x7fd1b8159bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47652/0 (socket says 172.21.15.98:47652) 2023-10-06T05:07:58.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:58.655+0000 7fd1bd3bf700 1 -- 172.21.15.98:0/952395351 learned_addr learned my addr 172.21.15.98:0/952395351 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:58.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:58.656+0000 7fd1bcbbe700 1 --2- 172.21.15.98:0/952395351 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd1b80b2620 0x7fd1b8146e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:58.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:58.656+0000 7fd1b7fff700 1 --2- 172.21.15.98:0/952395351 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd1b80b29f0 0x7fd1b8169820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:58.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:58.656+0000 7fd1bd3bf700 1 -- 172.21.15.98:0/952395351 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd1b80b29f0 msgr2=0x7fd1b8169820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:58.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:58.656+0000 7fd1bd3bf700 1 --2- 172.21.15.98:0/952395351 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd1b80b29f0 0x7fd1b8169820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:58.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:58.656+0000 7fd1bd3bf700 1 -- 172.21.15.98:0/952395351 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd1b80b2620 msgr2=0x7fd1b8146e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:58.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:58.656+0000 7fd1bd3bf700 1 --2- 172.21.15.98:0/952395351 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd1b80b2620 0x7fd1b8146e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:58.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:58.656+0000 7fd1bd3bf700 1 -- 172.21.15.98:0/952395351 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1a4009420 con 0x7fd1b80cd3c0 2023-10-06T05:07:58.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:58.656+0000 7fd1bd3bf700 1 --2- 172.21.15.98:0/952395351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1b80cd3c0 0x7fd1b8159bc0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fd1b000a380 tx=0x7fd1b00078d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:58.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:58.656+0000 7fd1b4ff9700 1 -- 172.21.15.98:0/952395351 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1b000ae10 con 0x7fd1b80cd3c0 2023-10-06T05:07:58.663 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:58.656+0000 7fd1b4ff9700 1 -- 172.21.15.98:0/952395351 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1b0008540 con 0x7fd1b80cd3c0 2023-10-06T05:07:58.663 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:58.656+0000 7fd1bdbc0700 1 -- 172.21.15.98:0/952395351 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1b816a2a0 con 0x7fd1b80cd3c0 2023-10-06T05:07:58.663 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:58.656+0000 7fd1b4ff9700 1 -- 172.21.15.98:0/952395351 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1b0008900 con 0x7fd1b80cd3c0 2023-10-06T05:07:58.663 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:58.656+0000 7fd1bdbc0700 1 -- 172.21.15.98:0/952395351 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1b80b4200 con 0x7fd1b80cd3c0 2023-10-06T05:07:58.663 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:58.657+0000 7fd1bdbc0700 1 -- 172.21.15.98:0/952395351 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1b80b4060 con 0x7fd1b80cd3c0 2023-10-06T05:07:58.663 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:58.657+0000 7fd1b4ff9700 1 -- 172.21.15.98:0/952395351 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fd1b0008cb0 con 0x7fd1b80cd3c0 2023-10-06T05:07:58.663 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:58.658+0000 7fd1b4ff9700 1 --2- 172.21.15.98:0/952395351 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fd19c066f30 0x7fd19c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:58.663 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:58.658+0000 7fd1b4ff9700 1 -- 172.21.15.98:0/952395351 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6378+0+0 (secure 0 0 0) 0x7fd1b008dd10 con 0x7fd1b80cd3c0 2023-10-06T05:07:58.663 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:58.658+0000 7fd1bcbbe700 1 --2- 172.21.15.98:0/952395351 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fd19c066f30 0x7fd19c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:58.663 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:58.658+0000 7fd1bcbbe700 1 --2- 172.21.15.98:0/952395351 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fd19c066f30 0x7fd19c0693e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac002800 tx=0x7fd1ac009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:58.664 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:58.662+0000 7fd1b4ff9700 1 -- 172.21.15.98:0/952395351 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd1b80b4060 con 0x7fd1b80cd3c0 2023-10-06T05:07:58.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:58.887+0000 7fd1bdbc0700 1 -- 172.21.15.98:0/952395351 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fd1b80b4060 con 0x7fd1b80cd3c0 2023-10-06T05:07:59.258 INFO:tasks.ceph.mds.c.smithi098.stderr:2023-10-06T05:07:59.256+0000 7f4a2ed1c700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-06T05:07:59.517 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.515+0000 7fd1b4ff9700 1 -- 172.21.15.98:0/952395351 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v17) v1 ==== 122+0+0 (secure 0 0 0) 0x7fd1b80b4060 con 0x7fd1b80cd3c0 2023-10-06T05:07:59.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.517+0000 7fd1bdbc0700 1 -- 172.21.15.98:0/952395351 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fd19c066f30 msgr2=0x7fd19c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:59.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.517+0000 7fd1bdbc0700 1 --2- 172.21.15.98:0/952395351 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fd19c066f30 0x7fd19c0693e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac002800 tx=0x7fd1ac009040 comp rx=0 tx=0).stop 2023-10-06T05:07:59.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.517+0000 7fd1bdbc0700 1 -- 172.21.15.98:0/952395351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1b80cd3c0 msgr2=0x7fd1b8159bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:59.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.517+0000 7fd1bdbc0700 1 --2- 172.21.15.98:0/952395351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1b80cd3c0 0x7fd1b8159bc0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fd1b000a380 tx=0x7fd1b00078d0 comp rx=0 tx=0).stop 2023-10-06T05:07:59.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.518+0000 7fd1bdbc0700 1 -- 172.21.15.98:0/952395351 shutdown_connections 2023-10-06T05:07:59.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.518+0000 7fd1bdbc0700 1 --2- 172.21.15.98:0/952395351 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fd19c066f30 0x7fd19c0693e0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:59.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.518+0000 7fd1bdbc0700 1 --2- 172.21.15.98:0/952395351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1b80cd3c0 0x7fd1b8159bc0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:59.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.518+0000 7fd1bdbc0700 1 --2- 172.21.15.98:0/952395351 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd1b80b2620 0x7fd1b8146e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:59.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.518+0000 7fd1bdbc0700 1 --2- 172.21.15.98:0/952395351 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd1b80b29f0 0x7fd1b8169820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:59.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.518+0000 7fd1bdbc0700 1 -- 172.21.15.98:0/952395351 >> 172.21.15.98:0/952395351 conn(0x7fd1b8099520 msgr2=0x7fd1b80c6970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:59.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.518+0000 7fd1bdbc0700 1 -- 172.21.15.98:0/952395351 shutdown_connections 2023-10-06T05:07:59.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.518+0000 7fd1bdbc0700 1 -- 172.21.15.98:0/952395351 wait complete. 2023-10-06T05:07:59.533 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-06T05:07:59.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.647+0000 7fc55ce45700 1 Processor -- start 2023-10-06T05:07:59.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.647+0000 7fc55ce45700 1 -- start start 2023-10-06T05:07:59.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.648+0000 7fc55ce45700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc558159440 0x7fc5580b3090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:59.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.648+0000 7fc55ce45700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc5580b35d0 0x7fc5580cbc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:59.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.648+0000 7fc55ce45700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5580cb7a0 con 0x7fc5580cc170 2023-10-06T05:07:59.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.648+0000 7fc55ce45700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5580b0860 con 0x7fc5580b35d0 2023-10-06T05:07:59.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.648+0000 7fc55ce45700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5580cb150 con 0x7fc558159440 2023-10-06T05:07:59.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.648+0000 7fc556ffd700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc5580b35d0 0x7fc5580cbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:59.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.648+0000 7fc556ffd700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc5580b35d0 0x7fc5580cbc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54746/0 (socket says 172.21.15.98:54746) 2023-10-06T05:07:59.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.648+0000 7fc556ffd700 1 -- 172.21.15.98:0/3291078661 learned_addr learned my addr 172.21.15.98:0/3291078661 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:59.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.648+0000 7fc5577fe700 1 --2- 172.21.15.98:0/3291078661 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc558159440 0x7fc5580b3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:59.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.648+0000 7fc556ffd700 1 -- 172.21.15.98:0/3291078661 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc558159440 msgr2=0x7fc5580b3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:07:59.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.648+0000 7fc556ffd700 1 --2- 172.21.15.98:0/3291078661 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc558159440 0x7fc5580b3090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:59.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.648+0000 7fc556ffd700 1 -- 172.21.15.98:0/3291078661 >> v1:172.21.15.174:6789/0 conn(0x7fc5580cc170 legacy=0x7fc5580b2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:59.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.648+0000 7fc556ffd700 1 -- 172.21.15.98:0/3291078661 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5580cb430 con 0x7fc5580b35d0 2023-10-06T05:07:59.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.648+0000 7fc556ffd700 1 --2- 172.21.15.98:0/3291078661 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5580b35d0 0x7fc5580cbc30 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fc54c0099b0 tx=0x7fc54c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a0914abec34bc8c server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:59.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.649+0000 7fc5567fc700 1 -- 172.21.15.98:0/3291078661 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc54c00b860 con 0x7fc5580b35d0 2023-10-06T05:07:59.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.649+0000 7fc5567fc700 1 -- 172.21.15.98:0/3291078661 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc54c00b9c0 con 0x7fc5580b35d0 2023-10-06T05:07:59.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.649+0000 7fc5567fc700 1 -- 172.21.15.98:0/3291078661 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc54c00bd60 con 0x7fc5580b35d0 2023-10-06T05:07:59.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.649+0000 7fc55ce45700 1 -- 172.21.15.98:0/3291078661 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5580b35d0 msgr2=0x7fc5580cbc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:59.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.649+0000 7fc55ce45700 1 --2- 172.21.15.98:0/3291078661 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5580b35d0 0x7fc5580cbc30 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fc54c0099b0 tx=0x7fc54c009250 comp rx=0 tx=0).stop 2023-10-06T05:07:59.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.649+0000 7fc55ce45700 1 -- 172.21.15.98:0/3291078661 shutdown_connections 2023-10-06T05:07:59.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.649+0000 7fc55ce45700 1 --2- 172.21.15.98:0/3291078661 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc558159440 0x7fc5580b3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:59.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.649+0000 7fc55ce45700 1 --2- 172.21.15.98:0/3291078661 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5580b35d0 0x7fc5580cbc30 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:59.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.649+0000 7fc55ce45700 1 -- 172.21.15.98:0/3291078661 >> 172.21.15.98:0/3291078661 conn(0x7fc55806ece0 msgr2=0x7fc55806f0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:59.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.649+0000 7fc55ce45700 1 -- 172.21.15.98:0/3291078661 shutdown_connections 2023-10-06T05:07:59.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.649+0000 7fc55ce45700 1 -- 172.21.15.98:0/3291078661 wait complete. 2023-10-06T05:07:59.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.650+0000 7fc55ce45700 1 Processor -- start 2023-10-06T05:07:59.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.650+0000 7fc55ce45700 1 -- start start 2023-10-06T05:07:59.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.651+0000 7fc55ce45700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc5580b35d0 0x7fc5580d0050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:59.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.651+0000 7fc55ce45700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5580cc170 0x7fc5580c9af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:59.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.651+0000 7fc55ce45700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc558159440 0x7fc5580ca030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:59.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.651+0000 7fc55ce45700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5580cf6f0 con 0x7fc5580cc170 2023-10-06T05:07:59.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.651+0000 7fc55ce45700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5580cee50 con 0x7fc5580b35d0 2023-10-06T05:07:59.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.651+0000 7fc55ce45700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5580cf130 con 0x7fc558159440 2023-10-06T05:07:59.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.651+0000 7fc556ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5580cc170 0x7fc5580c9af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:59.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.651+0000 7fc556ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5580cc170 0x7fc5580c9af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54762/0 (socket says 172.21.15.98:54762) 2023-10-06T05:07:59.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.651+0000 7fc556ffd700 1 -- 172.21.15.98:0/935506205 learned_addr learned my addr 172.21.15.98:0/935506205 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:07:59.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.651+0000 7fc5577fe700 1 --2- 172.21.15.98:0/935506205 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc5580b35d0 0x7fc5580d0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:59.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.651+0000 7fc557fff700 1 --2- 172.21.15.98:0/935506205 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc558159440 0x7fc5580ca030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:59.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.652+0000 7fc556ffd700 1 -- 172.21.15.98:0/935506205 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc558159440 msgr2=0x7fc5580ca030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:59.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.652+0000 7fc556ffd700 1 --2- 172.21.15.98:0/935506205 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc558159440 0x7fc5580ca030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:59.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.652+0000 7fc556ffd700 1 -- 172.21.15.98:0/935506205 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc5580b35d0 msgr2=0x7fc5580d0050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:59.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.652+0000 7fc556ffd700 1 --2- 172.21.15.98:0/935506205 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc5580b35d0 0x7fc5580d0050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:59.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.652+0000 7fc556ffd700 1 -- 172.21.15.98:0/935506205 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc54c0094f0 con 0x7fc5580cc170 2023-10-06T05:07:59.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.652+0000 7fc557fff700 1 --2- 172.21.15.98:0/935506205 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc558159440 0x7fc5580ca030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:07:59.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.652+0000 7fc5577fe700 1 --2- 172.21.15.98:0/935506205 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc5580b35d0 0x7fc5580d0050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:07:59.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.652+0000 7fc556ffd700 1 --2- 172.21.15.98:0/935506205 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5580cc170 0x7fc5580c9af0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fc558069150 tx=0x7fc54c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:59.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.652+0000 7fc53f7fe700 1 -- 172.21.15.98:0/935506205 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc54c024440 con 0x7fc5580cc170 2023-10-06T05:07:59.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.652+0000 7fc53f7fe700 1 -- 172.21.15.98:0/935506205 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc54c0245a0 con 0x7fc5580cc170 2023-10-06T05:07:59.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.652+0000 7fc55ce45700 1 -- 172.21.15.98:0/935506205 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5580cf410 con 0x7fc5580cc170 2023-10-06T05:07:59.659 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.652+0000 7fc53f7fe700 1 -- 172.21.15.98:0/935506205 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc54c024940 con 0x7fc5580cc170 2023-10-06T05:07:59.659 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.652+0000 7fc55ce45700 1 -- 172.21.15.98:0/935506205 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5580d0590 con 0x7fc5580cc170 2023-10-06T05:07:59.659 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.653+0000 7fc55ce45700 1 -- 172.21.15.98:0/935506205 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5580c7b00 con 0x7fc5580cc170 2023-10-06T05:07:59.659 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.653+0000 7fc53f7fe700 1 -- 172.21.15.98:0/935506205 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fc54c01a420 con 0x7fc5580cc170 2023-10-06T05:07:59.659 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.654+0000 7fc53f7fe700 1 --2- 172.21.15.98:0/935506205 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fc544066f30 0x7fc5440693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:07:59.659 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.654+0000 7fc53f7fe700 1 -- 172.21.15.98:0/935506205 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6378+0+0 (secure 0 0 0) 0x7fc54c0865c0 con 0x7fc5580cc170 2023-10-06T05:07:59.659 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.654+0000 7fc5577fe700 1 --2- 172.21.15.98:0/935506205 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fc544066f30 0x7fc5440693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:07:59.659 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.654+0000 7fc5577fe700 1 --2- 172.21.15.98:0/935506205 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fc544066f30 0x7fc5440693e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fc548009fd0 tx=0x7fc548009740 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:07:59.660 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.658+0000 7fc53f7fe700 1 -- 172.21.15.98:0/935506205 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc5580c7b00 con 0x7fc5580cc170 2023-10-06T05:07:59.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.887+0000 7fc55ce45700 1 -- 172.21.15.98:0/935506205 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc5580c7b00 con 0x7fc5580cc170 2023-10-06T05:07:59.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.887+0000 7fc53f7fe700 1 -- 172.21.15.98:0/935506205 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4603 (secure 0 0 0) 0x7fc5580c7b00 con 0x7fc5580cc170 2023-10-06T05:07:59.890 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:07:59.890 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4966,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6834/2320896583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2320896583},{"type":"v1","addr":"172.21.15.174:6834","nonce":2320896583}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":4969,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/1423644921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6833","nonce":1423644921},{"type":"v1","addr":"172.21.15.174:6835","nonce":1423644921}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5100,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/4291295874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":4291295874},{"type":"v1","addr":"172.21.15.98:6836","nonce":4291295874}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-06T05:07:55.873515+0000","modified":"2023-10-06T05:07:59.511013+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5103},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5103":{"gid":5103,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.98:6837/1617154626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1617154626},{"type":"v1","addr":"172.21.15.98:6837","nonce":1617154626}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2023-10-06T05:07:59.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.889+0000 7fc55ce45700 1 -- 172.21.15.98:0/935506205 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fc544066f30 msgr2=0x7fc5440693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:59.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.889+0000 7fc55ce45700 1 --2- 172.21.15.98:0/935506205 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fc544066f30 0x7fc5440693e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fc548009fd0 tx=0x7fc548009740 comp rx=0 tx=0).stop 2023-10-06T05:07:59.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.889+0000 7fc55ce45700 1 -- 172.21.15.98:0/935506205 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5580cc170 msgr2=0x7fc5580c9af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:07:59.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.889+0000 7fc55ce45700 1 --2- 172.21.15.98:0/935506205 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5580cc170 0x7fc5580c9af0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fc558069150 tx=0x7fc54c017000 comp rx=0 tx=0).stop 2023-10-06T05:07:59.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.889+0000 7fc55ce45700 1 -- 172.21.15.98:0/935506205 shutdown_connections 2023-10-06T05:07:59.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.889+0000 7fc55ce45700 1 --2- 172.21.15.98:0/935506205 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc558159440 0x7fc5580ca030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:59.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.889+0000 7fc55ce45700 1 --2- 172.21.15.98:0/935506205 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc5580b35d0 0x7fc5580d0050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:59.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.889+0000 7fc55ce45700 1 --2- 172.21.15.98:0/935506205 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fc544066f30 0x7fc5440693e0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:59.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.889+0000 7fc55ce45700 1 --2- 172.21.15.98:0/935506205 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc5580cc170 0x7fc5580c9af0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:07:59.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.890+0000 7fc55ce45700 1 -- 172.21.15.98:0/935506205 >> 172.21.15.98:0/935506205 conn(0x7fc55806ece0 msgr2=0x7fc558068080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:07:59.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.890+0000 7fc55ce45700 1 -- 172.21.15.98:0/935506205 shutdown_connections 2023-10-06T05:07:59.893 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:07:59.890+0000 7fc55ce45700 1 -- 172.21.15.98:0/935506205 wait complete. 2023-10-06T05:07:59.893 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 17 2023-10-06T05:07:59.903 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T05:08:00.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.016+0000 7fe6c5fc4700 1 Processor -- start 2023-10-06T05:08:00.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.017+0000 7fe6c5fc4700 1 -- start start 2023-10-06T05:08:00.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.017+0000 7fe6c5fc4700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fe6c00adb60 0x7fe6c00ce190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:00.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.017+0000 7fe6c5fc4700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe6c00ce6d0 0x7fe6c015aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:00.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.017+0000 7fe6c5fc4700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6c0157de0 con 0x7fe6c00ca370 2023-10-06T05:08:00.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.017+0000 7fe6c5fc4700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6c00cbb90 con 0x7fe6c00ce6d0 2023-10-06T05:08:00.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.017+0000 7fe6c5fc4700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6c00ceb10 con 0x7fe6c00adb60 2023-10-06T05:08:00.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.017+0000 7fe6bffff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe6c00ce6d0 0x7fe6c015aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:00.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.018+0000 7fe6bffff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe6c00ce6d0 0x7fe6c015aed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54770/0 (socket says 172.21.15.98:54770) 2023-10-06T05:08:00.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.018+0000 7fe6bffff700 1 -- 172.21.15.98:0/2551455499 learned_addr learned my addr 172.21.15.98:0/2551455499 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:08:00.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.018+0000 7fe6beffd700 1 --2- 172.21.15.98:0/2551455499 >> v2:172.21.15.174:3300/0 conn(0x7fe6c00adb60 0x7fe6c00ce190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:00.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.018+0000 7fe6bffff700 1 -- 172.21.15.98:0/2551455499 >> v2:172.21.15.174:3300/0 conn(0x7fe6c00adb60 msgr2=0x7fe6c00ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:00.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.018+0000 7fe6bffff700 1 --2- 172.21.15.98:0/2551455499 >> v2:172.21.15.174:3300/0 conn(0x7fe6c00adb60 0x7fe6c00ce190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:00.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.018+0000 7fe6bffff700 1 -- 172.21.15.98:0/2551455499 >> v1:172.21.15.174:6789/0 conn(0x7fe6c00ca370 legacy=0x7fe6c0158270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:00.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.018+0000 7fe6bffff700 1 -- 172.21.15.98:0/2551455499 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6c00cb5d0 con 0x7fe6c00ce6d0 2023-10-06T05:08:00.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.018+0000 7fe6bffff700 1 --2- 172.21.15.98:0/2551455499 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6c00ce6d0 0x7fe6c015aed0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fe6c006c350 tx=0x7fe6b8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac35c3644a8964bd server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:00.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.018+0000 7fe6be7fc700 1 -- 172.21.15.98:0/2551455499 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b8009e70 con 0x7fe6c00ce6d0 2023-10-06T05:08:00.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.018+0000 7fe6be7fc700 1 -- 172.21.15.98:0/2551455499 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6b800b860 con 0x7fe6c00ce6d0 2023-10-06T05:08:00.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.018+0000 7fe6be7fc700 1 -- 172.21.15.98:0/2551455499 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b800bc00 con 0x7fe6c00ce6d0 2023-10-06T05:08:00.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.018+0000 7fe6c5fc4700 1 -- 172.21.15.98:0/2551455499 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6c00ce6d0 msgr2=0x7fe6c015aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:00.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.018+0000 7fe6c5fc4700 1 --2- 172.21.15.98:0/2551455499 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6c00ce6d0 0x7fe6c015aed0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fe6c006c350 tx=0x7fe6b8009250 comp rx=0 tx=0).stop 2023-10-06T05:08:00.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.019+0000 7fe6c5fc4700 1 -- 172.21.15.98:0/2551455499 shutdown_connections 2023-10-06T05:08:00.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.019+0000 7fe6c5fc4700 1 --2- 172.21.15.98:0/2551455499 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6c00ce6d0 0x7fe6c015aed0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:00.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.019+0000 7fe6c5fc4700 1 --2- 172.21.15.98:0/2551455499 >> v2:172.21.15.174:3300/0 conn(0x7fe6c00adb60 0x7fe6c00ce190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:00.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.019+0000 7fe6c5fc4700 1 -- 172.21.15.98:0/2551455499 >> 172.21.15.98:0/2551455499 conn(0x7fe6c00707d0 msgr2=0x7fe6c0070bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:00.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.019+0000 7fe6c5fc4700 1 -- 172.21.15.98:0/2551455499 shutdown_connections 2023-10-06T05:08:00.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.019+0000 7fe6c5fc4700 1 -- 172.21.15.98:0/2551455499 wait complete. 2023-10-06T05:08:00.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.020+0000 7fe6c5fc4700 1 Processor -- start 2023-10-06T05:08:00.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.020+0000 7fe6c5fc4700 1 -- start start 2023-10-06T05:08:00.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.021+0000 7fe6c5fc4700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe6c00adb60 0x7fe6c0152950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:00.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.021+0000 7fe6c5fc4700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe6c00ca370 0x7fe6c00c2500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:00.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.021+0000 7fe6c5fc4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6c00ce6d0 0x7fe6c00c2a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:00.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.021+0000 7fe6c5fc4700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6c00c3c60 con 0x7fe6c00ce6d0 2023-10-06T05:08:00.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.021+0000 7fe6c5fc4700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6c00c3050 con 0x7fe6c00ca370 2023-10-06T05:08:00.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.021+0000 7fe6c5fc4700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6c00c3330 con 0x7fe6c00adb60 2023-10-06T05:08:00.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.021+0000 7fe6bffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6c00ce6d0 0x7fe6c00c2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:00.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.021+0000 7fe6bffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6c00ce6d0 0x7fe6c00c2a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54772/0 (socket says 172.21.15.98:54772) 2023-10-06T05:08:00.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.021+0000 7fe6bffff700 1 -- 172.21.15.98:0/212487119 learned_addr learned my addr 172.21.15.98:0/212487119 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:08:00.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.021+0000 7fe6bf7fe700 1 --2- 172.21.15.98:0/212487119 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe6c00adb60 0x7fe6c0152950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:00.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.021+0000 7fe6beffd700 1 --2- 172.21.15.98:0/212487119 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe6c00ca370 0x7fe6c00c2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:00.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.021+0000 7fe6bffff700 1 -- 172.21.15.98:0/212487119 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe6c00adb60 msgr2=0x7fe6c0152950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:00.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.021+0000 7fe6bffff700 1 --2- 172.21.15.98:0/212487119 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe6c00adb60 0x7fe6c0152950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:00.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.021+0000 7fe6bffff700 1 -- 172.21.15.98:0/212487119 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe6c00ca370 msgr2=0x7fe6c00c2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:00.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.021+0000 7fe6bffff700 1 --2- 172.21.15.98:0/212487119 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe6c00ca370 0x7fe6c00c2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:00.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.021+0000 7fe6bffff700 1 -- 172.21.15.98:0/212487119 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6b80094f0 con 0x7fe6c00ce6d0 2023-10-06T05:08:00.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.021+0000 7fe6beffd700 1 --2- 172.21.15.98:0/212487119 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe6c00ca370 0x7fe6c00c2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:08:00.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.021+0000 7fe6bf7fe700 1 --2- 172.21.15.98:0/212487119 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe6c00adb60 0x7fe6c0152950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:08:00.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.021+0000 7fe6bffff700 1 --2- 172.21.15.98:0/212487119 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6c00ce6d0 0x7fe6c00c2a40 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fe6b8009290 tx=0x7fe6b8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:00.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.021+0000 7fe6a77fe700 1 -- 172.21.15.98:0/212487119 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b8025420 con 0x7fe6c00ce6d0 2023-10-06T05:08:00.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.021+0000 7fe6a77fe700 1 -- 172.21.15.98:0/212487119 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6b80255a0 con 0x7fe6c00ce6d0 2023-10-06T05:08:00.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.021+0000 7fe6c5fc4700 1 -- 172.21.15.98:0/212487119 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6c00c3610 con 0x7fe6c00ce6d0 2023-10-06T05:08:00.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.021+0000 7fe6a77fe700 1 -- 172.21.15.98:0/212487119 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b8025960 con 0x7fe6c00ce6d0 2023-10-06T05:08:00.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.021+0000 7fe6c5fc4700 1 -- 172.21.15.98:0/212487119 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6c00c5560 con 0x7fe6c00ce6d0 2023-10-06T05:08:00.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.022+0000 7fe6c5fc4700 1 -- 172.21.15.98:0/212487119 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6c00c6ec0 con 0x7fe6c00ce6d0 2023-10-06T05:08:00.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.023+0000 7fe6a77fe700 1 -- 172.21.15.98:0/212487119 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fe6b8025d50 con 0x7fe6c00ce6d0 2023-10-06T05:08:00.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.023+0000 7fe6a77fe700 1 --2- 172.21.15.98:0/212487119 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fe6ac066f30 0x7fe6ac0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:00.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.023+0000 7fe6a77fe700 1 -- 172.21.15.98:0/212487119 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6378+0+0 (secure 0 0 0) 0x7fe6b80862f0 con 0x7fe6c00ce6d0 2023-10-06T05:08:00.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.023+0000 7fe6bf7fe700 1 --2- 172.21.15.98:0/212487119 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fe6ac066f30 0x7fe6ac0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:00.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.023+0000 7fe6bf7fe700 1 --2- 172.21.15.98:0/212487119 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fe6ac066f30 0x7fe6ac0693e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe6b0007fd0 tx=0x7fe6b0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:00.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.027+0000 7fe6a77fe700 1 -- 172.21.15.98:0/212487119 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe6b80532c0 con 0x7fe6c00ce6d0 2023-10-06T05:08:00.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.212+0000 7fe6c5fc4700 1 -- 172.21.15.98:0/212487119 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe6c0064a90 con 0x7fe6c00ce6d0 2023-10-06T05:08:00.215 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.213+0000 7fe6a77fe700 1 -- 172.21.15.98:0/212487119 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v43) v1 ==== 74+0+17071 (secure 0 0 0) 0x7fe6b80568e0 con 0x7fe6c00ce6d0 2023-10-06T05:08:00.216 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:08:00.216 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":43,"fsid":"f50c5df5-cad5-47a4-9bb1-8c791d4311ec","created":"2023-10-06T05:06:32.971175+0000","modified":"2023-10-06T05:07:57.482671+0000","last_up_change":"2023-10-06T05:06:44.016489+0000","last_in_change":"2023-10-06T05:06:35.544511+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-06T05:06:44.559671+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-06T05:06:51.432161+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-10-06T05:07:54.125452+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-10-06T05:07:54.831073+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d9495d95-da75-4c3b-8927-e9bae6a8b093","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":90627},{"type":"v1","addr":"172.21.15.98:6801","nonce":90627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":90627},{"type":"v1","addr":"172.21.15.98:6803","nonce":90627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":90627},{"type":"v1","addr":"172.21.15.98:6807","nonce":90627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":90627},{"type":"v1","addr":"172.21.15.98:6805","nonce":90627}]},"public_addr":"172.21.15.98:6801/90627","cluster_addr":"172.21.15.98:6803/90627","heartbeat_back_addr":"172.21.15.98:6807/90627","heartbeat_front_addr":"172.21.15.98:6805/90627","state":["exists","up"]},{"osd":1,"uuid":"f25ba5be-d65e-4707-a134-154092deb754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":90628},{"type":"v1","addr":"172.21.15.98:6809","nonce":90628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":90628},{"type":"v1","addr":"172.21.15.98:6811","nonce":90628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":90628},{"type":"v1","addr":"172.21.15.98:6815","nonce":90628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":90628},{"type":"v1","addr":"172.21.15.98:6813","nonce":90628}]},"public_addr":"172.21.15.98:6809/90628","cluster_addr":"172.21.15.98:6811/90628","heartbeat_back_addr":"172.21.15.98:6815/90628","heartbeat_front_addr":"172.21.15.98:6813/90628","state":["exists","up"]},{"osd":2,"uuid":"5ab9e028-945c-43e2-8c95-3b0b4cf38e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":90662},{"type":"v1","addr":"172.21.15.98:6818","nonce":90662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":90662},{"type":"v1","addr":"172.21.15.98:6822","nonce":90662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":90662},{"type":"v1","addr":"172.21.15.98:6830","nonce":90662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":90662},{"type":"v1","addr":"172.21.15.98:6826","nonce":90662}]},"public_addr":"172.21.15.98:6818/90662","cluster_addr":"172.21.15.98:6822/90662","heartbeat_back_addr":"172.21.15.98:6830/90662","heartbeat_front_addr":"172.21.15.98:6826/90662","state":["exists","up"]},{"osd":3,"uuid":"3bb803ff-1738-46bf-be62-2267bf378822","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6817","nonce":90663},{"type":"v1","addr":"172.21.15.98:6819","nonce":90663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6821","nonce":90663},{"type":"v1","addr":"172.21.15.98:6823","nonce":90663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6829","nonce":90663},{"type":"v1","addr":"172.21.15.98:6831","nonce":90663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6825","nonce":90663},{"type":"v1","addr":"172.21.15.98:6827","nonce":90663}]},"public_addr":"172.21.15.98:6819/90663","cluster_addr":"172.21.15.98:6823/90663","heartbeat_back_addr":"172.21.15.98:6831/90663","heartbeat_front_addr":"172.21.15.98:6827/90663","state":["exists","up"]},{"osd":4,"uuid":"7d57a4c9-6462-43f6-b64a-7f9e9b4353d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":90847},{"type":"v1","addr":"172.21.15.174:6801","nonce":90847}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":90847},{"type":"v1","addr":"172.21.15.174:6803","nonce":90847}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":90847},{"type":"v1","addr":"172.21.15.174:6807","nonce":90847}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":90847},{"type":"v1","addr":"172.21.15.174:6805","nonce":90847}]},"public_addr":"172.21.15.174:6801/90847","cluster_addr":"172.21.15.174:6803/90847","heartbeat_back_addr":"172.21.15.174:6807/90847","heartbeat_front_addr":"172.21.15.174:6805/90847","state":["exists","up"]},{"osd":5,"uuid":"83287ea4-e44d-4af9-bc77-36d47c60d4a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":90848},{"type":"v1","addr":"172.21.15.174:6809","nonce":90848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":90848},{"type":"v1","addr":"172.21.15.174:6811","nonce":90848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":90848},{"type":"v1","addr":"172.21.15.174:6815","nonce":90848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":90848},{"type":"v1","addr":"172.21.15.174:6813","nonce":90848}]},"public_addr":"172.21.15.174:6809/90848","cluster_addr":"172.21.15.174:6811/90848","heartbeat_back_addr":"172.21.15.174:6815/90848","heartbeat_front_addr":"172.21.15.174:6813/90848","state":["exists","up"]},{"osd":6,"uuid":"496a6272-892e-43c4-a145-206623c22f99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":90877},{"type":"v1","addr":"172.21.15.174:6817","nonce":90877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":90877},{"type":"v1","addr":"172.21.15.174:6819","nonce":90877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":90877},{"type":"v1","addr":"172.21.15.174:6823","nonce":90877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":90877},{"type":"v1","addr":"172.21.15.174:6821","nonce":90877}]},"public_addr":"172.21.15.174:6817/90877","cluster_addr":"172.21.15.174:6819/90877","heartbeat_back_addr":"172.21.15.174:6823/90877","heartbeat_front_addr":"172.21.15.174:6821/90877","state":["exists","up"]},{"osd":7,"uuid":"49967e34-8b07-4bab-a13e-0c16d6a0f3a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":90878},{"type":"v1","addr":"172.21.15.174:6825","nonce":90878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":90878},{"type":"v1","addr":"172.21.15.174:6827","nonce":90878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":90878},{"type":"v1","addr":"172.21.15.174:6831","nonce":90878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":90878},{"type":"v1","addr":"172.21.15.174:6829","nonce":90878}]},"public_addr":"172.21.15.174:6825/90878","cluster_addr":"172.21.15.174:6827/90878","heartbeat_back_addr":"172.21.15.174:6831/90878","heartbeat_front_addr":"172.21.15.174:6829/90878","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.446479+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.514266+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.794108+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.810894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.731919+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.717756+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.737435+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.508682+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T05:08:00.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.215+0000 7fe6c5fc4700 1 -- 172.21.15.98:0/212487119 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fe6ac066f30 msgr2=0x7fe6ac0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:00.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.215+0000 7fe6c5fc4700 1 --2- 172.21.15.98:0/212487119 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fe6ac066f30 0x7fe6ac0693e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe6b0007fd0 tx=0x7fe6b0009040 comp rx=0 tx=0).stop 2023-10-06T05:08:00.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.215+0000 7fe6c5fc4700 1 -- 172.21.15.98:0/212487119 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6c00ce6d0 msgr2=0x7fe6c00c2a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:00.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.215+0000 7fe6c5fc4700 1 --2- 172.21.15.98:0/212487119 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6c00ce6d0 0x7fe6c00c2a40 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fe6b8009290 tx=0x7fe6b8017000 comp rx=0 tx=0).stop 2023-10-06T05:08:00.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.215+0000 7fe6c5fc4700 1 -- 172.21.15.98:0/212487119 shutdown_connections 2023-10-06T05:08:00.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.215+0000 7fe6c5fc4700 1 --2- 172.21.15.98:0/212487119 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fe6ac066f30 0x7fe6ac0693e0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:00.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.215+0000 7fe6c5fc4700 1 --2- 172.21.15.98:0/212487119 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6c00ce6d0 0x7fe6c00c2a40 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:00.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.215+0000 7fe6c5fc4700 1 --2- 172.21.15.98:0/212487119 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe6c00adb60 0x7fe6c0152950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:00.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.215+0000 7fe6c5fc4700 1 --2- 172.21.15.98:0/212487119 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe6c00ca370 0x7fe6c00c2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:00.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.215+0000 7fe6c5fc4700 1 -- 172.21.15.98:0/212487119 >> 172.21.15.98:0/212487119 conn(0x7fe6c00707d0 msgr2=0x7fe6c00ca740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:00.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.215+0000 7fe6c5fc4700 1 -- 172.21.15.98:0/212487119 shutdown_connections 2023-10-06T05:08:00.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.215+0000 7fe6c5fc4700 1 -- 172.21.15.98:0/212487119 wait complete. 2023-10-06T05:08:00.229 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-06T05:08:00.230 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-06T05:08:00.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.342+0000 7ff4135db700 1 Processor -- start 2023-10-06T05:08:00.345 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.342+0000 7ff4135db700 1 -- start start 2023-10-06T05:08:00.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.343+0000 7ff4135db700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff40c159560 0x7ff40c0aa530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:00.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.343+0000 7ff4135db700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff40c0b49a0 0x7ff40c0aaa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:00.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.343+0000 7ff4135db700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff40c0b0c60 con 0x7ff40c0b0760 2023-10-06T05:08:00.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.343+0000 7ff4135db700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff40c0b0f40 con 0x7ff40c159560 2023-10-06T05:08:00.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.343+0000 7ff4135db700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff40c0b1220 con 0x7ff40c0b49a0 2023-10-06T05:08:00.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.343+0000 7ff411377700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff40c159560 0x7ff40c0aa530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:00.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.343+0000 7ff411377700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff40c159560 0x7ff40c0aa530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54774/0 (socket says 172.21.15.98:54774) 2023-10-06T05:08:00.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.343+0000 7ff411377700 1 -- 172.21.15.98:0/3909774860 learned_addr learned my addr 172.21.15.98:0/3909774860 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:08:00.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.343+0000 7ff410b76700 1 --2- 172.21.15.98:0/3909774860 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff40c0b49a0 0x7ff40c0aaa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:00.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.344+0000 7ff411377700 1 -- 172.21.15.98:0/3909774860 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff40c0b49a0 msgr2=0x7ff40c0aaa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:00.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.344+0000 7ff411377700 1 --2- 172.21.15.98:0/3909774860 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff40c0b49a0 0x7ff40c0aaa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:00.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.344+0000 7ff411377700 1 -- 172.21.15.98:0/3909774860 >> v1:172.21.15.174:6789/0 conn(0x7ff40c0b0760 legacy=0x7ff40c0b3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:00.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.344+0000 7ff411377700 1 -- 172.21.15.98:0/3909774860 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff40c0b1500 con 0x7ff40c159560 2023-10-06T05:08:00.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.344+0000 7ff411377700 1 --2- 172.21.15.98:0/3909774860 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff40c159560 0x7ff40c0aa530 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7ff408009990 tx=0x7ff408009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=233f4831123c989e server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:00.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.344+0000 7ff403fff700 1 -- 172.21.15.98:0/3909774860 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff408009e80 con 0x7ff40c159560 2023-10-06T05:08:00.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.344+0000 7ff403fff700 1 -- 172.21.15.98:0/3909774860 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff408005c10 con 0x7ff40c159560 2023-10-06T05:08:00.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.344+0000 7ff403fff700 1 -- 172.21.15.98:0/3909774860 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff40800b650 con 0x7ff40c159560 2023-10-06T05:08:00.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.344+0000 7ff4135db700 1 -- 172.21.15.98:0/3909774860 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff40c159560 msgr2=0x7ff40c0aa530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:00.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.344+0000 7ff4135db700 1 --2- 172.21.15.98:0/3909774860 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff40c159560 0x7ff40c0aa530 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7ff408009990 tx=0x7ff408009220 comp rx=0 tx=0).stop 2023-10-06T05:08:00.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.344+0000 7ff4135db700 1 -- 172.21.15.98:0/3909774860 shutdown_connections 2023-10-06T05:08:00.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.344+0000 7ff4135db700 1 --2- 172.21.15.98:0/3909774860 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff40c159560 0x7ff40c0aa530 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:00.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.344+0000 7ff4135db700 1 --2- 172.21.15.98:0/3909774860 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff40c0b49a0 0x7ff40c0aaa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:00.349 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.344+0000 7ff4135db700 1 -- 172.21.15.98:0/3909774860 >> 172.21.15.98:0/3909774860 conn(0x7ff40c0997f0 msgr2=0x7ff40c06f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:00.349 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.345+0000 7ff4135db700 1 -- 172.21.15.98:0/3909774860 shutdown_connections 2023-10-06T05:08:00.349 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.345+0000 7ff4135db700 1 -- 172.21.15.98:0/3909774860 wait complete. 2023-10-06T05:08:00.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.346+0000 7ff4135db700 1 Processor -- start 2023-10-06T05:08:00.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.346+0000 7ff4135db700 1 -- start start 2023-10-06T05:08:00.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.346+0000 7ff4135db700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff40c0b0760 0x7ff40c1540c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:00.354 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.346+0000 7ff4135db700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff40c0b49a0 0x7ff40c15c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:00.354 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.346+0000 7ff4135db700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff40c159560 0x7ff40c154e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:00.354 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.346+0000 7ff4135db700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff40c14e1d0 con 0x7ff40c159560 2023-10-06T05:08:00.354 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.346+0000 7ff4135db700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff40c1530d0 con 0x7ff40c0b49a0 2023-10-06T05:08:00.354 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.346+0000 7ff4135db700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff40c15ca10 con 0x7ff40c0b0760 2023-10-06T05:08:00.354 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.347+0000 7ff411b78700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff40c159560 0x7ff40c154e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:00.354 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.347+0000 7ff411b78700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff40c159560 0x7ff40c154e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54780/0 (socket says 172.21.15.98:54780) 2023-10-06T05:08:00.354 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.347+0000 7ff411b78700 1 -- 172.21.15.98:0/2293677859 learned_addr learned my addr 172.21.15.98:0/2293677859 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:08:00.354 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.347+0000 7ff411377700 1 --2- 172.21.15.98:0/2293677859 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff40c0b0760 0x7ff40c1540c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:00.354 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.347+0000 7ff410b76700 1 --2- 172.21.15.98:0/2293677859 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff40c0b49a0 0x7ff40c15c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:00.354 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.347+0000 7ff411b78700 1 -- 172.21.15.98:0/2293677859 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff40c0b0760 msgr2=0x7ff40c1540c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:00.354 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.347+0000 7ff411b78700 1 --2- 172.21.15.98:0/2293677859 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff40c0b0760 0x7ff40c1540c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:00.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.347+0000 7ff411b78700 1 -- 172.21.15.98:0/2293677859 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff40c0b49a0 msgr2=0x7ff40c15c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:00.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.347+0000 7ff411b78700 1 --2- 172.21.15.98:0/2293677859 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff40c0b49a0 0x7ff40c15c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:00.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.347+0000 7ff411b78700 1 -- 172.21.15.98:0/2293677859 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff408009480 con 0x7ff40c159560 2023-10-06T05:08:00.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.347+0000 7ff411b78700 1 --2- 172.21.15.98:0/2293677859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff40c159560 0x7ff40c154e80 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7ff40400c910 tx=0x7ff404001710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:00.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.347+0000 7ff4017fa700 1 -- 172.21.15.98:0/2293677859 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff404012630 con 0x7ff40c159560 2023-10-06T05:08:00.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.347+0000 7ff4017fa700 1 -- 172.21.15.98:0/2293677859 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4040127b0 con 0x7ff40c159560 2023-10-06T05:08:00.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.347+0000 7ff4135db700 1 -- 172.21.15.98:0/2293677859 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff40c153c30 con 0x7ff40c159560 2023-10-06T05:08:00.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.347+0000 7ff4017fa700 1 -- 172.21.15.98:0/2293677859 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff404012b70 con 0x7ff40c159560 2023-10-06T05:08:00.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.347+0000 7ff4135db700 1 -- 172.21.15.98:0/2293677859 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff40c1535e0 con 0x7ff40c159560 2023-10-06T05:08:00.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.348+0000 7ff4135db700 1 -- 172.21.15.98:0/2293677859 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff40c153dc0 con 0x7ff40c159560 2023-10-06T05:08:00.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.349+0000 7ff4017fa700 1 -- 172.21.15.98:0/2293677859 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7ff404003030 con 0x7ff40c159560 2023-10-06T05:08:00.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.349+0000 7ff4017fa700 1 --2- 172.21.15.98:0/2293677859 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7ff3f0066f30 0x7ff3f00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:00.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.349+0000 7ff4017fa700 1 -- 172.21.15.98:0/2293677859 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6378+0+0 (secure 0 0 0) 0x7ff4040846a0 con 0x7ff40c159560 2023-10-06T05:08:00.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.349+0000 7ff411377700 1 --2- 172.21.15.98:0/2293677859 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7ff3f0066f30 0x7ff3f00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:00.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.350+0000 7ff411377700 1 --2- 172.21.15.98:0/2293677859 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7ff3f0066f30 0x7ff3f00693e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ff408009260 tx=0x7ff408016000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:00.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.354+0000 7ff4017fa700 1 -- 172.21.15.98:0/2293677859 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff404051670 con 0x7ff40c159560 2023-10-06T05:08:00.537 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.535+0000 7ff4135db700 1 -- 172.21.15.98:0/2293677859 --> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff40c064ac0 con 0x7ff3f0066f30 2023-10-06T05:08:00.545 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.543+0000 7ff4017fa700 1 -- 172.21.15.98:0/2293677859 <== mgr.4104 v2:172.21.15.98:6832/90397 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+311205 (secure 0 0 0) 0x7ff40c064ac0 con 0x7ff3f0066f30 2023-10-06T05:08:00.550 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:08:00.553 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.546+0000 7ff4135db700 1 -- 172.21.15.98:0/2293677859 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7ff3f0066f30 msgr2=0x7ff3f00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:00.553 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.546+0000 7ff4135db700 1 --2- 172.21.15.98:0/2293677859 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7ff3f0066f30 0x7ff3f00693e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ff408009260 tx=0x7ff408016000 comp rx=0 tx=0).stop 2023-10-06T05:08:00.554 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.546+0000 7ff4135db700 1 -- 172.21.15.98:0/2293677859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff40c159560 msgr2=0x7ff40c154e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:00.554 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.546+0000 7ff4135db700 1 --2- 172.21.15.98:0/2293677859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff40c159560 0x7ff40c154e80 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7ff40400c910 tx=0x7ff404001710 comp rx=0 tx=0).stop 2023-10-06T05:08:00.554 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.546+0000 7ff4135db700 1 -- 172.21.15.98:0/2293677859 shutdown_connections 2023-10-06T05:08:00.554 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.546+0000 7ff4135db700 1 --2- 172.21.15.98:0/2293677859 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7ff3f0066f30 0x7ff3f00693e0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:00.554 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.546+0000 7ff4135db700 1 --2- 172.21.15.98:0/2293677859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff40c159560 0x7ff40c154e80 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:00.554 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.546+0000 7ff4135db700 1 --2- 172.21.15.98:0/2293677859 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff40c0b0760 0x7ff40c1540c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:00.554 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.546+0000 7ff4135db700 1 --2- 172.21.15.98:0/2293677859 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff40c0b49a0 0x7ff40c15c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:00.555 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.546+0000 7ff4135db700 1 -- 172.21.15.98:0/2293677859 >> 172.21.15.98:0/2293677859 conn(0x7ff40c0997f0 msgr2=0x7ff40c158970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:00.555 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.546+0000 7ff4135db700 1 -- 172.21.15.98:0/2293677859 shutdown_connections 2023-10-06T05:08:00.555 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.546+0000 7ff4135db700 1 -- 172.21.15.98:0/2293677859 wait complete. 2023-10-06T05:08:00.555 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2023-10-06T05:08:00.560 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":75,"stamp":"2023-10-06T05:08:00.532870+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":168,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2383332,"kb_used_data":5028,"kb_used_omap":0,"kb_used_meta":2378240,"kb_avail":752591388,"statfs":{"total":773094113280,"available":770653581312,"internally_reserved":0,"allocated":5148672,"data_stored":2913553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2435317760},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000953"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":8,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.497305+0000","last_change":"2023-10-06T05:07:56.497305+0000","last_active":"2023-10-06T05:07:56.497305+0000","last_peered":"2023-10-06T05:07:56.497305+0000","last_clean":"2023-10-06T05:07:56.497305+0000","last_became_active":"2023-10-06T05:07:56.496979+0000","last_became_peered":"2023-10-06T05:07:56.496979+0000","last_unstale":"2023-10-06T05:07:56.497305+0000","last_undegraded":"2023-10-06T05:07:56.497305+0000","last_fullsized":"2023-10-06T05:07:56.497305+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:05:12.786696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.505779+0000","last_change":"2023-10-06T05:07:56.496318+0000","last_active":"2023-10-06T05:07:56.505779+0000","last_peered":"2023-10-06T05:07:56.505779+0000","last_clean":"2023-10-06T05:07:56.505779+0000","last_became_active":"2023-10-06T05:07:56.495491+0000","last_became_peered":"2023-10-06T05:07:56.495491+0000","last_unstale":"2023-10-06T05:07:56.505779+0000","last_undegraded":"2023-10-06T05:07:56.505779+0000","last_fullsized":"2023-10-06T05:07:56.505779+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:12:32.521554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.505213+0000","last_change":"2023-10-06T05:07:56.501549+0000","last_active":"2023-10-06T05:07:56.505213+0000","last_peered":"2023-10-06T05:07:56.505213+0000","last_clean":"2023-10-06T05:07:56.505213+0000","last_became_active":"2023-10-06T05:07:56.501277+0000","last_became_peered":"2023-10-06T05:07:56.501277+0000","last_unstale":"2023-10-06T05:07:56.505213+0000","last_undegraded":"2023-10-06T05:07:56.505213+0000","last_fullsized":"2023-10-06T05:07:56.505213+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:39:55.924395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":8,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.506566+0000","last_change":"2023-10-06T05:07:56.506566+0000","last_active":"2023-10-06T05:07:56.506566+0000","last_peered":"2023-10-06T05:07:56.506566+0000","last_clean":"2023-10-06T05:07:56.506566+0000","last_became_active":"2023-10-06T05:07:56.505105+0000","last_became_peered":"2023-10-06T05:07:56.505105+0000","last_unstale":"2023-10-06T05:07:56.506566+0000","last_undegraded":"2023-10-06T05:07:56.506566+0000","last_fullsized":"2023-10-06T05:07:56.506566+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:51:57.567130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.505737+0000","last_change":"2023-10-06T05:07:56.496507+0000","last_active":"2023-10-06T05:07:56.505737+0000","last_peered":"2023-10-06T05:07:56.505737+0000","last_clean":"2023-10-06T05:07:56.505737+0000","last_became_active":"2023-10-06T05:07:56.496216+0000","last_became_peered":"2023-10-06T05:07:56.496216+0000","last_unstale":"2023-10-06T05:07:56.505737+0000","last_undegraded":"2023-10-06T05:07:56.505737+0000","last_fullsized":"2023-10-06T05:07:56.505737+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:50:46.462343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.499993+0000","last_change":"2023-10-06T05:07:56.498796+0000","last_active":"2023-10-06T05:07:56.499993+0000","last_peered":"2023-10-06T05:07:56.499993+0000","last_clean":"2023-10-06T05:07:56.499993+0000","last_became_active":"2023-10-06T05:07:56.498537+0000","last_became_peered":"2023-10-06T05:07:56.498537+0000","last_unstale":"2023-10-06T05:07:56.499993+0000","last_undegraded":"2023-10-06T05:07:56.499993+0000","last_fullsized":"2023-10-06T05:07:56.499993+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:01:24.910048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":8,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.507167+0000","last_change":"2023-10-06T05:07:56.507167+0000","last_active":"2023-10-06T05:07:56.507167+0000","last_peered":"2023-10-06T05:07:56.507167+0000","last_clean":"2023-10-06T05:07:56.507167+0000","last_became_active":"2023-10-06T05:07:56.506895+0000","last_became_peered":"2023-10-06T05:07:56.506895+0000","last_unstale":"2023-10-06T05:07:56.507167+0000","last_undegraded":"2023-10-06T05:07:56.507167+0000","last_fullsized":"2023-10-06T05:07:56.507167+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T16:18:50.165832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":8,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.504667+0000","last_change":"2023-10-06T05:07:56.504667+0000","last_active":"2023-10-06T05:07:56.504667+0000","last_peered":"2023-10-06T05:07:56.504667+0000","last_clean":"2023-10-06T05:07:56.504667+0000","last_became_active":"2023-10-06T05:07:56.504341+0000","last_became_peered":"2023-10-06T05:07:56.504341+0000","last_unstale":"2023-10-06T05:07:56.504667+0000","last_undegraded":"2023-10-06T05:07:56.504667+0000","last_fullsized":"2023-10-06T05:07:56.504667+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:05:44.930056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":8,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.507152+0000","last_change":"2023-10-06T05:07:56.507152+0000","last_active":"2023-10-06T05:07:56.507152+0000","last_peered":"2023-10-06T05:07:56.507152+0000","last_clean":"2023-10-06T05:07:56.507152+0000","last_became_active":"2023-10-06T05:07:56.506920+0000","last_became_peered":"2023-10-06T05:07:56.506920+0000","last_unstale":"2023-10-06T05:07:56.507152+0000","last_undegraded":"2023-10-06T05:07:56.507152+0000","last_fullsized":"2023-10-06T05:07:56.507152+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:23:42.009212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":8,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.504224+0000","last_change":"2023-10-06T05:07:56.504224+0000","last_active":"2023-10-06T05:07:56.504224+0000","last_peered":"2023-10-06T05:07:56.504224+0000","last_clean":"2023-10-06T05:07:56.504224+0000","last_became_active":"2023-10-06T05:07:56.503937+0000","last_became_peered":"2023-10-06T05:07:56.503937+0000","last_unstale":"2023-10-06T05:07:56.504224+0000","last_undegraded":"2023-10-06T05:07:56.504224+0000","last_fullsized":"2023-10-06T05:07:56.504224+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:25:49.962193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":8,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.510436+0000","last_change":"2023-10-06T05:07:56.510436+0000","last_active":"2023-10-06T05:07:56.510436+0000","last_peered":"2023-10-06T05:07:56.510436+0000","last_clean":"2023-10-06T05:07:56.510436+0000","last_became_active":"2023-10-06T05:07:56.510191+0000","last_became_peered":"2023-10-06T05:07:56.510191+0000","last_unstale":"2023-10-06T05:07:56.510436+0000","last_undegraded":"2023-10-06T05:07:56.510436+0000","last_fullsized":"2023-10-06T05:07:56.510436+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:46:33.207358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.512781+0000","last_change":"2023-10-06T05:07:56.511385+0000","last_active":"2023-10-06T05:07:56.512781+0000","last_peered":"2023-10-06T05:07:56.512781+0000","last_clean":"2023-10-06T05:07:56.512781+0000","last_became_active":"2023-10-06T05:07:56.511074+0000","last_became_peered":"2023-10-06T05:07:56.511074+0000","last_unstale":"2023-10-06T05:07:56.512781+0000","last_undegraded":"2023-10-06T05:07:56.512781+0000","last_fullsized":"2023-10-06T05:07:56.512781+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:02:41.335573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.517892+0000","last_change":"2023-10-06T05:07:56.516120+0000","last_active":"2023-10-06T05:07:56.517892+0000","last_peered":"2023-10-06T05:07:56.517892+0000","last_clean":"2023-10-06T05:07:56.517892+0000","last_became_active":"2023-10-06T05:07:56.515873+0000","last_became_peered":"2023-10-06T05:07:56.515873+0000","last_unstale":"2023-10-06T05:07:56.517892+0000","last_undegraded":"2023-10-06T05:07:56.517892+0000","last_fullsized":"2023-10-06T05:07:56.517892+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:02:33.928047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":8,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.508929+0000","last_change":"2023-10-06T05:07:56.508929+0000","last_active":"2023-10-06T05:07:56.508929+0000","last_peered":"2023-10-06T05:07:56.508929+0000","last_clean":"2023-10-06T05:07:56.508929+0000","last_became_active":"2023-10-06T05:07:56.508696+0000","last_became_peered":"2023-10-06T05:07:56.508696+0000","last_unstale":"2023-10-06T05:07:56.508929+0000","last_undegraded":"2023-10-06T05:07:56.508929+0000","last_fullsized":"2023-10-06T05:07:56.508929+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:30:08.339655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.498009+0000","last_change":"2023-10-06T05:07:56.497209+0000","last_active":"2023-10-06T05:07:56.498009+0000","last_peered":"2023-10-06T05:07:56.498009+0000","last_clean":"2023-10-06T05:07:56.498009+0000","last_became_active":"2023-10-06T05:07:56.496774+0000","last_became_peered":"2023-10-06T05:07:56.496774+0000","last_unstale":"2023-10-06T05:07:56.498009+0000","last_undegraded":"2023-10-06T05:07:56.498009+0000","last_fullsized":"2023-10-06T05:07:56.498009+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:11:44.842024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.505863+0000","last_change":"2023-10-06T05:07:56.500829+0000","last_active":"2023-10-06T05:07:56.505863+0000","last_peered":"2023-10-06T05:07:56.505863+0000","last_clean":"2023-10-06T05:07:56.505863+0000","last_became_active":"2023-10-06T05:07:56.499161+0000","last_became_peered":"2023-10-06T05:07:56.499161+0000","last_unstale":"2023-10-06T05:07:56.505863+0000","last_undegraded":"2023-10-06T05:07:56.505863+0000","last_fullsized":"2023-10-06T05:07:56.505863+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:02:33.928047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.502732+0000","last_change":"2023-10-06T05:07:56.493854+0000","last_active":"2023-10-06T05:07:56.502732+0000","last_peered":"2023-10-06T05:07:56.502732+0000","last_clean":"2023-10-06T05:07:56.502732+0000","last_became_active":"2023-10-06T05:07:56.493573+0000","last_became_peered":"2023-10-06T05:07:56.493573+0000","last_unstale":"2023-10-06T05:07:56.502732+0000","last_undegraded":"2023-10-06T05:07:56.502732+0000","last_fullsized":"2023-10-06T05:07:56.502732+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:51:44.109076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.504641+0000","last_change":"2023-10-06T05:07:56.500173+0000","last_active":"2023-10-06T05:07:56.504641+0000","last_peered":"2023-10-06T05:07:56.504641+0000","last_clean":"2023-10-06T05:07:56.504641+0000","last_became_active":"2023-10-06T05:07:56.499919+0000","last_became_peered":"2023-10-06T05:07:56.499919+0000","last_unstale":"2023-10-06T05:07:56.504641+0000","last_undegraded":"2023-10-06T05:07:56.504641+0000","last_fullsized":"2023-10-06T05:07:56.504641+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:03:00.432048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.505792+0000","last_change":"2023-10-06T05:07:56.500721+0000","last_active":"2023-10-06T05:07:56.505792+0000","last_peered":"2023-10-06T05:07:56.505792+0000","last_clean":"2023-10-06T05:07:56.505792+0000","last_became_active":"2023-10-06T05:07:56.500502+0000","last_became_peered":"2023-10-06T05:07:56.500502+0000","last_unstale":"2023-10-06T05:07:56.505792+0000","last_undegraded":"2023-10-06T05:07:56.505792+0000","last_fullsized":"2023-10-06T05:07:56.505792+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:32:31.940742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.512740+0000","last_change":"2023-10-06T05:07:56.507184+0000","last_active":"2023-10-06T05:07:56.512740+0000","last_peered":"2023-10-06T05:07:56.512740+0000","last_clean":"2023-10-06T05:07:56.512740+0000","last_became_active":"2023-10-06T05:07:56.502222+0000","last_became_peered":"2023-10-06T05:07:56.502222+0000","last_unstale":"2023-10-06T05:07:56.512740+0000","last_undegraded":"2023-10-06T05:07:56.512740+0000","last_fullsized":"2023-10-06T05:07:56.512740+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:32:31.940742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.515025+0000","last_change":"2023-10-06T05:07:56.506115+0000","last_active":"2023-10-06T05:07:56.515025+0000","last_peered":"2023-10-06T05:07:56.515025+0000","last_clean":"2023-10-06T05:07:56.515025+0000","last_became_active":"2023-10-06T05:07:56.505853+0000","last_became_peered":"2023-10-06T05:07:56.505853+0000","last_unstale":"2023-10-06T05:07:56.515025+0000","last_undegraded":"2023-10-06T05:07:56.515025+0000","last_fullsized":"2023-10-06T05:07:56.515025+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:08:06.961933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":8,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.504863+0000","last_change":"2023-10-06T05:07:56.504863+0000","last_active":"2023-10-06T05:07:56.504863+0000","last_peered":"2023-10-06T05:07:56.504863+0000","last_clean":"2023-10-06T05:07:56.504863+0000","last_became_active":"2023-10-06T05:07:56.504612+0000","last_became_peered":"2023-10-06T05:07:56.504612+0000","last_unstale":"2023-10-06T05:07:56.504863+0000","last_undegraded":"2023-10-06T05:07:56.504863+0000","last_fullsized":"2023-10-06T05:07:56.504863+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:30:08.409840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.505843+0000","last_change":"2023-10-06T05:07:56.501498+0000","last_active":"2023-10-06T05:07:56.505843+0000","last_peered":"2023-10-06T05:07:56.505843+0000","last_clean":"2023-10-06T05:07:56.505843+0000","last_became_active":"2023-10-06T05:07:56.501215+0000","last_became_peered":"2023-10-06T05:07:56.501215+0000","last_unstale":"2023-10-06T05:07:56.505843+0000","last_undegraded":"2023-10-06T05:07:56.505843+0000","last_fullsized":"2023-10-06T05:07:56.505843+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:09:26.304992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.505878+0000","last_change":"2023-10-06T05:07:56.501317+0000","last_active":"2023-10-06T05:07:56.505878+0000","last_peered":"2023-10-06T05:07:56.505878+0000","last_clean":"2023-10-06T05:07:56.505878+0000","last_became_active":"2023-10-06T05:07:56.501032+0000","last_became_peered":"2023-10-06T05:07:56.501032+0000","last_unstale":"2023-10-06T05:07:56.505878+0000","last_undegraded":"2023-10-06T05:07:56.505878+0000","last_fullsized":"2023-10-06T05:07:56.505878+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:45:59.605630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.510618+0000","last_change":"2023-10-06T05:07:56.507300+0000","last_active":"2023-10-06T05:07:56.510618+0000","last_peered":"2023-10-06T05:07:56.510618+0000","last_clean":"2023-10-06T05:07:56.510618+0000","last_became_active":"2023-10-06T05:07:56.507001+0000","last_became_peered":"2023-10-06T05:07:56.507001+0000","last_unstale":"2023-10-06T05:07:56.510618+0000","last_undegraded":"2023-10-06T05:07:56.510618+0000","last_fullsized":"2023-10-06T05:07:56.510618+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T16:39:01.958194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.517713+0000","last_change":"2023-10-06T05:07:56.515811+0000","last_active":"2023-10-06T05:07:56.517713+0000","last_peered":"2023-10-06T05:07:56.517713+0000","last_clean":"2023-10-06T05:07:56.517713+0000","last_became_active":"2023-10-06T05:07:56.515574+0000","last_became_peered":"2023-10-06T05:07:56.515574+0000","last_unstale":"2023-10-06T05:07:56.517713+0000","last_undegraded":"2023-10-06T05:07:56.517713+0000","last_fullsized":"2023-10-06T05:07:56.517713+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:39:38.455191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.503017+0000","last_change":"2023-10-06T05:07:56.494169+0000","last_active":"2023-10-06T05:07:56.503017+0000","last_peered":"2023-10-06T05:07:56.503017+0000","last_clean":"2023-10-06T05:07:56.503017+0000","last_became_active":"2023-10-06T05:07:56.493668+0000","last_became_peered":"2023-10-06T05:07:56.493668+0000","last_unstale":"2023-10-06T05:07:56.503017+0000","last_undegraded":"2023-10-06T05:07:56.503017+0000","last_fullsized":"2023-10-06T05:07:56.503017+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:30:14.123623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.506666+0000","last_change":"2023-10-06T05:07:56.503399+0000","last_active":"2023-10-06T05:07:56.506666+0000","last_peered":"2023-10-06T05:07:56.506666+0000","last_clean":"2023-10-06T05:07:56.506666+0000","last_became_active":"2023-10-06T05:07:56.503118+0000","last_became_peered":"2023-10-06T05:07:56.503118+0000","last_unstale":"2023-10-06T05:07:56.506666+0000","last_undegraded":"2023-10-06T05:07:56.506666+0000","last_fullsized":"2023-10-06T05:07:56.506666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:07:47.377546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.510648+0000","last_change":"2023-10-06T05:07:56.507856+0000","last_active":"2023-10-06T05:07:56.510648+0000","last_peered":"2023-10-06T05:07:56.510648+0000","last_clean":"2023-10-06T05:07:56.510648+0000","last_became_active":"2023-10-06T05:07:56.507585+0000","last_became_peered":"2023-10-06T05:07:56.507585+0000","last_unstale":"2023-10-06T05:07:56.510648+0000","last_undegraded":"2023-10-06T05:07:56.510648+0000","last_fullsized":"2023-10-06T05:07:56.510648+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:10:26.636623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.504516+0000","last_change":"2023-10-06T05:07:56.497371+0000","last_active":"2023-10-06T05:07:56.504516+0000","last_peered":"2023-10-06T05:07:56.504516+0000","last_clean":"2023-10-06T05:07:56.504516+0000","last_became_active":"2023-10-06T05:07:56.497098+0000","last_became_peered":"2023-10-06T05:07:56.497098+0000","last_unstale":"2023-10-06T05:07:56.504516+0000","last_undegraded":"2023-10-06T05:07:56.504516+0000","last_fullsized":"2023-10-06T05:07:56.504516+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:51:32.705673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.497992+0000","last_change":"2023-10-06T05:07:56.497088+0000","last_active":"2023-10-06T05:07:56.497992+0000","last_peered":"2023-10-06T05:07:56.497992+0000","last_clean":"2023-10-06T05:07:56.497992+0000","last_became_active":"2023-10-06T05:07:56.496480+0000","last_became_peered":"2023-10-06T05:07:56.496480+0000","last_unstale":"2023-10-06T05:07:56.497992+0000","last_undegraded":"2023-10-06T05:07:56.497992+0000","last_fullsized":"2023-10-06T05:07:56.497992+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:16:06.816623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.513651+0000","last_change":"2023-10-06T05:07:56.511574+0000","last_active":"2023-10-06T05:07:56.513651+0000","last_peered":"2023-10-06T05:07:56.513651+0000","last_clean":"2023-10-06T05:07:56.513651+0000","last_became_active":"2023-10-06T05:07:56.511303+0000","last_became_peered":"2023-10-06T05:07:56.511303+0000","last_unstale":"2023-10-06T05:07:56.513651+0000","last_undegraded":"2023-10-06T05:07:56.513651+0000","last_fullsized":"2023-10-06T05:07:56.513651+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T16:39:01.958194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":8,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.503991+0000","last_change":"2023-10-06T05:07:56.503991+0000","last_active":"2023-10-06T05:07:56.503991+0000","last_peered":"2023-10-06T05:07:56.503991+0000","last_clean":"2023-10-06T05:07:56.503991+0000","last_became_active":"2023-10-06T05:07:56.503397+0000","last_became_peered":"2023-10-06T05:07:56.503397+0000","last_unstale":"2023-10-06T05:07:56.503991+0000","last_undegraded":"2023-10-06T05:07:56.503991+0000","last_fullsized":"2023-10-06T05:07:56.503991+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:25:34.480685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"42'4","reported_seq":14,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.511666+0000","last_change":"2023-10-06T05:07:55.507746+0000","last_active":"2023-10-06T05:07:56.511666+0000","last_peered":"2023-10-06T05:07:56.511666+0000","last_clean":"2023-10-06T05:07:56.511666+0000","last_became_active":"2023-10-06T05:07:55.507486+0000","last_became_peered":"2023-10-06T05:07:55.507486+0000","last_unstale":"2023-10-06T05:07:56.511666+0000","last_undegraded":"2023-10-06T05:07:56.511666+0000","last_fullsized":"2023-10-06T05:07:56.511666+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_clean_scrub_stamp":"2023-10-06T05:07:54.474583+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:27:33.586746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.505881+0000","last_change":"2023-10-06T05:07:56.500309+0000","last_active":"2023-10-06T05:07:56.505881+0000","last_peered":"2023-10-06T05:07:56.505881+0000","last_clean":"2023-10-06T05:07:56.505881+0000","last_became_active":"2023-10-06T05:07:56.500066+0000","last_became_peered":"2023-10-06T05:07:56.500066+0000","last_unstale":"2023-10-06T05:07:56.505881+0000","last_undegraded":"2023-10-06T05:07:56.505881+0000","last_fullsized":"2023-10-06T05:07:56.505881+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:50:27.863154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"42'1","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.506186+0000","last_change":"2023-10-06T05:07:55.503820+0000","last_active":"2023-10-06T05:07:56.506186+0000","last_peered":"2023-10-06T05:07:56.506186+0000","last_clean":"2023-10-06T05:07:56.506186+0000","last_became_active":"2023-10-06T05:07:55.503546+0000","last_became_peered":"2023-10-06T05:07:55.503546+0000","last_unstale":"2023-10-06T05:07:56.506186+0000","last_undegraded":"2023-10-06T05:07:56.506186+0000","last_fullsized":"2023-10-06T05:07:56.506186+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_clean_scrub_stamp":"2023-10-06T05:07:54.474583+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:16:05.814512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.505572+0000","last_change":"2023-10-06T05:07:56.497520+0000","last_active":"2023-10-06T05:07:56.505572+0000","last_peered":"2023-10-06T05:07:56.505572+0000","last_clean":"2023-10-06T05:07:56.505572+0000","last_became_active":"2023-10-06T05:07:56.497239+0000","last_became_peered":"2023-10-06T05:07:56.497239+0000","last_unstale":"2023-10-06T05:07:56.505572+0000","last_undegraded":"2023-10-06T05:07:56.505572+0000","last_fullsized":"2023-10-06T05:07:56.505572+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:13:46.456823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"42'2","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.510676+0000","last_change":"2023-10-06T05:07:55.497994+0000","last_active":"2023-10-06T05:07:56.510676+0000","last_peered":"2023-10-06T05:07:56.510676+0000","last_clean":"2023-10-06T05:07:56.510676+0000","last_became_active":"2023-10-06T05:07:55.497142+0000","last_became_peered":"2023-10-06T05:07:55.497142+0000","last_unstale":"2023-10-06T05:07:56.510676+0000","last_undegraded":"2023-10-06T05:07:56.510676+0000","last_fullsized":"2023-10-06T05:07:56.510676+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_clean_scrub_stamp":"2023-10-06T05:07:54.474583+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:57:24.408712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":8,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.503243+0000","last_change":"2023-10-06T05:07:56.503243+0000","last_active":"2023-10-06T05:07:56.503243+0000","last_peered":"2023-10-06T05:07:56.503243+0000","last_clean":"2023-10-06T05:07:56.503243+0000","last_became_active":"2023-10-06T05:07:56.502982+0000","last_became_peered":"2023-10-06T05:07:56.502982+0000","last_unstale":"2023-10-06T05:07:56.503243+0000","last_undegraded":"2023-10-06T05:07:56.503243+0000","last_fullsized":"2023-10-06T05:07:56.503243+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:32:18.252043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"42'2","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.511077+0000","last_change":"2023-10-06T05:07:55.497209+0000","last_active":"2023-10-06T05:07:56.511077+0000","last_peered":"2023-10-06T05:07:56.511077+0000","last_clean":"2023-10-06T05:07:56.511077+0000","last_became_active":"2023-10-06T05:07:55.496946+0000","last_became_peered":"2023-10-06T05:07:55.496946+0000","last_unstale":"2023-10-06T05:07:56.511077+0000","last_undegraded":"2023-10-06T05:07:56.511077+0000","last_fullsized":"2023-10-06T05:07:56.511077+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_clean_scrub_stamp":"2023-10-06T05:07:54.474583+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:12:44.633219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.505179+0000","last_change":"2023-10-06T05:07:56.502083+0000","last_active":"2023-10-06T05:07:56.505179+0000","last_peered":"2023-10-06T05:07:56.505179+0000","last_clean":"2023-10-06T05:07:56.505179+0000","last_became_active":"2023-10-06T05:07:56.501590+0000","last_became_peered":"2023-10-06T05:07:56.501590+0000","last_unstale":"2023-10-06T05:07:56.505179+0000","last_undegraded":"2023-10-06T05:07:56.505179+0000","last_fullsized":"2023-10-06T05:07:56.505179+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:00:00.248723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"42'1","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.514261+0000","last_change":"2023-10-06T05:07:55.504304+0000","last_active":"2023-10-06T05:07:56.514261+0000","last_peered":"2023-10-06T05:07:56.514261+0000","last_clean":"2023-10-06T05:07:56.514261+0000","last_became_active":"2023-10-06T05:07:55.504020+0000","last_became_peered":"2023-10-06T05:07:55.504020+0000","last_unstale":"2023-10-06T05:07:56.514261+0000","last_undegraded":"2023-10-06T05:07:56.514261+0000","last_fullsized":"2023-10-06T05:07:56.514261+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_clean_scrub_stamp":"2023-10-06T05:07:54.474583+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:12:31.519443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":8,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.501017+0000","last_change":"2023-10-06T05:07:56.501017+0000","last_active":"2023-10-06T05:07:56.501017+0000","last_peered":"2023-10-06T05:07:56.501017+0000","last_clean":"2023-10-06T05:07:56.501017+0000","last_became_active":"2023-10-06T05:07:56.500761+0000","last_became_peered":"2023-10-06T05:07:56.500761+0000","last_unstale":"2023-10-06T05:07:56.501017+0000","last_undegraded":"2023-10-06T05:07:56.501017+0000","last_fullsized":"2023-10-06T05:07:56.501017+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:00:23.210220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"42'1","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.506054+0000","last_change":"2023-10-06T05:07:55.494922+0000","last_active":"2023-10-06T05:07:56.506054+0000","last_peered":"2023-10-06T05:07:56.506054+0000","last_clean":"2023-10-06T05:07:56.506054+0000","last_became_active":"2023-10-06T05:07:55.494394+0000","last_became_peered":"2023-10-06T05:07:55.494394+0000","last_unstale":"2023-10-06T05:07:56.506054+0000","last_undegraded":"2023-10-06T05:07:56.506054+0000","last_fullsized":"2023-10-06T05:07:56.506054+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_clean_scrub_stamp":"2023-10-06T05:07:54.474583+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:33:33.428032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":8,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.504500+0000","last_change":"2023-10-06T05:07:56.504500+0000","last_active":"2023-10-06T05:07:56.504500+0000","last_peered":"2023-10-06T05:07:56.504500+0000","last_clean":"2023-10-06T05:07:56.504500+0000","last_became_active":"2023-10-06T05:07:56.504082+0000","last_became_peered":"2023-10-06T05:07:56.504082+0000","last_unstale":"2023-10-06T05:07:56.504500+0000","last_undegraded":"2023-10-06T05:07:56.504500+0000","last_fullsized":"2023-10-06T05:07:56.504500+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:30:03.127010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.489306+0000","last_change":"2023-10-06T05:07:55.495889+0000","last_active":"2023-10-06T05:07:56.489306+0000","last_peered":"2023-10-06T05:07:56.489306+0000","last_clean":"2023-10-06T05:07:56.489306+0000","last_became_active":"2023-10-06T05:07:55.495612+0000","last_became_peered":"2023-10-06T05:07:55.495612+0000","last_unstale":"2023-10-06T05:07:56.489306+0000","last_undegraded":"2023-10-06T05:07:56.489306+0000","last_fullsized":"2023-10-06T05:07:56.489306+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_clean_scrub_stamp":"2023-10-06T05:07:54.474583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:59:18.760657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.505824+0000","last_change":"2023-10-06T05:07:56.499141+0000","last_active":"2023-10-06T05:07:56.505824+0000","last_peered":"2023-10-06T05:07:56.505824+0000","last_clean":"2023-10-06T05:07:56.505824+0000","last_became_active":"2023-10-06T05:07:56.498804+0000","last_became_peered":"2023-10-06T05:07:56.498804+0000","last_unstale":"2023-10-06T05:07:56.505824+0000","last_undegraded":"2023-10-06T05:07:56.505824+0000","last_fullsized":"2023-10-06T05:07:56.505824+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:24:45.237400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T05:07:55.507457+0000","last_change":"2023-10-06T05:07:55.502928+0000","last_active":"2023-10-06T05:07:55.507457+0000","last_peered":"2023-10-06T05:07:55.507457+0000","last_clean":"2023-10-06T05:07:55.507457+0000","last_became_active":"2023-10-06T05:07:55.502688+0000","last_became_peered":"2023-10-06T05:07:55.502688+0000","last_unstale":"2023-10-06T05:07:55.507457+0000","last_undegraded":"2023-10-06T05:07:55.507457+0000","last_fullsized":"2023-10-06T05:07:55.507457+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_clean_scrub_stamp":"2023-10-06T05:07:54.474583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:47:49.442467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.504622+0000","last_change":"2023-10-06T05:07:56.500092+0000","last_active":"2023-10-06T05:07:56.504622+0000","last_peered":"2023-10-06T05:07:56.504622+0000","last_clean":"2023-10-06T05:07:56.504622+0000","last_became_active":"2023-10-06T05:07:56.499592+0000","last_became_peered":"2023-10-06T05:07:56.499592+0000","last_unstale":"2023-10-06T05:07:56.504622+0000","last_undegraded":"2023-10-06T05:07:56.504622+0000","last_fullsized":"2023-10-06T05:07:56.504622+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:00:36.630586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.486588+0000","last_change":"2023-10-06T05:07:55.499904+0000","last_active":"2023-10-06T05:07:56.486588+0000","last_peered":"2023-10-06T05:07:56.486588+0000","last_clean":"2023-10-06T05:07:56.486588+0000","last_became_active":"2023-10-06T05:07:55.499588+0000","last_became_peered":"2023-10-06T05:07:55.499588+0000","last_unstale":"2023-10-06T05:07:56.486588+0000","last_undegraded":"2023-10-06T05:07:56.486588+0000","last_fullsized":"2023-10-06T05:07:56.486588+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_clean_scrub_stamp":"2023-10-06T05:07:54.474583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:17:05.388581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":8,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.497841+0000","last_change":"2023-10-06T05:07:56.497841+0000","last_active":"2023-10-06T05:07:56.497841+0000","last_peered":"2023-10-06T05:07:56.497841+0000","last_clean":"2023-10-06T05:07:56.497841+0000","last_became_active":"2023-10-06T05:07:56.497546+0000","last_became_peered":"2023-10-06T05:07:56.497546+0000","last_unstale":"2023-10-06T05:07:56.497841+0000","last_undegraded":"2023-10-06T05:07:56.497841+0000","last_fullsized":"2023-10-06T05:07:56.497841+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:44:40.634049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T05:07:55.502968+0000","last_change":"2023-10-06T05:07:55.499922+0000","last_active":"2023-10-06T05:07:55.502968+0000","last_peered":"2023-10-06T05:07:55.502968+0000","last_clean":"2023-10-06T05:07:55.502968+0000","last_became_active":"2023-10-06T05:07:55.499650+0000","last_became_peered":"2023-10-06T05:07:55.499650+0000","last_unstale":"2023-10-06T05:07:55.502968+0000","last_undegraded":"2023-10-06T05:07:55.502968+0000","last_fullsized":"2023-10-06T05:07:55.502968+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_clean_scrub_stamp":"2023-10-06T05:07:54.474583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:32:17.249933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.516227+0000","last_change":"2023-10-06T05:07:56.514719+0000","last_active":"2023-10-06T05:07:56.516227+0000","last_peered":"2023-10-06T05:07:56.516227+0000","last_clean":"2023-10-06T05:07:56.516227+0000","last_became_active":"2023-10-06T05:07:56.514340+0000","last_became_peered":"2023-10-06T05:07:56.514340+0000","last_unstale":"2023-10-06T05:07:56.516227+0000","last_undegraded":"2023-10-06T05:07:56.516227+0000","last_fullsized":"2023-10-06T05:07:56.516227+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:16:42.675860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"42'3","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.511704+0000","last_change":"2023-10-06T05:07:55.507730+0000","last_active":"2023-10-06T05:07:56.511704+0000","last_peered":"2023-10-06T05:07:56.511704+0000","last_clean":"2023-10-06T05:07:56.511704+0000","last_became_active":"2023-10-06T05:07:55.507440+0000","last_became_peered":"2023-10-06T05:07:55.507440+0000","last_unstale":"2023-10-06T05:07:56.511704+0000","last_undegraded":"2023-10-06T05:07:56.511704+0000","last_fullsized":"2023-10-06T05:07:56.511704+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_clean_scrub_stamp":"2023-10-06T05:07:54.474583+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:09:40.893258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":8,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.508020+0000","last_change":"2023-10-06T05:07:56.508020+0000","last_active":"2023-10-06T05:07:56.508020+0000","last_peered":"2023-10-06T05:07:56.508020+0000","last_clean":"2023-10-06T05:07:56.508020+0000","last_became_active":"2023-10-06T05:07:56.507765+0000","last_became_peered":"2023-10-06T05:07:56.507765+0000","last_unstale":"2023-10-06T05:07:56.508020+0000","last_undegraded":"2023-10-06T05:07:56.508020+0000","last_fullsized":"2023-10-06T05:07:56.508020+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:57:30.893504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"42'6","reported_seq":16,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.517939+0000","last_change":"2023-10-06T05:07:55.501591+0000","last_active":"2023-10-06T05:07:56.517939+0000","last_peered":"2023-10-06T05:07:56.517939+0000","last_clean":"2023-10-06T05:07:56.517939+0000","last_became_active":"2023-10-06T05:07:55.501265+0000","last_became_peered":"2023-10-06T05:07:55.501265+0000","last_unstale":"2023-10-06T05:07:56.517939+0000","last_undegraded":"2023-10-06T05:07:56.517939+0000","last_fullsized":"2023-10-06T05:07:56.517939+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_clean_scrub_stamp":"2023-10-06T05:07:54.474583+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T16:01:12.700532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.505592+0000","last_change":"2023-10-06T05:07:56.499987+0000","last_active":"2023-10-06T05:07:56.505592+0000","last_peered":"2023-10-06T05:07:56.505592+0000","last_clean":"2023-10-06T05:07:56.505592+0000","last_became_active":"2023-10-06T05:07:56.499700+0000","last_became_peered":"2023-10-06T05:07:56.499700+0000","last_unstale":"2023-10-06T05:07:56.505592+0000","last_undegraded":"2023-10-06T05:07:56.505592+0000","last_fullsized":"2023-10-06T05:07:56.505592+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:39:38.455191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"42'2","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.514948+0000","last_change":"2023-10-06T05:07:55.499932+0000","last_active":"2023-10-06T05:07:56.514948+0000","last_peered":"2023-10-06T05:07:56.514948+0000","last_clean":"2023-10-06T05:07:56.514948+0000","last_became_active":"2023-10-06T05:07:55.499650+0000","last_became_peered":"2023-10-06T05:07:55.499650+0000","last_unstale":"2023-10-06T05:07:56.514948+0000","last_undegraded":"2023-10-06T05:07:56.514948+0000","last_fullsized":"2023-10-06T05:07:56.514948+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_clean_scrub_stamp":"2023-10-06T05:07:54.474583+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:40:40.783028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":8,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.499040+0000","last_change":"2023-10-06T05:07:56.499040+0000","last_active":"2023-10-06T05:07:56.499040+0000","last_peered":"2023-10-06T05:07:56.499040+0000","last_clean":"2023-10-06T05:07:56.499040+0000","last_became_active":"2023-10-06T05:07:56.498803+0000","last_became_peered":"2023-10-06T05:07:56.498803+0000","last_unstale":"2023-10-06T05:07:56.499040+0000","last_undegraded":"2023-10-06T05:07:56.499040+0000","last_fullsized":"2023-10-06T05:07:56.499040+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:50:31.905666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T05:07:55.504922+0000","last_change":"2023-10-06T05:07:55.502323+0000","last_active":"2023-10-06T05:07:55.504922+0000","last_peered":"2023-10-06T05:07:55.504922+0000","last_clean":"2023-10-06T05:07:55.504922+0000","last_became_active":"2023-10-06T05:07:55.502126+0000","last_became_peered":"2023-10-06T05:07:55.502126+0000","last_unstale":"2023-10-06T05:07:55.504922+0000","last_undegraded":"2023-10-06T05:07:55.504922+0000","last_fullsized":"2023-10-06T05:07:55.504922+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_clean_scrub_stamp":"2023-10-06T05:07:54.474583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:25:33.478575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.515000+0000","last_change":"2023-10-06T05:07:56.502664+0000","last_active":"2023-10-06T05:07:56.515000+0000","last_peered":"2023-10-06T05:07:56.515000+0000","last_clean":"2023-10-06T05:07:56.515000+0000","last_became_active":"2023-10-06T05:07:56.502507+0000","last_became_peered":"2023-10-06T05:07:56.502507+0000","last_unstale":"2023-10-06T05:07:56.515000+0000","last_undegraded":"2023-10-06T05:07:56.515000+0000","last_fullsized":"2023-10-06T05:07:56.515000+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:49:48.970981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.490325+0000","last_change":"2023-10-06T05:07:55.496701+0000","last_active":"2023-10-06T05:07:56.490325+0000","last_peered":"2023-10-06T05:07:56.490325+0000","last_clean":"2023-10-06T05:07:56.490325+0000","last_became_active":"2023-10-06T05:07:55.494800+0000","last_became_peered":"2023-10-06T05:07:55.494800+0000","last_unstale":"2023-10-06T05:07:56.490325+0000","last_undegraded":"2023-10-06T05:07:56.490325+0000","last_fullsized":"2023-10-06T05:07:56.490325+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_clean_scrub_stamp":"2023-10-06T05:07:54.474583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:56:17.693306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":8,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.504348+0000","last_change":"2023-10-06T05:07:56.504348+0000","last_active":"2023-10-06T05:07:56.504348+0000","last_peered":"2023-10-06T05:07:56.504348+0000","last_clean":"2023-10-06T05:07:56.504348+0000","last_became_active":"2023-10-06T05:07:56.504094+0000","last_became_peered":"2023-10-06T05:07:56.504094+0000","last_unstale":"2023-10-06T05:07:56.504348+0000","last_undegraded":"2023-10-06T05:07:56.504348+0000","last_fullsized":"2023-10-06T05:07:56.504348+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:50:31.905666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"42'8","reported_seq":17,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.514711+0000","last_change":"2023-10-06T05:07:55.499223+0000","last_active":"2023-10-06T05:07:56.514711+0000","last_peered":"2023-10-06T05:07:56.514711+0000","last_clean":"2023-10-06T05:07:56.514711+0000","last_became_active":"2023-10-06T05:07:55.498891+0000","last_became_peered":"2023-10-06T05:07:55.498891+0000","last_unstale":"2023-10-06T05:07:56.514711+0000","last_undegraded":"2023-10-06T05:07:56.514711+0000","last_fullsized":"2023-10-06T05:07:56.514711+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_clean_scrub_stamp":"2023-10-06T05:07:54.474583+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:59:28.831493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.513600+0000","last_change":"2023-10-06T05:07:56.511252+0000","last_active":"2023-10-06T05:07:56.513600+0000","last_peered":"2023-10-06T05:07:56.513600+0000","last_clean":"2023-10-06T05:07:56.513600+0000","last_became_active":"2023-10-06T05:07:56.510997+0000","last_became_peered":"2023-10-06T05:07:56.510997+0000","last_unstale":"2023-10-06T05:07:56.513600+0000","last_undegraded":"2023-10-06T05:07:56.513600+0000","last_fullsized":"2023-10-06T05:07:56.513600+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:19:49.371903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"42'1","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.515525+0000","last_change":"2023-10-06T05:07:55.504049+0000","last_active":"2023-10-06T05:07:56.515525+0000","last_peered":"2023-10-06T05:07:56.515525+0000","last_clean":"2023-10-06T05:07:56.515525+0000","last_became_active":"2023-10-06T05:07:55.503734+0000","last_became_peered":"2023-10-06T05:07:55.503734+0000","last_unstale":"2023-10-06T05:07:56.515525+0000","last_undegraded":"2023-10-06T05:07:56.515525+0000","last_fullsized":"2023-10-06T05:07:56.515525+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_clean_scrub_stamp":"2023-10-06T05:07:54.474583+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:54:47.278321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":8,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.498002+0000","last_change":"2023-10-06T05:07:56.498002+0000","last_active":"2023-10-06T05:07:56.498002+0000","last_peered":"2023-10-06T05:07:56.498002+0000","last_clean":"2023-10-06T05:07:56.498002+0000","last_became_active":"2023-10-06T05:07:56.497774+0000","last_became_peered":"2023-10-06T05:07:56.497774+0000","last_unstale":"2023-10-06T05:07:56.498002+0000","last_undegraded":"2023-10-06T05:07:56.498002+0000","last_fullsized":"2023-10-06T05:07:56.498002+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:19:51.502890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"42'2","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.510262+0000","last_change":"2023-10-06T05:07:55.499170+0000","last_active":"2023-10-06T05:07:56.510262+0000","last_peered":"2023-10-06T05:07:56.510262+0000","last_clean":"2023-10-06T05:07:56.510262+0000","last_became_active":"2023-10-06T05:07:55.498896+0000","last_became_peered":"2023-10-06T05:07:55.498896+0000","last_unstale":"2023-10-06T05:07:56.510262+0000","last_undegraded":"2023-10-06T05:07:56.510262+0000","last_fullsized":"2023-10-06T05:07:56.510262+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_clean_scrub_stamp":"2023-10-06T05:07:54.474583+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:32:22.839464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.505645+0000","last_change":"2023-10-06T05:07:56.500218+0000","last_active":"2023-10-06T05:07:56.505645+0000","last_peered":"2023-10-06T05:07:56.505645+0000","last_clean":"2023-10-06T05:07:56.505645+0000","last_became_active":"2023-10-06T05:07:56.499968+0000","last_became_peered":"2023-10-06T05:07:56.499968+0000","last_unstale":"2023-10-06T05:07:56.505645+0000","last_undegraded":"2023-10-06T05:07:56.505645+0000","last_fullsized":"2023-10-06T05:07:56.505645+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:04:18.068814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"42'1","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.512967+0000","last_change":"2023-10-06T05:07:55.496827+0000","last_active":"2023-10-06T05:07:56.512967+0000","last_peered":"2023-10-06T05:07:56.512967+0000","last_clean":"2023-10-06T05:07:56.512967+0000","last_became_active":"2023-10-06T05:07:55.496252+0000","last_became_peered":"2023-10-06T05:07:55.496252+0000","last_unstale":"2023-10-06T05:07:56.512967+0000","last_undegraded":"2023-10-06T05:07:56.512967+0000","last_fullsized":"2023-10-06T05:07:56.512967+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_clean_scrub_stamp":"2023-10-06T05:07:54.474583+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:32:53.238080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.511412+0000","last_change":"2023-10-06T05:07:56.508203+0000","last_active":"2023-10-06T05:07:56.511412+0000","last_peered":"2023-10-06T05:07:56.511412+0000","last_clean":"2023-10-06T05:07:56.511412+0000","last_became_active":"2023-10-06T05:07:56.507596+0000","last_became_peered":"2023-10-06T05:07:56.507596+0000","last_unstale":"2023-10-06T05:07:56.511412+0000","last_undegraded":"2023-10-06T05:07:56.511412+0000","last_fullsized":"2023-10-06T05:07:56.511412+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:19:49.371903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.491515+0000","last_change":"2023-10-06T05:07:55.503922+0000","last_active":"2023-10-06T05:07:56.491515+0000","last_peered":"2023-10-06T05:07:56.491515+0000","last_clean":"2023-10-06T05:07:56.491515+0000","last_became_active":"2023-10-06T05:07:55.503647+0000","last_became_peered":"2023-10-06T05:07:55.503647+0000","last_unstale":"2023-10-06T05:07:56.491515+0000","last_undegraded":"2023-10-06T05:07:56.491515+0000","last_fullsized":"2023-10-06T05:07:56.491515+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_clean_scrub_stamp":"2023-10-06T05:07:54.474583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:23:02.744729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.515091+0000","last_change":"2023-10-06T05:07:56.503924+0000","last_active":"2023-10-06T05:07:56.515091+0000","last_peered":"2023-10-06T05:07:56.515091+0000","last_clean":"2023-10-06T05:07:56.515091+0000","last_became_active":"2023-10-06T05:07:56.503699+0000","last_became_peered":"2023-10-06T05:07:56.503699+0000","last_unstale":"2023-10-06T05:07:56.515091+0000","last_undegraded":"2023-10-06T05:07:56.515091+0000","last_fullsized":"2023-10-06T05:07:56.515091+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:02:55.738326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"42'2","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.514844+0000","last_change":"2023-10-06T05:07:55.504251+0000","last_active":"2023-10-06T05:07:56.514844+0000","last_peered":"2023-10-06T05:07:56.514844+0000","last_clean":"2023-10-06T05:07:56.514844+0000","last_became_active":"2023-10-06T05:07:55.503715+0000","last_became_peered":"2023-10-06T05:07:55.503715+0000","last_unstale":"2023-10-06T05:07:56.514844+0000","last_undegraded":"2023-10-06T05:07:56.514844+0000","last_fullsized":"2023-10-06T05:07:56.514844+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_clean_scrub_stamp":"2023-10-06T05:07:54.474583+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:49:35.919254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":8,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.494909+0000","last_change":"2023-10-06T05:07:56.494909+0000","last_active":"2023-10-06T05:07:56.494909+0000","last_peered":"2023-10-06T05:07:56.494909+0000","last_clean":"2023-10-06T05:07:56.494909+0000","last_became_active":"2023-10-06T05:07:56.494611+0000","last_became_peered":"2023-10-06T05:07:56.494611+0000","last_unstale":"2023-10-06T05:07:56.494909+0000","last_undegraded":"2023-10-06T05:07:56.494909+0000","last_fullsized":"2023-10-06T05:07:56.494909+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:47:50.444577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.489300+0000","last_change":"2023-10-06T05:07:55.503043+0000","last_active":"2023-10-06T05:07:56.489300+0000","last_peered":"2023-10-06T05:07:56.489300+0000","last_clean":"2023-10-06T05:07:56.489300+0000","last_became_active":"2023-10-06T05:07:55.502754+0000","last_became_peered":"2023-10-06T05:07:55.502754+0000","last_unstale":"2023-10-06T05:07:56.489300+0000","last_undegraded":"2023-10-06T05:07:56.489300+0000","last_fullsized":"2023-10-06T05:07:56.489300+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_clean_scrub_stamp":"2023-10-06T05:07:54.474583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:40:23.087915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.512679+0000","last_change":"2023-10-06T05:07:56.501912+0000","last_active":"2023-10-06T05:07:56.512679+0000","last_peered":"2023-10-06T05:07:56.512679+0000","last_clean":"2023-10-06T05:07:56.512679+0000","last_became_active":"2023-10-06T05:07:56.501696+0000","last_became_peered":"2023-10-06T05:07:56.501696+0000","last_unstale":"2023-10-06T05:07:56.512679+0000","last_undegraded":"2023-10-06T05:07:56.512679+0000","last_fullsized":"2023-10-06T05:07:56.512679+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:50:27.863154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T05:07:55.499254+0000","last_change":"2023-10-06T05:07:55.494220+0000","last_active":"2023-10-06T05:07:55.499254+0000","last_peered":"2023-10-06T05:07:55.499254+0000","last_clean":"2023-10-06T05:07:55.499254+0000","last_became_active":"2023-10-06T05:07:55.493933+0000","last_became_peered":"2023-10-06T05:07:55.493933+0000","last_unstale":"2023-10-06T05:07:55.499254+0000","last_undegraded":"2023-10-06T05:07:55.499254+0000","last_fullsized":"2023-10-06T05:07:55.499254+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_clean_scrub_stamp":"2023-10-06T05:07:54.474583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:25:33.478575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":8,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.507212+0000","last_change":"2023-10-06T05:07:56.507212+0000","last_active":"2023-10-06T05:07:56.507212+0000","last_peered":"2023-10-06T05:07:56.507212+0000","last_clean":"2023-10-06T05:07:56.507212+0000","last_became_active":"2023-10-06T05:07:56.506969+0000","last_became_peered":"2023-10-06T05:07:56.506969+0000","last_unstale":"2023-10-06T05:07:56.507212+0000","last_undegraded":"2023-10-06T05:07:56.507212+0000","last_fullsized":"2023-10-06T05:07:56.507212+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:24:08.047990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T05:07:55.504909+0000","last_change":"2023-10-06T05:07:55.502424+0000","last_active":"2023-10-06T05:07:55.504909+0000","last_peered":"2023-10-06T05:07:55.504909+0000","last_clean":"2023-10-06T05:07:55.504909+0000","last_became_active":"2023-10-06T05:07:55.502160+0000","last_became_peered":"2023-10-06T05:07:55.502160+0000","last_unstale":"2023-10-06T05:07:55.504909+0000","last_undegraded":"2023-10-06T05:07:55.504909+0000","last_fullsized":"2023-10-06T05:07:55.504909+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_clean_scrub_stamp":"2023-10-06T05:07:54.474583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:32:17.249933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.504423+0000","last_change":"2023-10-06T05:07:56.498223+0000","last_active":"2023-10-06T05:07:56.504423+0000","last_peered":"2023-10-06T05:07:56.504423+0000","last_clean":"2023-10-06T05:07:56.504423+0000","last_became_active":"2023-10-06T05:07:56.497954+0000","last_became_peered":"2023-10-06T05:07:56.497954+0000","last_unstale":"2023-10-06T05:07:56.504423+0000","last_undegraded":"2023-10-06T05:07:56.504423+0000","last_fullsized":"2023-10-06T05:07:56.504423+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:27:54.501823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T05:07:55.500347+0000","last_change":"2023-10-06T05:07:55.496823+0000","last_active":"2023-10-06T05:07:55.500347+0000","last_peered":"2023-10-06T05:07:55.500347+0000","last_clean":"2023-10-06T05:07:55.500347+0000","last_became_active":"2023-10-06T05:07:55.495650+0000","last_became_peered":"2023-10-06T05:07:55.495650+0000","last_unstale":"2023-10-06T05:07:55.500347+0000","last_undegraded":"2023-10-06T05:07:55.500347+0000","last_fullsized":"2023-10-06T05:07:55.500347+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_clean_scrub_stamp":"2023-10-06T05:07:54.474583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:07:49.191753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T05:07:55.486395+0000","last_change":"2023-10-06T05:06:52.627232+0000","last_active":"2023-10-06T05:07:55.486395+0000","last_peered":"2023-10-06T05:07:55.486395+0000","last_clean":"2023-10-06T05:07:55.486395+0000","last_became_active":"2023-10-06T05:06:52.626936+0000","last_became_peered":"2023-10-06T05:06:52.626936+0000","last_unstale":"2023-10-06T05:07:55.486395+0000","last_undegraded":"2023-10-06T05:07:55.486395+0000","last_fullsized":"2023-10-06T05:07:55.486395+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:20:13.925170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.505859+0000","last_change":"2023-10-06T05:07:56.499826+0000","last_active":"2023-10-06T05:07:56.505859+0000","last_peered":"2023-10-06T05:07:56.505859+0000","last_clean":"2023-10-06T05:07:56.505859+0000","last_became_active":"2023-10-06T05:07:56.499563+0000","last_became_peered":"2023-10-06T05:07:56.499563+0000","last_unstale":"2023-10-06T05:07:56.505859+0000","last_undegraded":"2023-10-06T05:07:56.505859+0000","last_fullsized":"2023-10-06T05:07:56.505859+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:21:30.591642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"43'1","reported_seq":11,"reported_epoch":43,"state":"active+clean","last_fresh":"2023-10-06T05:07:58.385493+0000","last_change":"2023-10-06T05:07:55.500013+0000","last_active":"2023-10-06T05:07:58.385493+0000","last_peered":"2023-10-06T05:07:58.385493+0000","last_clean":"2023-10-06T05:07:58.385493+0000","last_became_active":"2023-10-06T05:07:55.499366+0000","last_became_peered":"2023-10-06T05:07:55.499366+0000","last_unstale":"2023-10-06T05:07:58.385493+0000","last_undegraded":"2023-10-06T05:07:58.385493+0000","last_fullsized":"2023-10-06T05:07:58.385493+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_clean_scrub_stamp":"2023-10-06T05:07:54.474583+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:16:46.996093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":31,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.489452+0000","last_change":"2023-10-06T05:06:52.587795+0000","last_active":"2023-10-06T05:07:56.489452+0000","last_peered":"2023-10-06T05:07:56.489452+0000","last_clean":"2023-10-06T05:07:56.489452+0000","last_became_active":"2023-10-06T05:06:52.587407+0000","last_became_peered":"2023-10-06T05:06:52.587407+0000","last_unstale":"2023-10-06T05:07:56.489452+0000","last_undegraded":"2023-10-06T05:07:56.489452+0000","last_fullsized":"2023-10-06T05:07:56.489452+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:36:31.526532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":8,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.507196+0000","last_change":"2023-10-06T05:07:56.507196+0000","last_active":"2023-10-06T05:07:56.507196+0000","last_peered":"2023-10-06T05:07:56.507196+0000","last_clean":"2023-10-06T05:07:56.507196+0000","last_became_active":"2023-10-06T05:07:56.506806+0000","last_became_peered":"2023-10-06T05:07:56.506806+0000","last_unstale":"2023-10-06T05:07:56.507196+0000","last_undegraded":"2023-10-06T05:07:56.507196+0000","last_fullsized":"2023-10-06T05:07:56.507196+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:36:03.141466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T05:07:55.509630+0000","last_change":"2023-10-06T05:07:55.504758+0000","last_active":"2023-10-06T05:07:55.509630+0000","last_peered":"2023-10-06T05:07:55.509630+0000","last_clean":"2023-10-06T05:07:55.509630+0000","last_became_active":"2023-10-06T05:07:55.504470+0000","last_became_peered":"2023-10-06T05:07:55.504470+0000","last_unstale":"2023-10-06T05:07:55.509630+0000","last_undegraded":"2023-10-06T05:07:55.509630+0000","last_fullsized":"2023-10-06T05:07:55.509630+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_clean_scrub_stamp":"2023-10-06T05:07:54.474583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:03:15.774462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":28,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.499543+0000","last_change":"2023-10-06T05:06:52.675238+0000","last_active":"2023-10-06T05:07:56.499543+0000","last_peered":"2023-10-06T05:07:56.499543+0000","last_clean":"2023-10-06T05:07:56.499543+0000","last_became_active":"2023-10-06T05:06:52.674952+0000","last_became_peered":"2023-10-06T05:06:52.674952+0000","last_unstale":"2023-10-06T05:07:56.499543+0000","last_undegraded":"2023-10-06T05:07:56.499543+0000","last_fullsized":"2023-10-06T05:07:56.499543+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:27:51.141184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":8,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.499948+0000","last_change":"2023-10-06T05:07:56.499948+0000","last_active":"2023-10-06T05:07:56.499948+0000","last_peered":"2023-10-06T05:07:56.499948+0000","last_clean":"2023-10-06T05:07:56.499948+0000","last_became_active":"2023-10-06T05:07:56.499705+0000","last_became_peered":"2023-10-06T05:07:56.499705+0000","last_unstale":"2023-10-06T05:07:56.499948+0000","last_undegraded":"2023-10-06T05:07:56.499948+0000","last_fullsized":"2023-10-06T05:07:56.499948+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:05:12.786696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"42'2","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.515946+0000","last_change":"2023-10-06T05:07:55.503632+0000","last_active":"2023-10-06T05:07:56.515946+0000","last_peered":"2023-10-06T05:07:56.515946+0000","last_clean":"2023-10-06T05:07:56.515946+0000","last_became_active":"2023-10-06T05:07:55.503429+0000","last_became_peered":"2023-10-06T05:07:55.503429+0000","last_unstale":"2023-10-06T05:07:56.515946+0000","last_undegraded":"2023-10-06T05:07:56.515946+0000","last_fullsized":"2023-10-06T05:07:56.515946+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_clean_scrub_stamp":"2023-10-06T05:07:54.474583+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:45:37.139013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":31,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.489583+0000","last_change":"2023-10-06T05:06:52.587778+0000","last_active":"2023-10-06T05:07:56.489583+0000","last_peered":"2023-10-06T05:07:56.489583+0000","last_clean":"2023-10-06T05:07:56.489583+0000","last_became_active":"2023-10-06T05:06:52.587388+0000","last_became_peered":"2023-10-06T05:06:52.587388+0000","last_unstale":"2023-10-06T05:07:56.489583+0000","last_undegraded":"2023-10-06T05:07:56.489583+0000","last_fullsized":"2023-10-06T05:07:56.489583+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:45:52.502298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":181,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T05:07:55.490762+0000","last_change":"2023-10-06T05:06:46.307169+0000","last_active":"2023-10-06T05:07:55.490762+0000","last_peered":"2023-10-06T05:07:55.490762+0000","last_clean":"2023-10-06T05:07:55.490762+0000","last_became_active":"2023-10-06T05:06:46.306107+0000","last_became_peered":"2023-10-06T05:06:46.306107+0000","last_unstale":"2023-10-06T05:07:55.490762+0000","last_undegraded":"2023-10-06T05:07:55.490762+0000","last_fullsized":"2023-10-06T05:07:55.490762+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:45.018006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:45.018006+0000","last_clean_scrub_stamp":"2023-10-06T05:06:45.018006+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:38:17.064131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":8,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.502108+0000","last_change":"2023-10-06T05:07:56.502108+0000","last_active":"2023-10-06T05:07:56.502108+0000","last_peered":"2023-10-06T05:07:56.502108+0000","last_clean":"2023-10-06T05:07:56.502108+0000","last_became_active":"2023-10-06T05:07:56.501855+0000","last_became_peered":"2023-10-06T05:07:56.501855+0000","last_unstale":"2023-10-06T05:07:56.502108+0000","last_undegraded":"2023-10-06T05:07:56.502108+0000","last_fullsized":"2023-10-06T05:07:56.502108+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:07:44.560707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T05:07:55.502308+0000","last_change":"2023-10-06T05:07:55.497336+0000","last_active":"2023-10-06T05:07:55.502308+0000","last_peered":"2023-10-06T05:07:55.502308+0000","last_clean":"2023-10-06T05:07:55.502308+0000","last_became_active":"2023-10-06T05:07:55.497065+0000","last_became_peered":"2023-10-06T05:07:55.497065+0000","last_unstale":"2023-10-06T05:07:55.502308+0000","last_undegraded":"2023-10-06T05:07:55.502308+0000","last_fullsized":"2023-10-06T05:07:55.502308+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_clean_scrub_stamp":"2023-10-06T05:07:54.474583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:13:54.766249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":28,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T05:07:55.490849+0000","last_change":"2023-10-06T05:06:52.588475+0000","last_active":"2023-10-06T05:07:55.490849+0000","last_peered":"2023-10-06T05:07:55.490849+0000","last_clean":"2023-10-06T05:07:55.490849+0000","last_became_active":"2023-10-06T05:06:52.588134+0000","last_became_peered":"2023-10-06T05:06:52.588134+0000","last_unstale":"2023-10-06T05:07:55.490849+0000","last_undegraded":"2023-10-06T05:07:55.490849+0000","last_fullsized":"2023-10-06T05:07:55.490849+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:35:05.269778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.506698+0000","last_change":"2023-10-06T05:07:56.503635+0000","last_active":"2023-10-06T05:07:56.506698+0000","last_peered":"2023-10-06T05:07:56.506698+0000","last_clean":"2023-10-06T05:07:56.506698+0000","last_became_active":"2023-10-06T05:07:56.503368+0000","last_became_peered":"2023-10-06T05:07:56.503368+0000","last_unstale":"2023-10-06T05:07:56.506698+0000","last_undegraded":"2023-10-06T05:07:56.506698+0000","last_fullsized":"2023-10-06T05:07:56.506698+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:23:03.746840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T05:07:55.507716+0000","last_change":"2023-10-06T05:07:55.505883+0000","last_active":"2023-10-06T05:07:55.507716+0000","last_peered":"2023-10-06T05:07:55.507716+0000","last_clean":"2023-10-06T05:07:55.507716+0000","last_became_active":"2023-10-06T05:07:55.505622+0000","last_became_peered":"2023-10-06T05:07:55.505622+0000","last_unstale":"2023-10-06T05:07:55.507716+0000","last_undegraded":"2023-10-06T05:07:55.507716+0000","last_fullsized":"2023-10-06T05:07:55.507716+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_clean_scrub_stamp":"2023-10-06T05:07:54.474583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T16:44:35.675612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":28,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T05:07:55.492816+0000","last_change":"2023-10-06T05:06:52.627289+0000","last_active":"2023-10-06T05:07:55.492816+0000","last_peered":"2023-10-06T05:07:55.492816+0000","last_clean":"2023-10-06T05:07:55.492816+0000","last_became_active":"2023-10-06T05:06:52.626937+0000","last_became_peered":"2023-10-06T05:06:52.626937+0000","last_unstale":"2023-10-06T05:07:55.492816+0000","last_undegraded":"2023-10-06T05:07:55.492816+0000","last_fullsized":"2023-10-06T05:07:55.492816+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:46:59.070674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.515005+0000","last_change":"2023-10-06T05:07:56.506104+0000","last_active":"2023-10-06T05:07:56.515005+0000","last_peered":"2023-10-06T05:07:56.515005+0000","last_clean":"2023-10-06T05:07:56.515005+0000","last_became_active":"2023-10-06T05:07:56.505829+0000","last_became_peered":"2023-10-06T05:07:56.505829+0000","last_unstale":"2023-10-06T05:07:56.515005+0000","last_undegraded":"2023-10-06T05:07:56.515005+0000","last_fullsized":"2023-10-06T05:07:56.515005+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:27:31.635411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.487139+0000","last_change":"2023-10-06T05:07:55.492476+0000","last_active":"2023-10-06T05:07:56.487139+0000","last_peered":"2023-10-06T05:07:56.487139+0000","last_clean":"2023-10-06T05:07:56.487139+0000","last_became_active":"2023-10-06T05:07:55.492199+0000","last_became_peered":"2023-10-06T05:07:55.492199+0000","last_unstale":"2023-10-06T05:07:56.487139+0000","last_undegraded":"2023-10-06T05:07:56.487139+0000","last_fullsized":"2023-10-06T05:07:56.487139+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_clean_scrub_stamp":"2023-10-06T05:07:54.474583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T17:05:11.493391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":27,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T05:07:55.489889+0000","last_change":"2023-10-06T05:06:52.588853+0000","last_active":"2023-10-06T05:07:55.489889+0000","last_peered":"2023-10-06T05:07:55.489889+0000","last_clean":"2023-10-06T05:07:55.489889+0000","last_became_active":"2023-10-06T05:06:52.588561+0000","last_became_peered":"2023-10-06T05:06:52.588561+0000","last_unstale":"2023-10-06T05:07:55.489889+0000","last_undegraded":"2023-10-06T05:07:55.489889+0000","last_fullsized":"2023-10-06T05:07:55.489889+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:06:28.729493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.515110+0000","last_change":"2023-10-06T05:07:56.510457+0000","last_active":"2023-10-06T05:07:56.515110+0000","last_peered":"2023-10-06T05:07:56.515110+0000","last_clean":"2023-10-06T05:07:56.515110+0000","last_became_active":"2023-10-06T05:07:56.510184+0000","last_became_peered":"2023-10-06T05:07:56.510184+0000","last_unstale":"2023-10-06T05:07:56.515110+0000","last_undegraded":"2023-10-06T05:07:56.515110+0000","last_fullsized":"2023-10-06T05:07:56.515110+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:59:52.915174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T05:07:55.506995+0000","last_change":"2023-10-06T05:07:55.504325+0000","last_active":"2023-10-06T05:07:55.506995+0000","last_peered":"2023-10-06T05:07:55.506995+0000","last_clean":"2023-10-06T05:07:55.506995+0000","last_became_active":"2023-10-06T05:07:55.504037+0000","last_became_peered":"2023-10-06T05:07:55.504037+0000","last_unstale":"2023-10-06T05:07:55.506995+0000","last_undegraded":"2023-10-06T05:07:55.506995+0000","last_fullsized":"2023-10-06T05:07:55.506995+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_clean_scrub_stamp":"2023-10-06T05:07:54.474583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:44:32.978639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.491222+0000","last_change":"2023-10-06T05:06:52.589519+0000","last_active":"2023-10-06T05:07:56.491222+0000","last_peered":"2023-10-06T05:07:56.491222+0000","last_clean":"2023-10-06T05:07:56.491222+0000","last_became_active":"2023-10-06T05:06:52.589246+0000","last_became_peered":"2023-10-06T05:06:52.589246+0000","last_unstale":"2023-10-06T05:07:56.491222+0000","last_undegraded":"2023-10-06T05:07:56.491222+0000","last_fullsized":"2023-10-06T05:07:56.491222+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:54:23.352764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509459,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297776,"kb_used_data":488,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074064,"statfs":{"total":96636764160,"available":96331841536,"internally_reserved":0,"allocated":499712,"data_stored":219280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.437},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.46600000000000003}]},{"osd":1,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.39200000000000002}]},{"osd":2,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.41899999999999998}]},{"osd":3,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.54300000000000004}]},{"osd":4,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.57499999999999996}]},{"osd":5,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.64900000000000002}]},{"osd":7,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.70499999999999996}]}]},{"osd":5,"up_from":15,"seq":64424509459,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297772,"kb_used_data":484,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074068,"statfs":{"total":96636764160,"available":96331845632,"internally_reserved":0,"allocated":495616,"data_stored":218706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.44700000000000001}]},{"osd":1,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.5},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.48099999999999998}]},{"osd":2,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.63100000000000001}]},{"osd":3,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.65000000000000002}]},{"osd":4,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.752}]},{"osd":6,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.73699999999999999}]},{"osd":7,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.70799999999999996}]}]},{"osd":0,"up_from":14,"seq":60129542163,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298340,"kb_used_data":1052,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073500,"statfs":{"total":96636764160,"available":96331264000,"internally_reserved":0,"allocated":1077248,"data_stored":802077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":1.2809999999999999,"5min":1.2809999999999999,"15min":1.2809999999999999},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.35499999999999998}]},{"osd":2,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":4.1529999999999996,"5min":4.1529999999999996,"15min":4.1529999999999996},"last":0.308},{"interface":"front","average":{"1min":0.372,"5min":0.372,"15min":0.372},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":1.105,"5min":1.105,"15min":1.105},"last":0.26200000000000001}]},{"osd":3,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":1.327,"5min":1.327,"15min":1.327},"last":0.246},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":4.1900000000000004,"5min":4.1900000000000004,"15min":4.1900000000000004},"last":0.40000000000000002}]},{"osd":4,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.53200000000000003}]},{"osd":5,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.304,"5min":1.304,"15min":1.304},"last":0.60999999999999999}]},{"osd":6,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.622},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.54600000000000004}]},{"osd":7,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":1.079,"5min":1.079,"15min":1.079},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.64400000000000002}]}]},{"osd":7,"up_from":13,"seq":55834574866,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298340,"kb_used_data":1052,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073500,"statfs":{"total":96636764160,"available":96331264000,"internally_reserved":0,"allocated":1077248,"data_stored":802133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.49299999999999999}]},{"osd":1,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.36599999999999999}]},{"osd":2,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.45100000000000001}]},{"osd":3,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.56799999999999995}]},{"osd":4,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.60899999999999999}]},{"osd":5,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.54800000000000004}]},{"osd":6,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.66700000000000004}]}]},{"osd":1,"up_from":14,"seq":60129542162,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297760,"kb_used_data":472,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074080,"statfs":{"total":96636764160,"available":96331857920,"internally_reserved":0,"allocated":483328,"data_stored":211745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":1.8879999999999999,"5min":1.8879999999999999,"15min":1.8879999999999999},"last":0.251},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":1.595,"5min":1.595,"15min":1.595},"last":0.378}]},{"osd":2,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.123,"5min":1.123,"15min":1.123},"last":0.32300000000000001}]},{"osd":3,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.179,"5min":1.179,"15min":1.179},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1.657,"5min":1.657,"15min":1.657},"last":0.34200000000000003}]},{"osd":4,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":1.6319999999999999,"5min":1.6319999999999999,"15min":1.6319999999999999},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.837,"5min":1.837,"15min":1.837},"last":0.496}]},{"osd":5,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.8100000000000001,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.56999999999999995}]},{"osd":6,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.6850000000000001,"5min":1.6850000000000001,"15min":1.6850000000000001},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.8620000000000001,"5min":1.8620000000000001,"15min":1.8620000000000001},"last":0.52500000000000002}]},{"osd":7,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.7789999999999999,"5min":1.7789999999999999,"15min":1.7789999999999999},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.53900000000000003}]}]},{"osd":2,"up_from":15,"seq":64424509458,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297784,"kb_used_data":496,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074056,"statfs":{"total":96636764160,"available":96331833344,"internally_reserved":0,"allocated":507904,"data_stored":220194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.311,"5min":0.311,"15min":0.311},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.25800000000000001}]},{"osd":1,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"last":0.317}]},{"osd":3,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"last":0.36299999999999999}]},{"osd":4,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.70899999999999996}]},{"osd":5,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.71799999999999997}]},{"osd":6,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.753},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.63300000000000001}]},{"osd":7,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.78100000000000003}]}]},{"osd":3,"up_from":15,"seq":64424509458,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297780,"kb_used_data":492,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074060,"statfs":{"total":96636764160,"available":96331837440,"internally_reserved":0,"allocated":503808,"data_stored":219246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":3.391,"5min":3.391,"15min":3.391},"last":0.27800000000000002}]},{"osd":1,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":2.8740000000000001,"5min":2.8740000000000001,"15min":2.8740000000000001},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.376,"5min":0.376,"15min":0.376},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.308}]},{"osd":2,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":7.2210000000000001,"5min":7.2210000000000001,"15min":7.2210000000000001},"last":0.378}]},{"osd":4,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.018,"5min":1.018,"15min":1.018},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.6930000000000001,"5min":1.6930000000000001,"15min":1.6930000000000001},"last":0.71499999999999997}]},{"osd":5,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.607,"5min":1.607,"15min":1.607},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.67200000000000004}]},{"osd":6,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.657,"5min":1.657,"15min":1.657},"last":0.74199999999999999}]},{"osd":7,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.877,"5min":1.877,"15min":1.877},"last":0.64500000000000002}]}]},{"osd":4,"up_from":15,"seq":64424509459,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297780,"kb_used_data":492,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074060,"statfs":{"total":96636764160,"available":96331837440,"internally_reserved":0,"allocated":503808,"data_stored":220172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.59099999999999997}]},{"osd":1,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.63600000000000001}]},{"osd":2,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.69299999999999995}]},{"osd":3,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":11.385999999999999,"5min":11.385999999999999,"15min":11.385999999999999},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.72999999999999998}]},{"osd":5,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.66800000000000004}]},{"osd":6,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.75700000000000001}]},{"osd":7,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.81399999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-06T05:08:00.572 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-06T05:08:00.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.713+0000 7fba4cb40700 1 Processor -- start 2023-10-06T05:08:00.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.713+0000 7fba4cb40700 1 -- start start 2023-10-06T05:08:00.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.714+0000 7fba4cb40700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fba48159440 0x7fba480b3090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:00.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.714+0000 7fba4cb40700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba480cb7a0 con 0x7fba480b2100 2023-10-06T05:08:00.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.714+0000 7fba4cb40700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba480b0860 con 0x7fba480b35d0 2023-10-06T05:08:00.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.714+0000 7fba4cb40700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba480cb150 con 0x7fba48159440 2023-10-06T05:08:00.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.714+0000 7fba47fff700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fba480b2100 0x7fba480b2580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:38360/0 (socket says 172.21.15.98:38360) 2023-10-06T05:08:00.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.714+0000 7fba47fff700 1 -- 172.21.15.98:0/1133612159 learned_addr learned my addr 172.21.15.98:0/1133612159 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:08:00.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.714+0000 7fba477fe700 1 --2- 172.21.15.98:0/1133612159 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fba48159440 0x7fba480b3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:00.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.714+0000 7fba467fc700 1 -- 172.21.15.98:0/1133612159 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 419406779 0 0) 0x7fba480cb7a0 con 0x7fba480b2100 2023-10-06T05:08:00.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.714+0000 7fba467fc700 1 -- 172.21.15.98:0/1133612159 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba30003570 con 0x7fba480b2100 2023-10-06T05:08:00.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.714+0000 7fba467fc700 1 -- 172.21.15.98:0/1133612159 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1825890143 0 0) 0x7fba480b0860 con 0x7fba480b35d0 2023-10-06T05:08:00.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.714+0000 7fba467fc700 1 -- 172.21.15.98:0/1133612159 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba480cb7a0 con 0x7fba480b35d0 2023-10-06T05:08:00.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.714+0000 7fba467fc700 1 -- 172.21.15.98:0/1133612159 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1746138750 0 0) 0x7fba30003570 con 0x7fba480b2100 2023-10-06T05:08:00.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.714+0000 7fba467fc700 1 -- 172.21.15.98:0/1133612159 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fba48159440 msgr2=0x7fba480b3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:00.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.714+0000 7fba467fc700 1 --2- 172.21.15.98:0/1133612159 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fba48159440 0x7fba480b3090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:00.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.714+0000 7fba467fc700 1 -- 172.21.15.98:0/1133612159 >> v1:172.21.15.174:6789/0 conn(0x7fba480b35d0 legacy=0x7fba480cbc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:00.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.714+0000 7fba467fc700 1 -- 172.21.15.98:0/1133612159 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba480cb430 con 0x7fba480b2100 2023-10-06T05:08:00.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.714+0000 7fba467fc700 1 -- 172.21.15.98:0/1133612159 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1543991876 0 0) 0x7fba40001470 con 0x7fba480b2100 2023-10-06T05:08:00.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.715+0000 7fba467fc700 1 -- 172.21.15.98:0/1133612159 >> v1:172.21.15.98:6789/0 conn(0x7fba480b2100 legacy=0x7fba480b2580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:00.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.715+0000 7fba467fc700 1 --2- 172.21.15.98:0/1133612159 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fba48159440 0x7fba30003c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:00.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.715+0000 7fba467fc700 1 --2- 172.21.15.98:0/1133612159 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba30004180 0x7fba30004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:00.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.715+0000 7fba467fc700 1 --2- 172.21.15.98:0/1133612159 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fba30004d50 0x7fba300051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:00.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.715+0000 7fba467fc700 1 -- 172.21.15.98:0/1133612159 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba480cb150 con 0x7fba30004180 2023-10-06T05:08:00.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.715+0000 7fba467fc700 1 -- 172.21.15.98:0/1133612159 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba3000bc50 con 0x7fba30004d50 2023-10-06T05:08:00.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.715+0000 7fba467fc700 1 -- 172.21.15.98:0/1133612159 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba3000be10 con 0x7fba48159440 2023-10-06T05:08:00.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.715+0000 7fba46ffd700 1 --2- 172.21.15.98:0/1133612159 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba30004180 0x7fba30004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:00.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.715+0000 7fba477fe700 1 --2- 172.21.15.98:0/1133612159 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fba48159440 0x7fba30003c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:00.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.715+0000 7fba47fff700 1 --2- 172.21.15.98:0/1133612159 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fba30004d50 0x7fba300051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:00.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.715+0000 7fba46ffd700 1 -- 172.21.15.98:0/1133612159 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fba48159440 msgr2=0x7fba30003c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:00.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.715+0000 7fba46ffd700 1 --2- 172.21.15.98:0/1133612159 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fba48159440 0x7fba30003c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:00.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.715+0000 7fba46ffd700 1 -- 172.21.15.98:0/1133612159 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fba30004d50 msgr2=0x7fba300051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:00.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.715+0000 7fba46ffd700 1 --2- 172.21.15.98:0/1133612159 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fba30004d50 0x7fba300051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:00.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.715+0000 7fba46ffd700 1 -- 172.21.15.98:0/1133612159 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba30003570 con 0x7fba30004180 2023-10-06T05:08:00.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.715+0000 7fba47fff700 1 --2- 172.21.15.98:0/1133612159 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fba30004d50 0x7fba300051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:08:00.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.715+0000 7fba46ffd700 1 --2- 172.21.15.98:0/1133612159 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba30004180 0x7fba30004810 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fba3c0019e0 tx=0x7fba3c007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c5cdeba18d3eb49 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:00.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.715+0000 7fba467fc700 1 -- 172.21.15.98:0/1133612159 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba3c00c5d0 con 0x7fba30004180 2023-10-06T05:08:00.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.715+0000 7fba467fc700 1 -- 172.21.15.98:0/1133612159 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba3c00c730 con 0x7fba30004180 2023-10-06T05:08:00.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.715+0000 7fba467fc700 1 -- 172.21.15.98:0/1133612159 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba3c00cad0 con 0x7fba30004180 2023-10-06T05:08:00.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.716+0000 7fba4cb40700 1 -- 172.21.15.98:0/1133612159 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba30004180 msgr2=0x7fba30004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:00.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.716+0000 7fba4cb40700 1 --2- 172.21.15.98:0/1133612159 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba30004180 0x7fba30004810 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fba3c0019e0 tx=0x7fba3c007d30 comp rx=0 tx=0).stop 2023-10-06T05:08:00.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.716+0000 7fba47fff700 1 -- 172.21.15.98:0/1133612159 reap_dead start 2023-10-06T05:08:00.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.716+0000 7fba4cb40700 1 -- 172.21.15.98:0/1133612159 shutdown_connections 2023-10-06T05:08:00.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.716+0000 7fba4cb40700 1 -- 172.21.15.98:0/1133612159 >> 172.21.15.98:0/1133612159 conn(0x7fba4806ece0 msgr2=0x7fba4806f0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:00.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.716+0000 7fba4cb40700 1 -- 172.21.15.98:0/1133612159 shutdown_connections 2023-10-06T05:08:00.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.716+0000 7fba4cb40700 1 -- 172.21.15.98:0/1133612159 wait complete. 2023-10-06T05:08:00.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.717+0000 7fba4cb40700 1 Processor -- start 2023-10-06T05:08:00.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.717+0000 7fba4cb40700 1 -- start start 2023-10-06T05:08:00.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.718+0000 7fba4cb40700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fba30004180 0x7fba48155950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:00.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.718+0000 7fba4cb40700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fba480c7f20 0x7fba4815a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:00.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.718+0000 7fba4cb40700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba480d0250 0x7fba480d06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:00.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.718+0000 7fba4cb40700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba480c87e0 con 0x7fba480d0250 2023-10-06T05:08:00.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.718+0000 7fba4cb40700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba480d0be0 con 0x7fba480c7f20 2023-10-06T05:08:00.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.718+0000 7fba4cb40700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba4815a930 con 0x7fba30004180 2023-10-06T05:08:00.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.718+0000 7fba47fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba480d0250 0x7fba480d06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:00.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.718+0000 7fba47fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba480d0250 0x7fba480d06a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54798/0 (socket says 172.21.15.98:54798) 2023-10-06T05:08:00.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.718+0000 7fba47fff700 1 -- 172.21.15.98:0/2256299435 learned_addr learned my addr 172.21.15.98:0/2256299435 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:08:00.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.718+0000 7fba477fe700 1 --2- 172.21.15.98:0/2256299435 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fba30004180 0x7fba48155950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:00.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.718+0000 7fba46ffd700 1 --2- 172.21.15.98:0/2256299435 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fba480c7f20 0x7fba4815a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:00.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.718+0000 7fba47fff700 1 -- 172.21.15.98:0/2256299435 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fba30004180 msgr2=0x7fba48155950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:00.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.718+0000 7fba47fff700 1 --2- 172.21.15.98:0/2256299435 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fba30004180 0x7fba48155950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:00.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.718+0000 7fba47fff700 1 -- 172.21.15.98:0/2256299435 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fba480c7f20 msgr2=0x7fba4815a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:00.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.718+0000 7fba47fff700 1 --2- 172.21.15.98:0/2256299435 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fba480c7f20 0x7fba4815a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:00.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.718+0000 7fba47fff700 1 -- 172.21.15.98:0/2256299435 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba4815ac10 con 0x7fba480d0250 2023-10-06T05:08:00.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.718+0000 7fba47fff700 1 --2- 172.21.15.98:0/2256299435 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba480d0250 0x7fba480d06a0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fba48069150 tx=0x7fba40000f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:00.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.719+0000 7fba2f7fe700 1 -- 172.21.15.98:0/2256299435 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba40012590 con 0x7fba480d0250 2023-10-06T05:08:00.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.719+0000 7fba2f7fe700 1 -- 172.21.15.98:0/2256299435 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba400126f0 con 0x7fba480d0250 2023-10-06T05:08:00.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.719+0000 7fba4cb40700 1 -- 172.21.15.98:0/2256299435 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba4815aef0 con 0x7fba480d0250 2023-10-06T05:08:00.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.719+0000 7fba2f7fe700 1 -- 172.21.15.98:0/2256299435 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba40012ab0 con 0x7fba480d0250 2023-10-06T05:08:00.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.719+0000 7fba4cb40700 1 -- 172.21.15.98:0/2256299435 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba48155e90 con 0x7fba480d0250 2023-10-06T05:08:00.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.720+0000 7fba4cb40700 1 -- 172.21.15.98:0/2256299435 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba4805f230 con 0x7fba480d0250 2023-10-06T05:08:00.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.720+0000 7fba2f7fe700 1 -- 172.21.15.98:0/2256299435 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fba4001d460 con 0x7fba480d0250 2023-10-06T05:08:00.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.720+0000 7fba2f7fe700 1 --2- 172.21.15.98:0/2256299435 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fba34066f30 0x7fba340693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:00.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.720+0000 7fba477fe700 1 --2- 172.21.15.98:0/2256299435 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fba34066f30 0x7fba340693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:00.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.720+0000 7fba2f7fe700 1 -- 172.21.15.98:0/2256299435 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6378+0+0 (secure 0 0 0) 0x7fba40084610 con 0x7fba480d0250 2023-10-06T05:08:00.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.721+0000 7fba477fe700 1 --2- 172.21.15.98:0/2256299435 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fba34066f30 0x7fba340693e0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fba38009fd0 tx=0x7fba3800b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:00.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.725+0000 7fba2f7fe700 1 -- 172.21.15.98:0/2256299435 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fba400515e0 con 0x7fba480d0250 2023-10-06T05:08:00.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.904+0000 7fba4cb40700 1 -- 172.21.15.98:0/2256299435 --> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fba48159810 con 0x7fba34066f30 2023-10-06T05:08:00.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.912+0000 7fba2f7fe700 1 -- 172.21.15.98:0/2256299435 <== mgr.4104 v2:172.21.15.98:6832/90397 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+311205 (secure 0 0 0) 0x7fba48159810 con 0x7fba34066f30 2023-10-06T05:08:00.918 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:08:00.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.916+0000 7fba4cb40700 1 -- 172.21.15.98:0/2256299435 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fba34066f30 msgr2=0x7fba340693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:00.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.916+0000 7fba4cb40700 1 --2- 172.21.15.98:0/2256299435 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fba34066f30 0x7fba340693e0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fba38009fd0 tx=0x7fba3800b040 comp rx=0 tx=0).stop 2023-10-06T05:08:00.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.916+0000 7fba4cb40700 1 -- 172.21.15.98:0/2256299435 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba480d0250 msgr2=0x7fba480d06a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:00.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.916+0000 7fba4cb40700 1 --2- 172.21.15.98:0/2256299435 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba480d0250 0x7fba480d06a0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fba48069150 tx=0x7fba40000f00 comp rx=0 tx=0).stop 2023-10-06T05:08:00.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.916+0000 7fba4cb40700 1 -- 172.21.15.98:0/2256299435 shutdown_connections 2023-10-06T05:08:00.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.916+0000 7fba4cb40700 1 --2- 172.21.15.98:0/2256299435 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fba34066f30 0x7fba340693e0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:00.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.916+0000 7fba4cb40700 1 --2- 172.21.15.98:0/2256299435 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba480d0250 0x7fba480d06a0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:00.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.916+0000 7fba4cb40700 1 --2- 172.21.15.98:0/2256299435 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fba30004180 0x7fba48155950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:00.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.916+0000 7fba4cb40700 1 --2- 172.21.15.98:0/2256299435 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fba480c7f20 0x7fba4815a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:00.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.916+0000 7fba4cb40700 1 -- 172.21.15.98:0/2256299435 >> 172.21.15.98:0/2256299435 conn(0x7fba4806ece0 msgr2=0x7fba480cddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:00.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.916+0000 7fba4cb40700 1 -- 172.21.15.98:0/2256299435 shutdown_connections 2023-10-06T05:08:00.921 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:00.916+0000 7fba4cb40700 1 -- 172.21.15.98:0/2256299435 wait complete. 2023-10-06T05:08:00.921 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2023-10-06T05:08:00.930 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":75,"stamp":"2023-10-06T05:08:00.532870+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":168,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2383332,"kb_used_data":5028,"kb_used_omap":0,"kb_used_meta":2378240,"kb_avail":752591388,"statfs":{"total":773094113280,"available":770653581312,"internally_reserved":0,"allocated":5148672,"data_stored":2913553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2435317760},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000953"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":8,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.497305+0000","last_change":"2023-10-06T05:07:56.497305+0000","last_active":"2023-10-06T05:07:56.497305+0000","last_peered":"2023-10-06T05:07:56.497305+0000","last_clean":"2023-10-06T05:07:56.497305+0000","last_became_active":"2023-10-06T05:07:56.496979+0000","last_became_peered":"2023-10-06T05:07:56.496979+0000","last_unstale":"2023-10-06T05:07:56.497305+0000","last_undegraded":"2023-10-06T05:07:56.497305+0000","last_fullsized":"2023-10-06T05:07:56.497305+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:05:12.786696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.505779+0000","last_change":"2023-10-06T05:07:56.496318+0000","last_active":"2023-10-06T05:07:56.505779+0000","last_peered":"2023-10-06T05:07:56.505779+0000","last_clean":"2023-10-06T05:07:56.505779+0000","last_became_active":"2023-10-06T05:07:56.495491+0000","last_became_peered":"2023-10-06T05:07:56.495491+0000","last_unstale":"2023-10-06T05:07:56.505779+0000","last_undegraded":"2023-10-06T05:07:56.505779+0000","last_fullsized":"2023-10-06T05:07:56.505779+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:12:32.521554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.505213+0000","last_change":"2023-10-06T05:07:56.501549+0000","last_active":"2023-10-06T05:07:56.505213+0000","last_peered":"2023-10-06T05:07:56.505213+0000","last_clean":"2023-10-06T05:07:56.505213+0000","last_became_active":"2023-10-06T05:07:56.501277+0000","last_became_peered":"2023-10-06T05:07:56.501277+0000","last_unstale":"2023-10-06T05:07:56.505213+0000","last_undegraded":"2023-10-06T05:07:56.505213+0000","last_fullsized":"2023-10-06T05:07:56.505213+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:39:55.924395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":8,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.506566+0000","last_change":"2023-10-06T05:07:56.506566+0000","last_active":"2023-10-06T05:07:56.506566+0000","last_peered":"2023-10-06T05:07:56.506566+0000","last_clean":"2023-10-06T05:07:56.506566+0000","last_became_active":"2023-10-06T05:07:56.505105+0000","last_became_peered":"2023-10-06T05:07:56.505105+0000","last_unstale":"2023-10-06T05:07:56.506566+0000","last_undegraded":"2023-10-06T05:07:56.506566+0000","last_fullsized":"2023-10-06T05:07:56.506566+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:51:57.567130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.505737+0000","last_change":"2023-10-06T05:07:56.496507+0000","last_active":"2023-10-06T05:07:56.505737+0000","last_peered":"2023-10-06T05:07:56.505737+0000","last_clean":"2023-10-06T05:07:56.505737+0000","last_became_active":"2023-10-06T05:07:56.496216+0000","last_became_peered":"2023-10-06T05:07:56.496216+0000","last_unstale":"2023-10-06T05:07:56.505737+0000","last_undegraded":"2023-10-06T05:07:56.505737+0000","last_fullsized":"2023-10-06T05:07:56.505737+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:50:46.462343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.499993+0000","last_change":"2023-10-06T05:07:56.498796+0000","last_active":"2023-10-06T05:07:56.499993+0000","last_peered":"2023-10-06T05:07:56.499993+0000","last_clean":"2023-10-06T05:07:56.499993+0000","last_became_active":"2023-10-06T05:07:56.498537+0000","last_became_peered":"2023-10-06T05:07:56.498537+0000","last_unstale":"2023-10-06T05:07:56.499993+0000","last_undegraded":"2023-10-06T05:07:56.499993+0000","last_fullsized":"2023-10-06T05:07:56.499993+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:01:24.910048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":8,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.507167+0000","last_change":"2023-10-06T05:07:56.507167+0000","last_active":"2023-10-06T05:07:56.507167+0000","last_peered":"2023-10-06T05:07:56.507167+0000","last_clean":"2023-10-06T05:07:56.507167+0000","last_became_active":"2023-10-06T05:07:56.506895+0000","last_became_peered":"2023-10-06T05:07:56.506895+0000","last_unstale":"2023-10-06T05:07:56.507167+0000","last_undegraded":"2023-10-06T05:07:56.507167+0000","last_fullsized":"2023-10-06T05:07:56.507167+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T16:18:50.165832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":8,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.504667+0000","last_change":"2023-10-06T05:07:56.504667+0000","last_active":"2023-10-06T05:07:56.504667+0000","last_peered":"2023-10-06T05:07:56.504667+0000","last_clean":"2023-10-06T05:07:56.504667+0000","last_became_active":"2023-10-06T05:07:56.504341+0000","last_became_peered":"2023-10-06T05:07:56.504341+0000","last_unstale":"2023-10-06T05:07:56.504667+0000","last_undegraded":"2023-10-06T05:07:56.504667+0000","last_fullsized":"2023-10-06T05:07:56.504667+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:05:44.930056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":8,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.507152+0000","last_change":"2023-10-06T05:07:56.507152+0000","last_active":"2023-10-06T05:07:56.507152+0000","last_peered":"2023-10-06T05:07:56.507152+0000","last_clean":"2023-10-06T05:07:56.507152+0000","last_became_active":"2023-10-06T05:07:56.506920+0000","last_became_peered":"2023-10-06T05:07:56.506920+0000","last_unstale":"2023-10-06T05:07:56.507152+0000","last_undegraded":"2023-10-06T05:07:56.507152+0000","last_fullsized":"2023-10-06T05:07:56.507152+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:23:42.009212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":8,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.504224+0000","last_change":"2023-10-06T05:07:56.504224+0000","last_active":"2023-10-06T05:07:56.504224+0000","last_peered":"2023-10-06T05:07:56.504224+0000","last_clean":"2023-10-06T05:07:56.504224+0000","last_became_active":"2023-10-06T05:07:56.503937+0000","last_became_peered":"2023-10-06T05:07:56.503937+0000","last_unstale":"2023-10-06T05:07:56.504224+0000","last_undegraded":"2023-10-06T05:07:56.504224+0000","last_fullsized":"2023-10-06T05:07:56.504224+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:25:49.962193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":8,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.510436+0000","last_change":"2023-10-06T05:07:56.510436+0000","last_active":"2023-10-06T05:07:56.510436+0000","last_peered":"2023-10-06T05:07:56.510436+0000","last_clean":"2023-10-06T05:07:56.510436+0000","last_became_active":"2023-10-06T05:07:56.510191+0000","last_became_peered":"2023-10-06T05:07:56.510191+0000","last_unstale":"2023-10-06T05:07:56.510436+0000","last_undegraded":"2023-10-06T05:07:56.510436+0000","last_fullsized":"2023-10-06T05:07:56.510436+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:46:33.207358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.512781+0000","last_change":"2023-10-06T05:07:56.511385+0000","last_active":"2023-10-06T05:07:56.512781+0000","last_peered":"2023-10-06T05:07:56.512781+0000","last_clean":"2023-10-06T05:07:56.512781+0000","last_became_active":"2023-10-06T05:07:56.511074+0000","last_became_peered":"2023-10-06T05:07:56.511074+0000","last_unstale":"2023-10-06T05:07:56.512781+0000","last_undegraded":"2023-10-06T05:07:56.512781+0000","last_fullsized":"2023-10-06T05:07:56.512781+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:02:41.335573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.517892+0000","last_change":"2023-10-06T05:07:56.516120+0000","last_active":"2023-10-06T05:07:56.517892+0000","last_peered":"2023-10-06T05:07:56.517892+0000","last_clean":"2023-10-06T05:07:56.517892+0000","last_became_active":"2023-10-06T05:07:56.515873+0000","last_became_peered":"2023-10-06T05:07:56.515873+0000","last_unstale":"2023-10-06T05:07:56.517892+0000","last_undegraded":"2023-10-06T05:07:56.517892+0000","last_fullsized":"2023-10-06T05:07:56.517892+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:02:33.928047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":8,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.508929+0000","last_change":"2023-10-06T05:07:56.508929+0000","last_active":"2023-10-06T05:07:56.508929+0000","last_peered":"2023-10-06T05:07:56.508929+0000","last_clean":"2023-10-06T05:07:56.508929+0000","last_became_active":"2023-10-06T05:07:56.508696+0000","last_became_peered":"2023-10-06T05:07:56.508696+0000","last_unstale":"2023-10-06T05:07:56.508929+0000","last_undegraded":"2023-10-06T05:07:56.508929+0000","last_fullsized":"2023-10-06T05:07:56.508929+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:30:08.339655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.498009+0000","last_change":"2023-10-06T05:07:56.497209+0000","last_active":"2023-10-06T05:07:56.498009+0000","last_peered":"2023-10-06T05:07:56.498009+0000","last_clean":"2023-10-06T05:07:56.498009+0000","last_became_active":"2023-10-06T05:07:56.496774+0000","last_became_peered":"2023-10-06T05:07:56.496774+0000","last_unstale":"2023-10-06T05:07:56.498009+0000","last_undegraded":"2023-10-06T05:07:56.498009+0000","last_fullsized":"2023-10-06T05:07:56.498009+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:11:44.842024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.505863+0000","last_change":"2023-10-06T05:07:56.500829+0000","last_active":"2023-10-06T05:07:56.505863+0000","last_peered":"2023-10-06T05:07:56.505863+0000","last_clean":"2023-10-06T05:07:56.505863+0000","last_became_active":"2023-10-06T05:07:56.499161+0000","last_became_peered":"2023-10-06T05:07:56.499161+0000","last_unstale":"2023-10-06T05:07:56.505863+0000","last_undegraded":"2023-10-06T05:07:56.505863+0000","last_fullsized":"2023-10-06T05:07:56.505863+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:02:33.928047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.502732+0000","last_change":"2023-10-06T05:07:56.493854+0000","last_active":"2023-10-06T05:07:56.502732+0000","last_peered":"2023-10-06T05:07:56.502732+0000","last_clean":"2023-10-06T05:07:56.502732+0000","last_became_active":"2023-10-06T05:07:56.493573+0000","last_became_peered":"2023-10-06T05:07:56.493573+0000","last_unstale":"2023-10-06T05:07:56.502732+0000","last_undegraded":"2023-10-06T05:07:56.502732+0000","last_fullsized":"2023-10-06T05:07:56.502732+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:51:44.109076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.504641+0000","last_change":"2023-10-06T05:07:56.500173+0000","last_active":"2023-10-06T05:07:56.504641+0000","last_peered":"2023-10-06T05:07:56.504641+0000","last_clean":"2023-10-06T05:07:56.504641+0000","last_became_active":"2023-10-06T05:07:56.499919+0000","last_became_peered":"2023-10-06T05:07:56.499919+0000","last_unstale":"2023-10-06T05:07:56.504641+0000","last_undegraded":"2023-10-06T05:07:56.504641+0000","last_fullsized":"2023-10-06T05:07:56.504641+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:03:00.432048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.505792+0000","last_change":"2023-10-06T05:07:56.500721+0000","last_active":"2023-10-06T05:07:56.505792+0000","last_peered":"2023-10-06T05:07:56.505792+0000","last_clean":"2023-10-06T05:07:56.505792+0000","last_became_active":"2023-10-06T05:07:56.500502+0000","last_became_peered":"2023-10-06T05:07:56.500502+0000","last_unstale":"2023-10-06T05:07:56.505792+0000","last_undegraded":"2023-10-06T05:07:56.505792+0000","last_fullsized":"2023-10-06T05:07:56.505792+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:32:31.940742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.512740+0000","last_change":"2023-10-06T05:07:56.507184+0000","last_active":"2023-10-06T05:07:56.512740+0000","last_peered":"2023-10-06T05:07:56.512740+0000","last_clean":"2023-10-06T05:07:56.512740+0000","last_became_active":"2023-10-06T05:07:56.502222+0000","last_became_peered":"2023-10-06T05:07:56.502222+0000","last_unstale":"2023-10-06T05:07:56.512740+0000","last_undegraded":"2023-10-06T05:07:56.512740+0000","last_fullsized":"2023-10-06T05:07:56.512740+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:32:31.940742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.515025+0000","last_change":"2023-10-06T05:07:56.506115+0000","last_active":"2023-10-06T05:07:56.515025+0000","last_peered":"2023-10-06T05:07:56.515025+0000","last_clean":"2023-10-06T05:07:56.515025+0000","last_became_active":"2023-10-06T05:07:56.505853+0000","last_became_peered":"2023-10-06T05:07:56.505853+0000","last_unstale":"2023-10-06T05:07:56.515025+0000","last_undegraded":"2023-10-06T05:07:56.515025+0000","last_fullsized":"2023-10-06T05:07:56.515025+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:08:06.961933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":8,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.504863+0000","last_change":"2023-10-06T05:07:56.504863+0000","last_active":"2023-10-06T05:07:56.504863+0000","last_peered":"2023-10-06T05:07:56.504863+0000","last_clean":"2023-10-06T05:07:56.504863+0000","last_became_active":"2023-10-06T05:07:56.504612+0000","last_became_peered":"2023-10-06T05:07:56.504612+0000","last_unstale":"2023-10-06T05:07:56.504863+0000","last_undegraded":"2023-10-06T05:07:56.504863+0000","last_fullsized":"2023-10-06T05:07:56.504863+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:30:08.409840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.505843+0000","last_change":"2023-10-06T05:07:56.501498+0000","last_active":"2023-10-06T05:07:56.505843+0000","last_peered":"2023-10-06T05:07:56.505843+0000","last_clean":"2023-10-06T05:07:56.505843+0000","last_became_active":"2023-10-06T05:07:56.501215+0000","last_became_peered":"2023-10-06T05:07:56.501215+0000","last_unstale":"2023-10-06T05:07:56.505843+0000","last_undegraded":"2023-10-06T05:07:56.505843+0000","last_fullsized":"2023-10-06T05:07:56.505843+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:09:26.304992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.505878+0000","last_change":"2023-10-06T05:07:56.501317+0000","last_active":"2023-10-06T05:07:56.505878+0000","last_peered":"2023-10-06T05:07:56.505878+0000","last_clean":"2023-10-06T05:07:56.505878+0000","last_became_active":"2023-10-06T05:07:56.501032+0000","last_became_peered":"2023-10-06T05:07:56.501032+0000","last_unstale":"2023-10-06T05:07:56.505878+0000","last_undegraded":"2023-10-06T05:07:56.505878+0000","last_fullsized":"2023-10-06T05:07:56.505878+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:45:59.605630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.510618+0000","last_change":"2023-10-06T05:07:56.507300+0000","last_active":"2023-10-06T05:07:56.510618+0000","last_peered":"2023-10-06T05:07:56.510618+0000","last_clean":"2023-10-06T05:07:56.510618+0000","last_became_active":"2023-10-06T05:07:56.507001+0000","last_became_peered":"2023-10-06T05:07:56.507001+0000","last_unstale":"2023-10-06T05:07:56.510618+0000","last_undegraded":"2023-10-06T05:07:56.510618+0000","last_fullsized":"2023-10-06T05:07:56.510618+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T16:39:01.958194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.517713+0000","last_change":"2023-10-06T05:07:56.515811+0000","last_active":"2023-10-06T05:07:56.517713+0000","last_peered":"2023-10-06T05:07:56.517713+0000","last_clean":"2023-10-06T05:07:56.517713+0000","last_became_active":"2023-10-06T05:07:56.515574+0000","last_became_peered":"2023-10-06T05:07:56.515574+0000","last_unstale":"2023-10-06T05:07:56.517713+0000","last_undegraded":"2023-10-06T05:07:56.517713+0000","last_fullsized":"2023-10-06T05:07:56.517713+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:39:38.455191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.503017+0000","last_change":"2023-10-06T05:07:56.494169+0000","last_active":"2023-10-06T05:07:56.503017+0000","last_peered":"2023-10-06T05:07:56.503017+0000","last_clean":"2023-10-06T05:07:56.503017+0000","last_became_active":"2023-10-06T05:07:56.493668+0000","last_became_peered":"2023-10-06T05:07:56.493668+0000","last_unstale":"2023-10-06T05:07:56.503017+0000","last_undegraded":"2023-10-06T05:07:56.503017+0000","last_fullsized":"2023-10-06T05:07:56.503017+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:30:14.123623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.506666+0000","last_change":"2023-10-06T05:07:56.503399+0000","last_active":"2023-10-06T05:07:56.506666+0000","last_peered":"2023-10-06T05:07:56.506666+0000","last_clean":"2023-10-06T05:07:56.506666+0000","last_became_active":"2023-10-06T05:07:56.503118+0000","last_became_peered":"2023-10-06T05:07:56.503118+0000","last_unstale":"2023-10-06T05:07:56.506666+0000","last_undegraded":"2023-10-06T05:07:56.506666+0000","last_fullsized":"2023-10-06T05:07:56.506666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:07:47.377546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.510648+0000","last_change":"2023-10-06T05:07:56.507856+0000","last_active":"2023-10-06T05:07:56.510648+0000","last_peered":"2023-10-06T05:07:56.510648+0000","last_clean":"2023-10-06T05:07:56.510648+0000","last_became_active":"2023-10-06T05:07:56.507585+0000","last_became_peered":"2023-10-06T05:07:56.507585+0000","last_unstale":"2023-10-06T05:07:56.510648+0000","last_undegraded":"2023-10-06T05:07:56.510648+0000","last_fullsized":"2023-10-06T05:07:56.510648+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:10:26.636623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.504516+0000","last_change":"2023-10-06T05:07:56.497371+0000","last_active":"2023-10-06T05:07:56.504516+0000","last_peered":"2023-10-06T05:07:56.504516+0000","last_clean":"2023-10-06T05:07:56.504516+0000","last_became_active":"2023-10-06T05:07:56.497098+0000","last_became_peered":"2023-10-06T05:07:56.497098+0000","last_unstale":"2023-10-06T05:07:56.504516+0000","last_undegraded":"2023-10-06T05:07:56.504516+0000","last_fullsized":"2023-10-06T05:07:56.504516+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:51:32.705673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.497992+0000","last_change":"2023-10-06T05:07:56.497088+0000","last_active":"2023-10-06T05:07:56.497992+0000","last_peered":"2023-10-06T05:07:56.497992+0000","last_clean":"2023-10-06T05:07:56.497992+0000","last_became_active":"2023-10-06T05:07:56.496480+0000","last_became_peered":"2023-10-06T05:07:56.496480+0000","last_unstale":"2023-10-06T05:07:56.497992+0000","last_undegraded":"2023-10-06T05:07:56.497992+0000","last_fullsized":"2023-10-06T05:07:56.497992+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:16:06.816623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.513651+0000","last_change":"2023-10-06T05:07:56.511574+0000","last_active":"2023-10-06T05:07:56.513651+0000","last_peered":"2023-10-06T05:07:56.513651+0000","last_clean":"2023-10-06T05:07:56.513651+0000","last_became_active":"2023-10-06T05:07:56.511303+0000","last_became_peered":"2023-10-06T05:07:56.511303+0000","last_unstale":"2023-10-06T05:07:56.513651+0000","last_undegraded":"2023-10-06T05:07:56.513651+0000","last_fullsized":"2023-10-06T05:07:56.513651+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T16:39:01.958194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":8,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.503991+0000","last_change":"2023-10-06T05:07:56.503991+0000","last_active":"2023-10-06T05:07:56.503991+0000","last_peered":"2023-10-06T05:07:56.503991+0000","last_clean":"2023-10-06T05:07:56.503991+0000","last_became_active":"2023-10-06T05:07:56.503397+0000","last_became_peered":"2023-10-06T05:07:56.503397+0000","last_unstale":"2023-10-06T05:07:56.503991+0000","last_undegraded":"2023-10-06T05:07:56.503991+0000","last_fullsized":"2023-10-06T05:07:56.503991+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:25:34.480685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"42'4","reported_seq":14,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.511666+0000","last_change":"2023-10-06T05:07:55.507746+0000","last_active":"2023-10-06T05:07:56.511666+0000","last_peered":"2023-10-06T05:07:56.511666+0000","last_clean":"2023-10-06T05:07:56.511666+0000","last_became_active":"2023-10-06T05:07:55.507486+0000","last_became_peered":"2023-10-06T05:07:55.507486+0000","last_unstale":"2023-10-06T05:07:56.511666+0000","last_undegraded":"2023-10-06T05:07:56.511666+0000","last_fullsized":"2023-10-06T05:07:56.511666+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_clean_scrub_stamp":"2023-10-06T05:07:54.474583+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:27:33.586746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.505881+0000","last_change":"2023-10-06T05:07:56.500309+0000","last_active":"2023-10-06T05:07:56.505881+0000","last_peered":"2023-10-06T05:07:56.505881+0000","last_clean":"2023-10-06T05:07:56.505881+0000","last_became_active":"2023-10-06T05:07:56.500066+0000","last_became_peered":"2023-10-06T05:07:56.500066+0000","last_unstale":"2023-10-06T05:07:56.505881+0000","last_undegraded":"2023-10-06T05:07:56.505881+0000","last_fullsized":"2023-10-06T05:07:56.505881+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:50:27.863154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"42'1","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.506186+0000","last_change":"2023-10-06T05:07:55.503820+0000","last_active":"2023-10-06T05:07:56.506186+0000","last_peered":"2023-10-06T05:07:56.506186+0000","last_clean":"2023-10-06T05:07:56.506186+0000","last_became_active":"2023-10-06T05:07:55.503546+0000","last_became_peered":"2023-10-06T05:07:55.503546+0000","last_unstale":"2023-10-06T05:07:56.506186+0000","last_undegraded":"2023-10-06T05:07:56.506186+0000","last_fullsized":"2023-10-06T05:07:56.506186+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_clean_scrub_stamp":"2023-10-06T05:07:54.474583+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:16:05.814512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.505572+0000","last_change":"2023-10-06T05:07:56.497520+0000","last_active":"2023-10-06T05:07:56.505572+0000","last_peered":"2023-10-06T05:07:56.505572+0000","last_clean":"2023-10-06T05:07:56.505572+0000","last_became_active":"2023-10-06T05:07:56.497239+0000","last_became_peered":"2023-10-06T05:07:56.497239+0000","last_unstale":"2023-10-06T05:07:56.505572+0000","last_undegraded":"2023-10-06T05:07:56.505572+0000","last_fullsized":"2023-10-06T05:07:56.505572+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:13:46.456823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"42'2","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.510676+0000","last_change":"2023-10-06T05:07:55.497994+0000","last_active":"2023-10-06T05:07:56.510676+0000","last_peered":"2023-10-06T05:07:56.510676+0000","last_clean":"2023-10-06T05:07:56.510676+0000","last_became_active":"2023-10-06T05:07:55.497142+0000","last_became_peered":"2023-10-06T05:07:55.497142+0000","last_unstale":"2023-10-06T05:07:56.510676+0000","last_undegraded":"2023-10-06T05:07:56.510676+0000","last_fullsized":"2023-10-06T05:07:56.510676+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_clean_scrub_stamp":"2023-10-06T05:07:54.474583+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:57:24.408712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":8,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.503243+0000","last_change":"2023-10-06T05:07:56.503243+0000","last_active":"2023-10-06T05:07:56.503243+0000","last_peered":"2023-10-06T05:07:56.503243+0000","last_clean":"2023-10-06T05:07:56.503243+0000","last_became_active":"2023-10-06T05:07:56.502982+0000","last_became_peered":"2023-10-06T05:07:56.502982+0000","last_unstale":"2023-10-06T05:07:56.503243+0000","last_undegraded":"2023-10-06T05:07:56.503243+0000","last_fullsized":"2023-10-06T05:07:56.503243+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:32:18.252043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"42'2","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.511077+0000","last_change":"2023-10-06T05:07:55.497209+0000","last_active":"2023-10-06T05:07:56.511077+0000","last_peered":"2023-10-06T05:07:56.511077+0000","last_clean":"2023-10-06T05:07:56.511077+0000","last_became_active":"2023-10-06T05:07:55.496946+0000","last_became_peered":"2023-10-06T05:07:55.496946+0000","last_unstale":"2023-10-06T05:07:56.511077+0000","last_undegraded":"2023-10-06T05:07:56.511077+0000","last_fullsized":"2023-10-06T05:07:56.511077+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_clean_scrub_stamp":"2023-10-06T05:07:54.474583+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:12:44.633219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.505179+0000","last_change":"2023-10-06T05:07:56.502083+0000","last_active":"2023-10-06T05:07:56.505179+0000","last_peered":"2023-10-06T05:07:56.505179+0000","last_clean":"2023-10-06T05:07:56.505179+0000","last_became_active":"2023-10-06T05:07:56.501590+0000","last_became_peered":"2023-10-06T05:07:56.501590+0000","last_unstale":"2023-10-06T05:07:56.505179+0000","last_undegraded":"2023-10-06T05:07:56.505179+0000","last_fullsized":"2023-10-06T05:07:56.505179+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:00:00.248723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"42'1","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.514261+0000","last_change":"2023-10-06T05:07:55.504304+0000","last_active":"2023-10-06T05:07:56.514261+0000","last_peered":"2023-10-06T05:07:56.514261+0000","last_clean":"2023-10-06T05:07:56.514261+0000","last_became_active":"2023-10-06T05:07:55.504020+0000","last_became_peered":"2023-10-06T05:07:55.504020+0000","last_unstale":"2023-10-06T05:07:56.514261+0000","last_undegraded":"2023-10-06T05:07:56.514261+0000","last_fullsized":"2023-10-06T05:07:56.514261+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_clean_scrub_stamp":"2023-10-06T05:07:54.474583+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:12:31.519443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":8,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.501017+0000","last_change":"2023-10-06T05:07:56.501017+0000","last_active":"2023-10-06T05:07:56.501017+0000","last_peered":"2023-10-06T05:07:56.501017+0000","last_clean":"2023-10-06T05:07:56.501017+0000","last_became_active":"2023-10-06T05:07:56.500761+0000","last_became_peered":"2023-10-06T05:07:56.500761+0000","last_unstale":"2023-10-06T05:07:56.501017+0000","last_undegraded":"2023-10-06T05:07:56.501017+0000","last_fullsized":"2023-10-06T05:07:56.501017+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:00:23.210220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"42'1","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.506054+0000","last_change":"2023-10-06T05:07:55.494922+0000","last_active":"2023-10-06T05:07:56.506054+0000","last_peered":"2023-10-06T05:07:56.506054+0000","last_clean":"2023-10-06T05:07:56.506054+0000","last_became_active":"2023-10-06T05:07:55.494394+0000","last_became_peered":"2023-10-06T05:07:55.494394+0000","last_unstale":"2023-10-06T05:07:56.506054+0000","last_undegraded":"2023-10-06T05:07:56.506054+0000","last_fullsized":"2023-10-06T05:07:56.506054+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_clean_scrub_stamp":"2023-10-06T05:07:54.474583+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:33:33.428032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":8,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.504500+0000","last_change":"2023-10-06T05:07:56.504500+0000","last_active":"2023-10-06T05:07:56.504500+0000","last_peered":"2023-10-06T05:07:56.504500+0000","last_clean":"2023-10-06T05:07:56.504500+0000","last_became_active":"2023-10-06T05:07:56.504082+0000","last_became_peered":"2023-10-06T05:07:56.504082+0000","last_unstale":"2023-10-06T05:07:56.504500+0000","last_undegraded":"2023-10-06T05:07:56.504500+0000","last_fullsized":"2023-10-06T05:07:56.504500+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:30:03.127010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.489306+0000","last_change":"2023-10-06T05:07:55.495889+0000","last_active":"2023-10-06T05:07:56.489306+0000","last_peered":"2023-10-06T05:07:56.489306+0000","last_clean":"2023-10-06T05:07:56.489306+0000","last_became_active":"2023-10-06T05:07:55.495612+0000","last_became_peered":"2023-10-06T05:07:55.495612+0000","last_unstale":"2023-10-06T05:07:56.489306+0000","last_undegraded":"2023-10-06T05:07:56.489306+0000","last_fullsized":"2023-10-06T05:07:56.489306+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_clean_scrub_stamp":"2023-10-06T05:07:54.474583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:59:18.760657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.505824+0000","last_change":"2023-10-06T05:07:56.499141+0000","last_active":"2023-10-06T05:07:56.505824+0000","last_peered":"2023-10-06T05:07:56.505824+0000","last_clean":"2023-10-06T05:07:56.505824+0000","last_became_active":"2023-10-06T05:07:56.498804+0000","last_became_peered":"2023-10-06T05:07:56.498804+0000","last_unstale":"2023-10-06T05:07:56.505824+0000","last_undegraded":"2023-10-06T05:07:56.505824+0000","last_fullsized":"2023-10-06T05:07:56.505824+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:24:45.237400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T05:07:55.507457+0000","last_change":"2023-10-06T05:07:55.502928+0000","last_active":"2023-10-06T05:07:55.507457+0000","last_peered":"2023-10-06T05:07:55.507457+0000","last_clean":"2023-10-06T05:07:55.507457+0000","last_became_active":"2023-10-06T05:07:55.502688+0000","last_became_peered":"2023-10-06T05:07:55.502688+0000","last_unstale":"2023-10-06T05:07:55.507457+0000","last_undegraded":"2023-10-06T05:07:55.507457+0000","last_fullsized":"2023-10-06T05:07:55.507457+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_clean_scrub_stamp":"2023-10-06T05:07:54.474583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:47:49.442467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.504622+0000","last_change":"2023-10-06T05:07:56.500092+0000","last_active":"2023-10-06T05:07:56.504622+0000","last_peered":"2023-10-06T05:07:56.504622+0000","last_clean":"2023-10-06T05:07:56.504622+0000","last_became_active":"2023-10-06T05:07:56.499592+0000","last_became_peered":"2023-10-06T05:07:56.499592+0000","last_unstale":"2023-10-06T05:07:56.504622+0000","last_undegraded":"2023-10-06T05:07:56.504622+0000","last_fullsized":"2023-10-06T05:07:56.504622+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:00:36.630586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.486588+0000","last_change":"2023-10-06T05:07:55.499904+0000","last_active":"2023-10-06T05:07:56.486588+0000","last_peered":"2023-10-06T05:07:56.486588+0000","last_clean":"2023-10-06T05:07:56.486588+0000","last_became_active":"2023-10-06T05:07:55.499588+0000","last_became_peered":"2023-10-06T05:07:55.499588+0000","last_unstale":"2023-10-06T05:07:56.486588+0000","last_undegraded":"2023-10-06T05:07:56.486588+0000","last_fullsized":"2023-10-06T05:07:56.486588+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_clean_scrub_stamp":"2023-10-06T05:07:54.474583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:17:05.388581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":8,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.497841+0000","last_change":"2023-10-06T05:07:56.497841+0000","last_active":"2023-10-06T05:07:56.497841+0000","last_peered":"2023-10-06T05:07:56.497841+0000","last_clean":"2023-10-06T05:07:56.497841+0000","last_became_active":"2023-10-06T05:07:56.497546+0000","last_became_peered":"2023-10-06T05:07:56.497546+0000","last_unstale":"2023-10-06T05:07:56.497841+0000","last_undegraded":"2023-10-06T05:07:56.497841+0000","last_fullsized":"2023-10-06T05:07:56.497841+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:44:40.634049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T05:07:55.502968+0000","last_change":"2023-10-06T05:07:55.499922+0000","last_active":"2023-10-06T05:07:55.502968+0000","last_peered":"2023-10-06T05:07:55.502968+0000","last_clean":"2023-10-06T05:07:55.502968+0000","last_became_active":"2023-10-06T05:07:55.499650+0000","last_became_peered":"2023-10-06T05:07:55.499650+0000","last_unstale":"2023-10-06T05:07:55.502968+0000","last_undegraded":"2023-10-06T05:07:55.502968+0000","last_fullsized":"2023-10-06T05:07:55.502968+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_clean_scrub_stamp":"2023-10-06T05:07:54.474583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:32:17.249933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.516227+0000","last_change":"2023-10-06T05:07:56.514719+0000","last_active":"2023-10-06T05:07:56.516227+0000","last_peered":"2023-10-06T05:07:56.516227+0000","last_clean":"2023-10-06T05:07:56.516227+0000","last_became_active":"2023-10-06T05:07:56.514340+0000","last_became_peered":"2023-10-06T05:07:56.514340+0000","last_unstale":"2023-10-06T05:07:56.516227+0000","last_undegraded":"2023-10-06T05:07:56.516227+0000","last_fullsized":"2023-10-06T05:07:56.516227+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:16:42.675860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"42'3","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.511704+0000","last_change":"2023-10-06T05:07:55.507730+0000","last_active":"2023-10-06T05:07:56.511704+0000","last_peered":"2023-10-06T05:07:56.511704+0000","last_clean":"2023-10-06T05:07:56.511704+0000","last_became_active":"2023-10-06T05:07:55.507440+0000","last_became_peered":"2023-10-06T05:07:55.507440+0000","last_unstale":"2023-10-06T05:07:56.511704+0000","last_undegraded":"2023-10-06T05:07:56.511704+0000","last_fullsized":"2023-10-06T05:07:56.511704+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_clean_scrub_stamp":"2023-10-06T05:07:54.474583+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:09:40.893258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":8,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.508020+0000","last_change":"2023-10-06T05:07:56.508020+0000","last_active":"2023-10-06T05:07:56.508020+0000","last_peered":"2023-10-06T05:07:56.508020+0000","last_clean":"2023-10-06T05:07:56.508020+0000","last_became_active":"2023-10-06T05:07:56.507765+0000","last_became_peered":"2023-10-06T05:07:56.507765+0000","last_unstale":"2023-10-06T05:07:56.508020+0000","last_undegraded":"2023-10-06T05:07:56.508020+0000","last_fullsized":"2023-10-06T05:07:56.508020+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:57:30.893504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"42'6","reported_seq":16,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.517939+0000","last_change":"2023-10-06T05:07:55.501591+0000","last_active":"2023-10-06T05:07:56.517939+0000","last_peered":"2023-10-06T05:07:56.517939+0000","last_clean":"2023-10-06T05:07:56.517939+0000","last_became_active":"2023-10-06T05:07:55.501265+0000","last_became_peered":"2023-10-06T05:07:55.501265+0000","last_unstale":"2023-10-06T05:07:56.517939+0000","last_undegraded":"2023-10-06T05:07:56.517939+0000","last_fullsized":"2023-10-06T05:07:56.517939+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_clean_scrub_stamp":"2023-10-06T05:07:54.474583+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T16:01:12.700532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.505592+0000","last_change":"2023-10-06T05:07:56.499987+0000","last_active":"2023-10-06T05:07:56.505592+0000","last_peered":"2023-10-06T05:07:56.505592+0000","last_clean":"2023-10-06T05:07:56.505592+0000","last_became_active":"2023-10-06T05:07:56.499700+0000","last_became_peered":"2023-10-06T05:07:56.499700+0000","last_unstale":"2023-10-06T05:07:56.505592+0000","last_undegraded":"2023-10-06T05:07:56.505592+0000","last_fullsized":"2023-10-06T05:07:56.505592+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:39:38.455191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"42'2","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.514948+0000","last_change":"2023-10-06T05:07:55.499932+0000","last_active":"2023-10-06T05:07:56.514948+0000","last_peered":"2023-10-06T05:07:56.514948+0000","last_clean":"2023-10-06T05:07:56.514948+0000","last_became_active":"2023-10-06T05:07:55.499650+0000","last_became_peered":"2023-10-06T05:07:55.499650+0000","last_unstale":"2023-10-06T05:07:56.514948+0000","last_undegraded":"2023-10-06T05:07:56.514948+0000","last_fullsized":"2023-10-06T05:07:56.514948+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_clean_scrub_stamp":"2023-10-06T05:07:54.474583+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:40:40.783028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":8,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.499040+0000","last_change":"2023-10-06T05:07:56.499040+0000","last_active":"2023-10-06T05:07:56.499040+0000","last_peered":"2023-10-06T05:07:56.499040+0000","last_clean":"2023-10-06T05:07:56.499040+0000","last_became_active":"2023-10-06T05:07:56.498803+0000","last_became_peered":"2023-10-06T05:07:56.498803+0000","last_unstale":"2023-10-06T05:07:56.499040+0000","last_undegraded":"2023-10-06T05:07:56.499040+0000","last_fullsized":"2023-10-06T05:07:56.499040+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:50:31.905666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T05:07:55.504922+0000","last_change":"2023-10-06T05:07:55.502323+0000","last_active":"2023-10-06T05:07:55.504922+0000","last_peered":"2023-10-06T05:07:55.504922+0000","last_clean":"2023-10-06T05:07:55.504922+0000","last_became_active":"2023-10-06T05:07:55.502126+0000","last_became_peered":"2023-10-06T05:07:55.502126+0000","last_unstale":"2023-10-06T05:07:55.504922+0000","last_undegraded":"2023-10-06T05:07:55.504922+0000","last_fullsized":"2023-10-06T05:07:55.504922+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_clean_scrub_stamp":"2023-10-06T05:07:54.474583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:25:33.478575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.515000+0000","last_change":"2023-10-06T05:07:56.502664+0000","last_active":"2023-10-06T05:07:56.515000+0000","last_peered":"2023-10-06T05:07:56.515000+0000","last_clean":"2023-10-06T05:07:56.515000+0000","last_became_active":"2023-10-06T05:07:56.502507+0000","last_became_peered":"2023-10-06T05:07:56.502507+0000","last_unstale":"2023-10-06T05:07:56.515000+0000","last_undegraded":"2023-10-06T05:07:56.515000+0000","last_fullsized":"2023-10-06T05:07:56.515000+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:49:48.970981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.490325+0000","last_change":"2023-10-06T05:07:55.496701+0000","last_active":"2023-10-06T05:07:56.490325+0000","last_peered":"2023-10-06T05:07:56.490325+0000","last_clean":"2023-10-06T05:07:56.490325+0000","last_became_active":"2023-10-06T05:07:55.494800+0000","last_became_peered":"2023-10-06T05:07:55.494800+0000","last_unstale":"2023-10-06T05:07:56.490325+0000","last_undegraded":"2023-10-06T05:07:56.490325+0000","last_fullsized":"2023-10-06T05:07:56.490325+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_clean_scrub_stamp":"2023-10-06T05:07:54.474583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:56:17.693306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":8,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.504348+0000","last_change":"2023-10-06T05:07:56.504348+0000","last_active":"2023-10-06T05:07:56.504348+0000","last_peered":"2023-10-06T05:07:56.504348+0000","last_clean":"2023-10-06T05:07:56.504348+0000","last_became_active":"2023-10-06T05:07:56.504094+0000","last_became_peered":"2023-10-06T05:07:56.504094+0000","last_unstale":"2023-10-06T05:07:56.504348+0000","last_undegraded":"2023-10-06T05:07:56.504348+0000","last_fullsized":"2023-10-06T05:07:56.504348+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:50:31.905666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"42'8","reported_seq":17,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.514711+0000","last_change":"2023-10-06T05:07:55.499223+0000","last_active":"2023-10-06T05:07:56.514711+0000","last_peered":"2023-10-06T05:07:56.514711+0000","last_clean":"2023-10-06T05:07:56.514711+0000","last_became_active":"2023-10-06T05:07:55.498891+0000","last_became_peered":"2023-10-06T05:07:55.498891+0000","last_unstale":"2023-10-06T05:07:56.514711+0000","last_undegraded":"2023-10-06T05:07:56.514711+0000","last_fullsized":"2023-10-06T05:07:56.514711+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_clean_scrub_stamp":"2023-10-06T05:07:54.474583+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:59:28.831493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.513600+0000","last_change":"2023-10-06T05:07:56.511252+0000","last_active":"2023-10-06T05:07:56.513600+0000","last_peered":"2023-10-06T05:07:56.513600+0000","last_clean":"2023-10-06T05:07:56.513600+0000","last_became_active":"2023-10-06T05:07:56.510997+0000","last_became_peered":"2023-10-06T05:07:56.510997+0000","last_unstale":"2023-10-06T05:07:56.513600+0000","last_undegraded":"2023-10-06T05:07:56.513600+0000","last_fullsized":"2023-10-06T05:07:56.513600+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:19:49.371903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"42'1","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.515525+0000","last_change":"2023-10-06T05:07:55.504049+0000","last_active":"2023-10-06T05:07:56.515525+0000","last_peered":"2023-10-06T05:07:56.515525+0000","last_clean":"2023-10-06T05:07:56.515525+0000","last_became_active":"2023-10-06T05:07:55.503734+0000","last_became_peered":"2023-10-06T05:07:55.503734+0000","last_unstale":"2023-10-06T05:07:56.515525+0000","last_undegraded":"2023-10-06T05:07:56.515525+0000","last_fullsized":"2023-10-06T05:07:56.515525+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_clean_scrub_stamp":"2023-10-06T05:07:54.474583+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:54:47.278321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":8,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.498002+0000","last_change":"2023-10-06T05:07:56.498002+0000","last_active":"2023-10-06T05:07:56.498002+0000","last_peered":"2023-10-06T05:07:56.498002+0000","last_clean":"2023-10-06T05:07:56.498002+0000","last_became_active":"2023-10-06T05:07:56.497774+0000","last_became_peered":"2023-10-06T05:07:56.497774+0000","last_unstale":"2023-10-06T05:07:56.498002+0000","last_undegraded":"2023-10-06T05:07:56.498002+0000","last_fullsized":"2023-10-06T05:07:56.498002+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:19:51.502890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"42'2","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.510262+0000","last_change":"2023-10-06T05:07:55.499170+0000","last_active":"2023-10-06T05:07:56.510262+0000","last_peered":"2023-10-06T05:07:56.510262+0000","last_clean":"2023-10-06T05:07:56.510262+0000","last_became_active":"2023-10-06T05:07:55.498896+0000","last_became_peered":"2023-10-06T05:07:55.498896+0000","last_unstale":"2023-10-06T05:07:56.510262+0000","last_undegraded":"2023-10-06T05:07:56.510262+0000","last_fullsized":"2023-10-06T05:07:56.510262+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_clean_scrub_stamp":"2023-10-06T05:07:54.474583+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:32:22.839464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.505645+0000","last_change":"2023-10-06T05:07:56.500218+0000","last_active":"2023-10-06T05:07:56.505645+0000","last_peered":"2023-10-06T05:07:56.505645+0000","last_clean":"2023-10-06T05:07:56.505645+0000","last_became_active":"2023-10-06T05:07:56.499968+0000","last_became_peered":"2023-10-06T05:07:56.499968+0000","last_unstale":"2023-10-06T05:07:56.505645+0000","last_undegraded":"2023-10-06T05:07:56.505645+0000","last_fullsized":"2023-10-06T05:07:56.505645+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:04:18.068814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"42'1","reported_seq":11,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.512967+0000","last_change":"2023-10-06T05:07:55.496827+0000","last_active":"2023-10-06T05:07:56.512967+0000","last_peered":"2023-10-06T05:07:56.512967+0000","last_clean":"2023-10-06T05:07:56.512967+0000","last_became_active":"2023-10-06T05:07:55.496252+0000","last_became_peered":"2023-10-06T05:07:55.496252+0000","last_unstale":"2023-10-06T05:07:56.512967+0000","last_undegraded":"2023-10-06T05:07:56.512967+0000","last_fullsized":"2023-10-06T05:07:56.512967+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_clean_scrub_stamp":"2023-10-06T05:07:54.474583+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:32:53.238080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.511412+0000","last_change":"2023-10-06T05:07:56.508203+0000","last_active":"2023-10-06T05:07:56.511412+0000","last_peered":"2023-10-06T05:07:56.511412+0000","last_clean":"2023-10-06T05:07:56.511412+0000","last_became_active":"2023-10-06T05:07:56.507596+0000","last_became_peered":"2023-10-06T05:07:56.507596+0000","last_unstale":"2023-10-06T05:07:56.511412+0000","last_undegraded":"2023-10-06T05:07:56.511412+0000","last_fullsized":"2023-10-06T05:07:56.511412+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:19:49.371903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.491515+0000","last_change":"2023-10-06T05:07:55.503922+0000","last_active":"2023-10-06T05:07:56.491515+0000","last_peered":"2023-10-06T05:07:56.491515+0000","last_clean":"2023-10-06T05:07:56.491515+0000","last_became_active":"2023-10-06T05:07:55.503647+0000","last_became_peered":"2023-10-06T05:07:55.503647+0000","last_unstale":"2023-10-06T05:07:56.491515+0000","last_undegraded":"2023-10-06T05:07:56.491515+0000","last_fullsized":"2023-10-06T05:07:56.491515+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_clean_scrub_stamp":"2023-10-06T05:07:54.474583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:23:02.744729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.515091+0000","last_change":"2023-10-06T05:07:56.503924+0000","last_active":"2023-10-06T05:07:56.515091+0000","last_peered":"2023-10-06T05:07:56.515091+0000","last_clean":"2023-10-06T05:07:56.515091+0000","last_became_active":"2023-10-06T05:07:56.503699+0000","last_became_peered":"2023-10-06T05:07:56.503699+0000","last_unstale":"2023-10-06T05:07:56.515091+0000","last_undegraded":"2023-10-06T05:07:56.515091+0000","last_fullsized":"2023-10-06T05:07:56.515091+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:02:55.738326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"42'2","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.514844+0000","last_change":"2023-10-06T05:07:55.504251+0000","last_active":"2023-10-06T05:07:56.514844+0000","last_peered":"2023-10-06T05:07:56.514844+0000","last_clean":"2023-10-06T05:07:56.514844+0000","last_became_active":"2023-10-06T05:07:55.503715+0000","last_became_peered":"2023-10-06T05:07:55.503715+0000","last_unstale":"2023-10-06T05:07:56.514844+0000","last_undegraded":"2023-10-06T05:07:56.514844+0000","last_fullsized":"2023-10-06T05:07:56.514844+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_clean_scrub_stamp":"2023-10-06T05:07:54.474583+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:49:35.919254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":8,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.494909+0000","last_change":"2023-10-06T05:07:56.494909+0000","last_active":"2023-10-06T05:07:56.494909+0000","last_peered":"2023-10-06T05:07:56.494909+0000","last_clean":"2023-10-06T05:07:56.494909+0000","last_became_active":"2023-10-06T05:07:56.494611+0000","last_became_peered":"2023-10-06T05:07:56.494611+0000","last_unstale":"2023-10-06T05:07:56.494909+0000","last_undegraded":"2023-10-06T05:07:56.494909+0000","last_fullsized":"2023-10-06T05:07:56.494909+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:47:50.444577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.489300+0000","last_change":"2023-10-06T05:07:55.503043+0000","last_active":"2023-10-06T05:07:56.489300+0000","last_peered":"2023-10-06T05:07:56.489300+0000","last_clean":"2023-10-06T05:07:56.489300+0000","last_became_active":"2023-10-06T05:07:55.502754+0000","last_became_peered":"2023-10-06T05:07:55.502754+0000","last_unstale":"2023-10-06T05:07:56.489300+0000","last_undegraded":"2023-10-06T05:07:56.489300+0000","last_fullsized":"2023-10-06T05:07:56.489300+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_clean_scrub_stamp":"2023-10-06T05:07:54.474583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:40:23.087915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.512679+0000","last_change":"2023-10-06T05:07:56.501912+0000","last_active":"2023-10-06T05:07:56.512679+0000","last_peered":"2023-10-06T05:07:56.512679+0000","last_clean":"2023-10-06T05:07:56.512679+0000","last_became_active":"2023-10-06T05:07:56.501696+0000","last_became_peered":"2023-10-06T05:07:56.501696+0000","last_unstale":"2023-10-06T05:07:56.512679+0000","last_undegraded":"2023-10-06T05:07:56.512679+0000","last_fullsized":"2023-10-06T05:07:56.512679+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:50:27.863154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T05:07:55.499254+0000","last_change":"2023-10-06T05:07:55.494220+0000","last_active":"2023-10-06T05:07:55.499254+0000","last_peered":"2023-10-06T05:07:55.499254+0000","last_clean":"2023-10-06T05:07:55.499254+0000","last_became_active":"2023-10-06T05:07:55.493933+0000","last_became_peered":"2023-10-06T05:07:55.493933+0000","last_unstale":"2023-10-06T05:07:55.499254+0000","last_undegraded":"2023-10-06T05:07:55.499254+0000","last_fullsized":"2023-10-06T05:07:55.499254+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_clean_scrub_stamp":"2023-10-06T05:07:54.474583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:25:33.478575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":8,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.507212+0000","last_change":"2023-10-06T05:07:56.507212+0000","last_active":"2023-10-06T05:07:56.507212+0000","last_peered":"2023-10-06T05:07:56.507212+0000","last_clean":"2023-10-06T05:07:56.507212+0000","last_became_active":"2023-10-06T05:07:56.506969+0000","last_became_peered":"2023-10-06T05:07:56.506969+0000","last_unstale":"2023-10-06T05:07:56.507212+0000","last_undegraded":"2023-10-06T05:07:56.507212+0000","last_fullsized":"2023-10-06T05:07:56.507212+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:24:08.047990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T05:07:55.504909+0000","last_change":"2023-10-06T05:07:55.502424+0000","last_active":"2023-10-06T05:07:55.504909+0000","last_peered":"2023-10-06T05:07:55.504909+0000","last_clean":"2023-10-06T05:07:55.504909+0000","last_became_active":"2023-10-06T05:07:55.502160+0000","last_became_peered":"2023-10-06T05:07:55.502160+0000","last_unstale":"2023-10-06T05:07:55.504909+0000","last_undegraded":"2023-10-06T05:07:55.504909+0000","last_fullsized":"2023-10-06T05:07:55.504909+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_clean_scrub_stamp":"2023-10-06T05:07:54.474583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:32:17.249933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.504423+0000","last_change":"2023-10-06T05:07:56.498223+0000","last_active":"2023-10-06T05:07:56.504423+0000","last_peered":"2023-10-06T05:07:56.504423+0000","last_clean":"2023-10-06T05:07:56.504423+0000","last_became_active":"2023-10-06T05:07:56.497954+0000","last_became_peered":"2023-10-06T05:07:56.497954+0000","last_unstale":"2023-10-06T05:07:56.504423+0000","last_undegraded":"2023-10-06T05:07:56.504423+0000","last_fullsized":"2023-10-06T05:07:56.504423+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:27:54.501823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T05:07:55.500347+0000","last_change":"2023-10-06T05:07:55.496823+0000","last_active":"2023-10-06T05:07:55.500347+0000","last_peered":"2023-10-06T05:07:55.500347+0000","last_clean":"2023-10-06T05:07:55.500347+0000","last_became_active":"2023-10-06T05:07:55.495650+0000","last_became_peered":"2023-10-06T05:07:55.495650+0000","last_unstale":"2023-10-06T05:07:55.500347+0000","last_undegraded":"2023-10-06T05:07:55.500347+0000","last_fullsized":"2023-10-06T05:07:55.500347+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_clean_scrub_stamp":"2023-10-06T05:07:54.474583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:07:49.191753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T05:07:55.486395+0000","last_change":"2023-10-06T05:06:52.627232+0000","last_active":"2023-10-06T05:07:55.486395+0000","last_peered":"2023-10-06T05:07:55.486395+0000","last_clean":"2023-10-06T05:07:55.486395+0000","last_became_active":"2023-10-06T05:06:52.626936+0000","last_became_peered":"2023-10-06T05:06:52.626936+0000","last_unstale":"2023-10-06T05:07:55.486395+0000","last_undegraded":"2023-10-06T05:07:55.486395+0000","last_fullsized":"2023-10-06T05:07:55.486395+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:20:13.925170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.505859+0000","last_change":"2023-10-06T05:07:56.499826+0000","last_active":"2023-10-06T05:07:56.505859+0000","last_peered":"2023-10-06T05:07:56.505859+0000","last_clean":"2023-10-06T05:07:56.505859+0000","last_became_active":"2023-10-06T05:07:56.499563+0000","last_became_peered":"2023-10-06T05:07:56.499563+0000","last_unstale":"2023-10-06T05:07:56.505859+0000","last_undegraded":"2023-10-06T05:07:56.505859+0000","last_fullsized":"2023-10-06T05:07:56.505859+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:21:30.591642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"43'1","reported_seq":11,"reported_epoch":43,"state":"active+clean","last_fresh":"2023-10-06T05:07:58.385493+0000","last_change":"2023-10-06T05:07:55.500013+0000","last_active":"2023-10-06T05:07:58.385493+0000","last_peered":"2023-10-06T05:07:58.385493+0000","last_clean":"2023-10-06T05:07:58.385493+0000","last_became_active":"2023-10-06T05:07:55.499366+0000","last_became_peered":"2023-10-06T05:07:55.499366+0000","last_unstale":"2023-10-06T05:07:58.385493+0000","last_undegraded":"2023-10-06T05:07:58.385493+0000","last_fullsized":"2023-10-06T05:07:58.385493+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_clean_scrub_stamp":"2023-10-06T05:07:54.474583+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:16:46.996093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":31,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.489452+0000","last_change":"2023-10-06T05:06:52.587795+0000","last_active":"2023-10-06T05:07:56.489452+0000","last_peered":"2023-10-06T05:07:56.489452+0000","last_clean":"2023-10-06T05:07:56.489452+0000","last_became_active":"2023-10-06T05:06:52.587407+0000","last_became_peered":"2023-10-06T05:06:52.587407+0000","last_unstale":"2023-10-06T05:07:56.489452+0000","last_undegraded":"2023-10-06T05:07:56.489452+0000","last_fullsized":"2023-10-06T05:07:56.489452+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:36:31.526532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":8,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.507196+0000","last_change":"2023-10-06T05:07:56.507196+0000","last_active":"2023-10-06T05:07:56.507196+0000","last_peered":"2023-10-06T05:07:56.507196+0000","last_clean":"2023-10-06T05:07:56.507196+0000","last_became_active":"2023-10-06T05:07:56.506806+0000","last_became_peered":"2023-10-06T05:07:56.506806+0000","last_unstale":"2023-10-06T05:07:56.507196+0000","last_undegraded":"2023-10-06T05:07:56.507196+0000","last_fullsized":"2023-10-06T05:07:56.507196+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:36:03.141466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T05:07:55.509630+0000","last_change":"2023-10-06T05:07:55.504758+0000","last_active":"2023-10-06T05:07:55.509630+0000","last_peered":"2023-10-06T05:07:55.509630+0000","last_clean":"2023-10-06T05:07:55.509630+0000","last_became_active":"2023-10-06T05:07:55.504470+0000","last_became_peered":"2023-10-06T05:07:55.504470+0000","last_unstale":"2023-10-06T05:07:55.509630+0000","last_undegraded":"2023-10-06T05:07:55.509630+0000","last_fullsized":"2023-10-06T05:07:55.509630+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_clean_scrub_stamp":"2023-10-06T05:07:54.474583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:03:15.774462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":28,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.499543+0000","last_change":"2023-10-06T05:06:52.675238+0000","last_active":"2023-10-06T05:07:56.499543+0000","last_peered":"2023-10-06T05:07:56.499543+0000","last_clean":"2023-10-06T05:07:56.499543+0000","last_became_active":"2023-10-06T05:06:52.674952+0000","last_became_peered":"2023-10-06T05:06:52.674952+0000","last_unstale":"2023-10-06T05:07:56.499543+0000","last_undegraded":"2023-10-06T05:07:56.499543+0000","last_fullsized":"2023-10-06T05:07:56.499543+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:27:51.141184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":8,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.499948+0000","last_change":"2023-10-06T05:07:56.499948+0000","last_active":"2023-10-06T05:07:56.499948+0000","last_peered":"2023-10-06T05:07:56.499948+0000","last_clean":"2023-10-06T05:07:56.499948+0000","last_became_active":"2023-10-06T05:07:56.499705+0000","last_became_peered":"2023-10-06T05:07:56.499705+0000","last_unstale":"2023-10-06T05:07:56.499948+0000","last_undegraded":"2023-10-06T05:07:56.499948+0000","last_fullsized":"2023-10-06T05:07:56.499948+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:05:12.786696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"42'2","reported_seq":12,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.515946+0000","last_change":"2023-10-06T05:07:55.503632+0000","last_active":"2023-10-06T05:07:56.515946+0000","last_peered":"2023-10-06T05:07:56.515946+0000","last_clean":"2023-10-06T05:07:56.515946+0000","last_became_active":"2023-10-06T05:07:55.503429+0000","last_became_peered":"2023-10-06T05:07:55.503429+0000","last_unstale":"2023-10-06T05:07:56.515946+0000","last_undegraded":"2023-10-06T05:07:56.515946+0000","last_fullsized":"2023-10-06T05:07:56.515946+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_clean_scrub_stamp":"2023-10-06T05:07:54.474583+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:45:37.139013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":31,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.489583+0000","last_change":"2023-10-06T05:06:52.587778+0000","last_active":"2023-10-06T05:07:56.489583+0000","last_peered":"2023-10-06T05:07:56.489583+0000","last_clean":"2023-10-06T05:07:56.489583+0000","last_became_active":"2023-10-06T05:06:52.587388+0000","last_became_peered":"2023-10-06T05:06:52.587388+0000","last_unstale":"2023-10-06T05:07:56.489583+0000","last_undegraded":"2023-10-06T05:07:56.489583+0000","last_fullsized":"2023-10-06T05:07:56.489583+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:45:52.502298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":181,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T05:07:55.490762+0000","last_change":"2023-10-06T05:06:46.307169+0000","last_active":"2023-10-06T05:07:55.490762+0000","last_peered":"2023-10-06T05:07:55.490762+0000","last_clean":"2023-10-06T05:07:55.490762+0000","last_became_active":"2023-10-06T05:06:46.306107+0000","last_became_peered":"2023-10-06T05:06:46.306107+0000","last_unstale":"2023-10-06T05:07:55.490762+0000","last_undegraded":"2023-10-06T05:07:55.490762+0000","last_fullsized":"2023-10-06T05:07:55.490762+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:45.018006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:45.018006+0000","last_clean_scrub_stamp":"2023-10-06T05:06:45.018006+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:38:17.064131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":8,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.502108+0000","last_change":"2023-10-06T05:07:56.502108+0000","last_active":"2023-10-06T05:07:56.502108+0000","last_peered":"2023-10-06T05:07:56.502108+0000","last_clean":"2023-10-06T05:07:56.502108+0000","last_became_active":"2023-10-06T05:07:56.501855+0000","last_became_peered":"2023-10-06T05:07:56.501855+0000","last_unstale":"2023-10-06T05:07:56.502108+0000","last_undegraded":"2023-10-06T05:07:56.502108+0000","last_fullsized":"2023-10-06T05:07:56.502108+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:07:44.560707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T05:07:55.502308+0000","last_change":"2023-10-06T05:07:55.497336+0000","last_active":"2023-10-06T05:07:55.502308+0000","last_peered":"2023-10-06T05:07:55.502308+0000","last_clean":"2023-10-06T05:07:55.502308+0000","last_became_active":"2023-10-06T05:07:55.497065+0000","last_became_peered":"2023-10-06T05:07:55.497065+0000","last_unstale":"2023-10-06T05:07:55.502308+0000","last_undegraded":"2023-10-06T05:07:55.502308+0000","last_fullsized":"2023-10-06T05:07:55.502308+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_clean_scrub_stamp":"2023-10-06T05:07:54.474583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:13:54.766249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":28,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T05:07:55.490849+0000","last_change":"2023-10-06T05:06:52.588475+0000","last_active":"2023-10-06T05:07:55.490849+0000","last_peered":"2023-10-06T05:07:55.490849+0000","last_clean":"2023-10-06T05:07:55.490849+0000","last_became_active":"2023-10-06T05:06:52.588134+0000","last_became_peered":"2023-10-06T05:06:52.588134+0000","last_unstale":"2023-10-06T05:07:55.490849+0000","last_undegraded":"2023-10-06T05:07:55.490849+0000","last_fullsized":"2023-10-06T05:07:55.490849+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:35:05.269778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.506698+0000","last_change":"2023-10-06T05:07:56.503635+0000","last_active":"2023-10-06T05:07:56.506698+0000","last_peered":"2023-10-06T05:07:56.506698+0000","last_clean":"2023-10-06T05:07:56.506698+0000","last_became_active":"2023-10-06T05:07:56.503368+0000","last_became_peered":"2023-10-06T05:07:56.503368+0000","last_unstale":"2023-10-06T05:07:56.506698+0000","last_undegraded":"2023-10-06T05:07:56.506698+0000","last_fullsized":"2023-10-06T05:07:56.506698+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:23:03.746840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T05:07:55.507716+0000","last_change":"2023-10-06T05:07:55.505883+0000","last_active":"2023-10-06T05:07:55.507716+0000","last_peered":"2023-10-06T05:07:55.507716+0000","last_clean":"2023-10-06T05:07:55.507716+0000","last_became_active":"2023-10-06T05:07:55.505622+0000","last_became_peered":"2023-10-06T05:07:55.505622+0000","last_unstale":"2023-10-06T05:07:55.507716+0000","last_undegraded":"2023-10-06T05:07:55.507716+0000","last_fullsized":"2023-10-06T05:07:55.507716+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_clean_scrub_stamp":"2023-10-06T05:07:54.474583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T16:44:35.675612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":28,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T05:07:55.492816+0000","last_change":"2023-10-06T05:06:52.627289+0000","last_active":"2023-10-06T05:07:55.492816+0000","last_peered":"2023-10-06T05:07:55.492816+0000","last_clean":"2023-10-06T05:07:55.492816+0000","last_became_active":"2023-10-06T05:06:52.626937+0000","last_became_peered":"2023-10-06T05:06:52.626937+0000","last_unstale":"2023-10-06T05:07:55.492816+0000","last_undegraded":"2023-10-06T05:07:55.492816+0000","last_fullsized":"2023-10-06T05:07:55.492816+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:46:59.070674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.515005+0000","last_change":"2023-10-06T05:07:56.506104+0000","last_active":"2023-10-06T05:07:56.515005+0000","last_peered":"2023-10-06T05:07:56.515005+0000","last_clean":"2023-10-06T05:07:56.515005+0000","last_became_active":"2023-10-06T05:07:56.505829+0000","last_became_peered":"2023-10-06T05:07:56.505829+0000","last_unstale":"2023-10-06T05:07:56.515005+0000","last_undegraded":"2023-10-06T05:07:56.515005+0000","last_fullsized":"2023-10-06T05:07:56.515005+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:27:31.635411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":10,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.487139+0000","last_change":"2023-10-06T05:07:55.492476+0000","last_active":"2023-10-06T05:07:56.487139+0000","last_peered":"2023-10-06T05:07:56.487139+0000","last_clean":"2023-10-06T05:07:56.487139+0000","last_became_active":"2023-10-06T05:07:55.492199+0000","last_became_peered":"2023-10-06T05:07:55.492199+0000","last_unstale":"2023-10-06T05:07:56.487139+0000","last_undegraded":"2023-10-06T05:07:56.487139+0000","last_fullsized":"2023-10-06T05:07:56.487139+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_clean_scrub_stamp":"2023-10-06T05:07:54.474583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T17:05:11.493391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":27,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T05:07:55.489889+0000","last_change":"2023-10-06T05:06:52.588853+0000","last_active":"2023-10-06T05:07:55.489889+0000","last_peered":"2023-10-06T05:07:55.489889+0000","last_clean":"2023-10-06T05:07:55.489889+0000","last_became_active":"2023-10-06T05:06:52.588561+0000","last_became_peered":"2023-10-06T05:06:52.588561+0000","last_unstale":"2023-10-06T05:07:55.489889+0000","last_undegraded":"2023-10-06T05:07:55.489889+0000","last_fullsized":"2023-10-06T05:07:55.489889+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:06:28.729493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":9,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.515110+0000","last_change":"2023-10-06T05:07:56.510457+0000","last_active":"2023-10-06T05:07:56.515110+0000","last_peered":"2023-10-06T05:07:56.515110+0000","last_clean":"2023-10-06T05:07:56.515110+0000","last_became_active":"2023-10-06T05:07:56.510184+0000","last_became_peered":"2023-10-06T05:07:56.510184+0000","last_unstale":"2023-10-06T05:07:56.515110+0000","last_undegraded":"2023-10-06T05:07:56.515110+0000","last_fullsized":"2023-10-06T05:07:56.515110+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:55.476694+0000","last_clean_scrub_stamp":"2023-10-06T05:07:55.476694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:59:52.915174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":9,"reported_epoch":41,"state":"active+clean","last_fresh":"2023-10-06T05:07:55.506995+0000","last_change":"2023-10-06T05:07:55.504325+0000","last_active":"2023-10-06T05:07:55.506995+0000","last_peered":"2023-10-06T05:07:55.506995+0000","last_clean":"2023-10-06T05:07:55.506995+0000","last_became_active":"2023-10-06T05:07:55.504037+0000","last_became_peered":"2023-10-06T05:07:55.504037+0000","last_unstale":"2023-10-06T05:07:55.506995+0000","last_undegraded":"2023-10-06T05:07:55.506995+0000","last_fullsized":"2023-10-06T05:07:55.506995+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":40,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:07:54.474583+0000","last_clean_scrub_stamp":"2023-10-06T05:07:54.474583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:44:32.978639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.491222+0000","last_change":"2023-10-06T05:06:52.589519+0000","last_active":"2023-10-06T05:07:56.491222+0000","last_peered":"2023-10-06T05:07:56.491222+0000","last_clean":"2023-10-06T05:07:56.491222+0000","last_became_active":"2023-10-06T05:06:52.589246+0000","last_became_peered":"2023-10-06T05:06:52.589246+0000","last_unstale":"2023-10-06T05:07:56.491222+0000","last_undegraded":"2023-10-06T05:07:56.491222+0000","last_fullsized":"2023-10-06T05:07:56.491222+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:54:23.352764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509459,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297776,"kb_used_data":488,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074064,"statfs":{"total":96636764160,"available":96331841536,"internally_reserved":0,"allocated":499712,"data_stored":219280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.437},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.46600000000000003}]},{"osd":1,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.39200000000000002}]},{"osd":2,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.41899999999999998}]},{"osd":3,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.54300000000000004}]},{"osd":4,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.57499999999999996}]},{"osd":5,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.64900000000000002}]},{"osd":7,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.70499999999999996}]}]},{"osd":5,"up_from":15,"seq":64424509459,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297772,"kb_used_data":484,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074068,"statfs":{"total":96636764160,"available":96331845632,"internally_reserved":0,"allocated":495616,"data_stored":218706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.44700000000000001}]},{"osd":1,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.5},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.48099999999999998}]},{"osd":2,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.63100000000000001}]},{"osd":3,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.65000000000000002}]},{"osd":4,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.752}]},{"osd":6,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.73699999999999999}]},{"osd":7,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.70799999999999996}]}]},{"osd":0,"up_from":14,"seq":60129542163,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298340,"kb_used_data":1052,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073500,"statfs":{"total":96636764160,"available":96331264000,"internally_reserved":0,"allocated":1077248,"data_stored":802077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":1.2809999999999999,"5min":1.2809999999999999,"15min":1.2809999999999999},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.35499999999999998}]},{"osd":2,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":4.1529999999999996,"5min":4.1529999999999996,"15min":4.1529999999999996},"last":0.308},{"interface":"front","average":{"1min":0.372,"5min":0.372,"15min":0.372},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":1.105,"5min":1.105,"15min":1.105},"last":0.26200000000000001}]},{"osd":3,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":1.327,"5min":1.327,"15min":1.327},"last":0.246},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":4.1900000000000004,"5min":4.1900000000000004,"15min":4.1900000000000004},"last":0.40000000000000002}]},{"osd":4,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.53200000000000003}]},{"osd":5,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.304,"5min":1.304,"15min":1.304},"last":0.60999999999999999}]},{"osd":6,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.622},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.54600000000000004}]},{"osd":7,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":1.079,"5min":1.079,"15min":1.079},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.64400000000000002}]}]},{"osd":7,"up_from":13,"seq":55834574866,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298340,"kb_used_data":1052,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073500,"statfs":{"total":96636764160,"available":96331264000,"internally_reserved":0,"allocated":1077248,"data_stored":802133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.49299999999999999}]},{"osd":1,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.36599999999999999}]},{"osd":2,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.45100000000000001}]},{"osd":3,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.56799999999999995}]},{"osd":4,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.60899999999999999}]},{"osd":5,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.54800000000000004}]},{"osd":6,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.66700000000000004}]}]},{"osd":1,"up_from":14,"seq":60129542162,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297760,"kb_used_data":472,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074080,"statfs":{"total":96636764160,"available":96331857920,"internally_reserved":0,"allocated":483328,"data_stored":211745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":1.8879999999999999,"5min":1.8879999999999999,"15min":1.8879999999999999},"last":0.251},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":1.595,"5min":1.595,"15min":1.595},"last":0.378}]},{"osd":2,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.123,"5min":1.123,"15min":1.123},"last":0.32300000000000001}]},{"osd":3,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.179,"5min":1.179,"15min":1.179},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1.657,"5min":1.657,"15min":1.657},"last":0.34200000000000003}]},{"osd":4,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":1.6319999999999999,"5min":1.6319999999999999,"15min":1.6319999999999999},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.837,"5min":1.837,"15min":1.837},"last":0.496}]},{"osd":5,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.8100000000000001,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.56999999999999995}]},{"osd":6,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.6850000000000001,"5min":1.6850000000000001,"15min":1.6850000000000001},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.8620000000000001,"5min":1.8620000000000001,"15min":1.8620000000000001},"last":0.52500000000000002}]},{"osd":7,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.7789999999999999,"5min":1.7789999999999999,"15min":1.7789999999999999},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.53900000000000003}]}]},{"osd":2,"up_from":15,"seq":64424509458,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297784,"kb_used_data":496,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074056,"statfs":{"total":96636764160,"available":96331833344,"internally_reserved":0,"allocated":507904,"data_stored":220194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.311,"5min":0.311,"15min":0.311},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.25800000000000001}]},{"osd":1,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"last":0.317}]},{"osd":3,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"last":0.36299999999999999}]},{"osd":4,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.70899999999999996}]},{"osd":5,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.71799999999999997}]},{"osd":6,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.753},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.63300000000000001}]},{"osd":7,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.78100000000000003}]}]},{"osd":3,"up_from":15,"seq":64424509458,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297780,"kb_used_data":492,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074060,"statfs":{"total":96636764160,"available":96331837440,"internally_reserved":0,"allocated":503808,"data_stored":219246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":3.391,"5min":3.391,"15min":3.391},"last":0.27800000000000002}]},{"osd":1,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":2.8740000000000001,"5min":2.8740000000000001,"15min":2.8740000000000001},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.376,"5min":0.376,"15min":0.376},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.308}]},{"osd":2,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":7.2210000000000001,"5min":7.2210000000000001,"15min":7.2210000000000001},"last":0.378}]},{"osd":4,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.018,"5min":1.018,"15min":1.018},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.6930000000000001,"5min":1.6930000000000001,"15min":1.6930000000000001},"last":0.71499999999999997}]},{"osd":5,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.607,"5min":1.607,"15min":1.607},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.67200000000000004}]},{"osd":6,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.657,"5min":1.657,"15min":1.657},"last":0.74199999999999999}]},{"osd":7,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.877,"5min":1.877,"15min":1.877},"last":0.64500000000000002}]}]},{"osd":4,"up_from":15,"seq":64424509459,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297780,"kb_used_data":492,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074060,"statfs":{"total":96636764160,"available":96331837440,"internally_reserved":0,"allocated":503808,"data_stored":220172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.59099999999999997}]},{"osd":1,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.63600000000000001}]},{"osd":2,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.69299999999999995}]},{"osd":3,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":11.385999999999999,"5min":11.385999999999999,"15min":11.385999999999999},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.72999999999999998}]},{"osd":5,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.66800000000000004}]},{"osd":6,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.75700000000000001}]},{"osd":7,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.81399999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-06T05:08:00.936 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-06T05:08:00.937 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-06T05:08:01.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.090+0000 7f25b7864700 1 Processor -- start 2023-10-06T05:08:01.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.090+0000 7f25b7864700 1 -- start start 2023-10-06T05:08:01.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.091+0000 7f25b7864700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f25b01598a0 0x7f25b00ce7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:01.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.091+0000 7f25b7864700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f25b00cbc70 0x7f25b00ced20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:01.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.091+0000 7f25b7864700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25b00cf5d0 con 0x7f25b01594d0 2023-10-06T05:08:01.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.091+0000 7f25b7864700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25b00cf8b0 con 0x7f25b00cbc70 2023-10-06T05:08:01.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.091+0000 7f25b7864700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25b00cfb90 con 0x7f25b01598a0 2023-10-06T05:08:01.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.091+0000 7f25b5600700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f25b01594d0 0x7f25b00cb560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:38368/0 (socket says 172.21.15.98:38368) 2023-10-06T05:08:01.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.091+0000 7f25b5600700 1 -- 172.21.15.98:0/144978278 learned_addr learned my addr 172.21.15.98:0/144978278 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:08:01.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.091+0000 7f25b5e01700 1 --2- 172.21.15.98:0/144978278 >> v2:172.21.15.98:3300/0 conn(0x7f25b00cbc70 0x7f25b00ced20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:01.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.091+0000 7f25b4dff700 1 --2- 172.21.15.98:0/144978278 >> v2:172.21.15.174:3300/0 conn(0x7f25b01598a0 0x7f25b00ce7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:01.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.091+0000 7f25a7fff700 1 -- 172.21.15.98:0/144978278 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 420299503 0 0) 0x7f25b00cf5d0 con 0x7f25b01594d0 2023-10-06T05:08:01.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.091+0000 7f25a7fff700 1 -- 172.21.15.98:0/144978278 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f259c003570 con 0x7f25b01594d0 2023-10-06T05:08:01.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.091+0000 7f25b5e01700 1 -- 172.21.15.98:0/144978278 >> v2:172.21.15.174:3300/0 conn(0x7f25b01598a0 msgr2=0x7f25b00ce7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:01.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.091+0000 7f25b5e01700 1 --2- 172.21.15.98:0/144978278 >> v2:172.21.15.174:3300/0 conn(0x7f25b01598a0 0x7f25b00ce7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:01.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.091+0000 7f25b5e01700 1 -- 172.21.15.98:0/144978278 >> v1:172.21.15.98:6789/0 conn(0x7f25b01594d0 legacy=0x7f25b00cb560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:01.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.091+0000 7f25b5e01700 1 -- 172.21.15.98:0/144978278 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25b00cfe70 con 0x7f25b00cbc70 2023-10-06T05:08:01.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.091+0000 7f25b5e01700 1 --2- 172.21.15.98:0/144978278 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25b00cbc70 0x7f25b00ced20 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f25b00c7370 tx=0x7f25a8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7203f486fb83415 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:01.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.092+0000 7f25a7fff700 1 -- 172.21.15.98:0/144978278 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25a800e070 con 0x7f25b00cbc70 2023-10-06T05:08:01.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.092+0000 7f25a7fff700 1 -- 172.21.15.98:0/144978278 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25a800bb60 con 0x7f25b00cbc70 2023-10-06T05:08:01.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.092+0000 7f25a7fff700 1 -- 172.21.15.98:0/144978278 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25a8012430 con 0x7f25b00cbc70 2023-10-06T05:08:01.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.092+0000 7f25b7864700 1 -- 172.21.15.98:0/144978278 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25b00cbc70 msgr2=0x7f25b00ced20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:01.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.092+0000 7f25b7864700 1 --2- 172.21.15.98:0/144978278 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25b00cbc70 0x7f25b00ced20 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f25b00c7370 tx=0x7f25a8009780 comp rx=0 tx=0).stop 2023-10-06T05:08:01.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.092+0000 7f25b7864700 1 -- 172.21.15.98:0/144978278 shutdown_connections 2023-10-06T05:08:01.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.092+0000 7f25b7864700 1 --2- 172.21.15.98:0/144978278 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25b00cbc70 0x7f25b00ced20 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:01.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.092+0000 7f25b7864700 1 --2- 172.21.15.98:0/144978278 >> v2:172.21.15.174:3300/0 conn(0x7f25b01598a0 0x7f25b00ce7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:01.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.092+0000 7f25b7864700 1 -- 172.21.15.98:0/144978278 >> 172.21.15.98:0/144978278 conn(0x7f25b0089e50 msgr2=0x7f25b0099770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:01.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.092+0000 7f25b7864700 1 -- 172.21.15.98:0/144978278 shutdown_connections 2023-10-06T05:08:01.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.092+0000 7f25b7864700 1 -- 172.21.15.98:0/144978278 wait complete. 2023-10-06T05:08:01.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.093+0000 7f25b7864700 1 Processor -- start 2023-10-06T05:08:01.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.093+0000 7f25b7864700 1 -- start start 2023-10-06T05:08:01.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.094+0000 7f25b7864700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25b00cbc70 0x7f25b014a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:01.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.094+0000 7f25b7864700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f25b01594d0 0x7f25b00694f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:01.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.094+0000 7f25b7864700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25b01598a0 0x7f25b0069a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:01.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.094+0000 7f25b7864700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25b0159e60 con 0x7f25b00cbc70 2023-10-06T05:08:01.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.094+0000 7f25b7864700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25b00b35c0 con 0x7f25b01598a0 2023-10-06T05:08:01.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.094+0000 7f25b7864700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25b014a8a0 con 0x7f25b01594d0 2023-10-06T05:08:01.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.094+0000 7f25b5600700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25b00cbc70 0x7f25b014a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:01.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.094+0000 7f25b5600700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25b00cbc70 0x7f25b014a360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54812/0 (socket says 172.21.15.98:54812) 2023-10-06T05:08:01.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.094+0000 7f25b5600700 1 -- 172.21.15.98:0/1940075129 learned_addr learned my addr 172.21.15.98:0/1940075129 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:08:01.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.095+0000 7f25b5600700 1 -- 172.21.15.98:0/1940075129 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f25b01594d0 msgr2=0x7f25b00694f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:01.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.095+0000 7f25b5600700 1 --2- 172.21.15.98:0/1940075129 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f25b01594d0 0x7f25b00694f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:01.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.095+0000 7f25b5600700 1 -- 172.21.15.98:0/1940075129 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25b01598a0 msgr2=0x7f25b0069a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:01.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.095+0000 7f25b5600700 1 --2- 172.21.15.98:0/1940075129 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25b01598a0 0x7f25b0069a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:01.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.095+0000 7f25b5600700 1 -- 172.21.15.98:0/1940075129 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25a8009a20 con 0x7f25b00cbc70 2023-10-06T05:08:01.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.095+0000 7f25b5600700 1 --2- 172.21.15.98:0/1940075129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25b00cbc70 0x7f25b014a360 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f25ac00c7f0 tx=0x7f25ac007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:01.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.095+0000 7f25a57fa700 1 -- 172.21.15.98:0/1940075129 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25ac005b30 con 0x7f25b00cbc70 2023-10-06T05:08:01.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.095+0000 7f25a57fa700 1 -- 172.21.15.98:0/1940075129 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25ac005c90 con 0x7f25b00cbc70 2023-10-06T05:08:01.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.095+0000 7f25b7864700 1 -- 172.21.15.98:0/1940075129 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25b00b5880 con 0x7f25b00cbc70 2023-10-06T05:08:01.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.095+0000 7f25a57fa700 1 -- 172.21.15.98:0/1940075129 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25ac007320 con 0x7f25b00cbc70 2023-10-06T05:08:01.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.095+0000 7f25b7864700 1 -- 172.21.15.98:0/1940075129 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25b00b5e40 con 0x7f25b00cbc70 2023-10-06T05:08:01.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.096+0000 7f25b7864700 1 -- 172.21.15.98:0/1940075129 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25b0064a90 con 0x7f25b00cbc70 2023-10-06T05:08:01.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.096+0000 7f25a57fa700 1 -- 172.21.15.98:0/1940075129 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f25ac01b400 con 0x7f25b00cbc70 2023-10-06T05:08:01.102 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.097+0000 7f25a57fa700 1 --2- 172.21.15.98:0/1940075129 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f2594066f30 0x7f25940693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:01.102 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.097+0000 7f25b4dff700 1 --2- 172.21.15.98:0/1940075129 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f2594066f30 0x7f25940693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:01.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.097+0000 7f25a57fa700 1 -- 172.21.15.98:0/1940075129 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6378+0+0 (secure 0 0 0) 0x7f25ac0842b0 con 0x7f25b00cbc70 2023-10-06T05:08:01.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.097+0000 7f25b4dff700 1 --2- 172.21.15.98:0/1940075129 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f2594066f30 0x7f25940693e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f25b0158bf0 tx=0x7f25a0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:01.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.101+0000 7f25a57fa700 1 -- 172.21.15.98:0/1940075129 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f25ac051300 con 0x7f25b00cbc70 2023-10-06T05:08:01.361 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.359+0000 7f25b7864700 1 -- 172.21.15.98:0/1940075129 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f25b004bb40 con 0x7f25b00cbc70 2023-10-06T05:08:01.365 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.363+0000 7f25a57fa700 1 -- 172.21.15.98:0/1940075129 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v11) v1 ==== 190+0+0 (secure 0 0 0) 0x7f25ac054920 con 0x7f25b00cbc70 2023-10-06T05:08:01.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.365+0000 7f25b7864700 1 -- 172.21.15.98:0/1940075129 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f2594066f30 msgr2=0x7f25940693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:01.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.365+0000 7f25b7864700 1 --2- 172.21.15.98:0/1940075129 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f2594066f30 0x7f25940693e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f25b0158bf0 tx=0x7f25a0008040 comp rx=0 tx=0).stop 2023-10-06T05:08:01.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.365+0000 7f25b7864700 1 -- 172.21.15.98:0/1940075129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25b00cbc70 msgr2=0x7f25b014a360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:01.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.365+0000 7f25b7864700 1 --2- 172.21.15.98:0/1940075129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25b00cbc70 0x7f25b014a360 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f25ac00c7f0 tx=0x7f25ac007e00 comp rx=0 tx=0).stop 2023-10-06T05:08:01.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.365+0000 7f25b7864700 1 -- 172.21.15.98:0/1940075129 shutdown_connections 2023-10-06T05:08:01.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.365+0000 7f25b7864700 1 --2- 172.21.15.98:0/1940075129 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25b01598a0 0x7f25b0069a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:01.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.365+0000 7f25b7864700 1 --2- 172.21.15.98:0/1940075129 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f2594066f30 0x7f25940693e0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:01.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.365+0000 7f25b7864700 1 --2- 172.21.15.98:0/1940075129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25b00cbc70 0x7f25b014a360 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:01.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.365+0000 7f25b7864700 1 --2- 172.21.15.98:0/1940075129 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f25b01594d0 0x7f25b00694f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:01.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.365+0000 7f25b7864700 1 -- 172.21.15.98:0/1940075129 >> 172.21.15.98:0/1940075129 conn(0x7f25b0089e50 msgr2=0x7f25b014b120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:01.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.365+0000 7f25b7864700 1 -- 172.21.15.98:0/1940075129 shutdown_connections 2023-10-06T05:08:01.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.365+0000 7f25b7864700 1 -- 172.21.15.98:0/1940075129 wait complete. 2023-10-06T05:08:01.369 INFO:teuthology.orchestra.run.smithi098.stderr:updated caps for client.0 2023-10-06T05:08:01.379 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-06T05:08:01.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.492+0000 7f93d222b700 1 Processor -- start 2023-10-06T05:08:01.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.492+0000 7f93d222b700 1 -- start start 2023-10-06T05:08:01.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.492+0000 7f93d222b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f93cc0b49a0 0x7f93cc0aac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:01.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.493+0000 7f93d222b700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93cc0b0c60 con 0x7f93cc159560 2023-10-06T05:08:01.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.493+0000 7f93d222b700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93cc0b0f40 con 0x7f93cc0b0760 2023-10-06T05:08:01.497 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.493+0000 7f93d222b700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93cc0b1220 con 0x7f93cc0b49a0 2023-10-06T05:08:01.497 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.493+0000 7f93caffd700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f93cc0b49a0 0x7f93cc0aac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:01.497 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.493+0000 7f93cb7fe700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f93cc159560 0x7f93cc0aa530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:38376/0 (socket says 172.21.15.98:38376) 2023-10-06T05:08:01.497 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.493+0000 7f93cb7fe700 1 -- 172.21.15.98:0/1009057018 learned_addr learned my addr 172.21.15.98:0/1009057018 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:08:01.497 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.493+0000 7f93caffd700 1 -- 172.21.15.98:0/1009057018 >> v1:172.21.15.174:6789/0 conn(0x7f93cc0b0760 legacy=0x7f93cc0b1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:01.497 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.493+0000 7f93caffd700 1 -- 172.21.15.98:0/1009057018 >> v1:172.21.15.98:6789/0 conn(0x7f93cc159560 legacy=0x7f93cc0aa530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:01.497 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.493+0000 7f93ca7fc700 1 -- 172.21.15.98:0/1009057018 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2439105090 0 0) 0x7f93cc0b0c60 con 0x7f93cc159560 2023-10-06T05:08:01.497 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.493+0000 7f93caffd700 1 -- 172.21.15.98:0/1009057018 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93cc0b1500 con 0x7f93cc0b49a0 2023-10-06T05:08:01.497 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.493+0000 7f93caffd700 1 --2- 172.21.15.98:0/1009057018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93cc0b49a0 0x7f93cc0aac40 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f93b4009910 tx=0x7f93b40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b13af29c65538b2 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:01.497 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.493+0000 7f93ca7fc700 1 -- 172.21.15.98:0/1009057018 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93b400b7e0 con 0x7f93cc0b49a0 2023-10-06T05:08:01.497 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.493+0000 7f93ca7fc700 1 -- 172.21.15.98:0/1009057018 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93b400b940 con 0x7f93cc0b49a0 2023-10-06T05:08:01.497 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.493+0000 7f93ca7fc700 1 -- 172.21.15.98:0/1009057018 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93b400bce0 con 0x7f93cc0b49a0 2023-10-06T05:08:01.498 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.494+0000 7f93d222b700 1 -- 172.21.15.98:0/1009057018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93cc0b49a0 msgr2=0x7f93cc0aac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:01.498 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.494+0000 7f93d222b700 1 --2- 172.21.15.98:0/1009057018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93cc0b49a0 0x7f93cc0aac40 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f93b4009910 tx=0x7f93b40091a0 comp rx=0 tx=0).stop 2023-10-06T05:08:01.498 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.494+0000 7f93d222b700 1 -- 172.21.15.98:0/1009057018 shutdown_connections 2023-10-06T05:08:01.498 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.494+0000 7f93d222b700 1 --2- 172.21.15.98:0/1009057018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93cc0b49a0 0x7f93cc0aac40 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:01.498 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.494+0000 7f93d222b700 1 -- 172.21.15.98:0/1009057018 >> 172.21.15.98:0/1009057018 conn(0x7f93cc0997f0 msgr2=0x7f93cc06f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:01.498 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.494+0000 7f93d222b700 1 -- 172.21.15.98:0/1009057018 shutdown_connections 2023-10-06T05:08:01.498 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.494+0000 7f93d222b700 1 -- 172.21.15.98:0/1009057018 wait complete. 2023-10-06T05:08:01.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.495+0000 7f93d222b700 1 Processor -- start 2023-10-06T05:08:01.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.495+0000 7f93d222b700 1 -- start start 2023-10-06T05:08:01.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.496+0000 7f93d222b700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f93cc0b0760 0x7f93cc09a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:01.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.496+0000 7f93d222b700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f93cc0b49a0 0x7f93cc09b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:01.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.496+0000 7f93d222b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93cc159560 0x7f93cc09d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:01.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.496+0000 7f93d222b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93cc15be10 con 0x7f93cc159560 2023-10-06T05:08:01.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.496+0000 7f93d222b700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93cc156f40 con 0x7f93cc0b49a0 2023-10-06T05:08:01.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.496+0000 7f93d222b700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93cc09afd0 con 0x7f93cc0b0760 2023-10-06T05:08:01.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.496+0000 7f93cbfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93cc159560 0x7f93cc09d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:01.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.496+0000 7f93cbfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93cc159560 0x7f93cc09d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54824/0 (socket says 172.21.15.98:54824) 2023-10-06T05:08:01.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.496+0000 7f93cbfff700 1 -- 172.21.15.98:0/304622379 learned_addr learned my addr 172.21.15.98:0/304622379 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:08:01.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.496+0000 7f93caffd700 1 --2- 172.21.15.98:0/304622379 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f93cc0b49a0 0x7f93cc09b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:01.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.496+0000 7f93cb7fe700 1 --2- 172.21.15.98:0/304622379 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f93cc0b0760 0x7f93cc09a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:01.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.496+0000 7f93cbfff700 1 -- 172.21.15.98:0/304622379 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f93cc0b0760 msgr2=0x7f93cc09a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:01.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.496+0000 7f93cbfff700 1 --2- 172.21.15.98:0/304622379 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f93cc0b0760 0x7f93cc09a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:01.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.496+0000 7f93cbfff700 1 -- 172.21.15.98:0/304622379 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f93cc0b49a0 msgr2=0x7f93cc09b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:01.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.496+0000 7f93cbfff700 1 --2- 172.21.15.98:0/304622379 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f93cc0b49a0 0x7f93cc09b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:01.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.496+0000 7f93cbfff700 1 -- 172.21.15.98:0/304622379 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93b4009420 con 0x7f93cc159560 2023-10-06T05:08:01.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.496+0000 7f93caffd700 1 --2- 172.21.15.98:0/304622379 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f93cc0b49a0 0x7f93cc09b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:08:01.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.496+0000 7f93cbfff700 1 --2- 172.21.15.98:0/304622379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93cc159560 0x7f93cc09d410 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f93bc00c5b0 tx=0x7f93bc007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:01.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.497+0000 7f93c37fe700 1 -- 172.21.15.98:0/304622379 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93bc011590 con 0x7f93cc159560 2023-10-06T05:08:01.503 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.497+0000 7f93c37fe700 1 -- 172.21.15.98:0/304622379 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93bc0116f0 con 0x7f93cc159560 2023-10-06T05:08:01.503 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.497+0000 7f93d222b700 1 -- 172.21.15.98:0/304622379 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93cc09a730 con 0x7f93cc159560 2023-10-06T05:08:01.503 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.497+0000 7f93c37fe700 1 -- 172.21.15.98:0/304622379 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93bc011ab0 con 0x7f93cc159560 2023-10-06T05:08:01.503 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.497+0000 7f93d222b700 1 -- 172.21.15.98:0/304622379 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93cc09ad70 con 0x7f93cc159560 2023-10-06T05:08:01.503 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.498+0000 7f93d222b700 1 -- 172.21.15.98:0/304622379 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93cc064a90 con 0x7f93cc159560 2023-10-06T05:08:01.503 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.498+0000 7f93c37fe700 1 -- 172.21.15.98:0/304622379 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f93bc01c460 con 0x7f93cc159560 2023-10-06T05:08:01.503 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.498+0000 7f93c37fe700 1 --2- 172.21.15.98:0/304622379 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f93ac066f30 0x7f93ac0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:01.503 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.499+0000 7f93cb7fe700 1 --2- 172.21.15.98:0/304622379 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f93ac066f30 0x7f93ac0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:01.503 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.499+0000 7f93c37fe700 1 -- 172.21.15.98:0/304622379 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6378+0+0 (secure 0 0 0) 0x7f93bc084400 con 0x7f93cc159560 2023-10-06T05:08:01.503 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.499+0000 7f93cb7fe700 1 --2- 172.21.15.98:0/304622379 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f93ac066f30 0x7f93ac0693e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f93c4002ef0 tx=0x7f93c4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:01.505 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.503+0000 7f93c37fe700 1 -- 172.21.15.98:0/304622379 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f93bc0513d0 con 0x7f93cc159560 2023-10-06T05:08:01.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.726+0000 7f93d222b700 1 -- 172.21.15.98:0/304622379 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f93cc0b17e0 con 0x7f93cc159560 2023-10-06T05:08:01.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.726+0000 7f93c37fe700 1 -- 172.21.15.98:0/304622379 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4603 (secure 0 0 0) 0x7f93bc0549f0 con 0x7f93cc159560 2023-10-06T05:08:01.729 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:08:01.729 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4966,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6834/2320896583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2320896583},{"type":"v1","addr":"172.21.15.174:6834","nonce":2320896583}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":4969,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/1423644921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6833","nonce":1423644921},{"type":"v1","addr":"172.21.15.174:6835","nonce":1423644921}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5100,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/4291295874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":4291295874},{"type":"v1","addr":"172.21.15.98:6836","nonce":4291295874}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-06T05:07:55.873515+0000","modified":"2023-10-06T05:07:59.511013+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5103},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5103":{"gid":5103,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.98:6837/1617154626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1617154626},{"type":"v1","addr":"172.21.15.98:6837","nonce":1617154626}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2023-10-06T05:08:01.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.728+0000 7f93d222b700 1 -- 172.21.15.98:0/304622379 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f93ac066f30 msgr2=0x7f93ac0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:01.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.728+0000 7f93d222b700 1 --2- 172.21.15.98:0/304622379 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f93ac066f30 0x7f93ac0693e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f93c4002ef0 tx=0x7f93c4009040 comp rx=0 tx=0).stop 2023-10-06T05:08:01.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.728+0000 7f93d222b700 1 -- 172.21.15.98:0/304622379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93cc159560 msgr2=0x7f93cc09d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:01.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.728+0000 7f93d222b700 1 --2- 172.21.15.98:0/304622379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93cc159560 0x7f93cc09d410 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f93bc00c5b0 tx=0x7f93bc007f60 comp rx=0 tx=0).stop 2023-10-06T05:08:01.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.728+0000 7f93d222b700 1 -- 172.21.15.98:0/304622379 shutdown_connections 2023-10-06T05:08:01.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.728+0000 7f93d222b700 1 --2- 172.21.15.98:0/304622379 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f93ac066f30 0x7f93ac0693e0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:01.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.728+0000 7f93d222b700 1 --2- 172.21.15.98:0/304622379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f93cc159560 0x7f93cc09d410 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:01.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.728+0000 7f93d222b700 1 --2- 172.21.15.98:0/304622379 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f93cc0b0760 0x7f93cc09a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:01.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.728+0000 7f93d222b700 1 --2- 172.21.15.98:0/304622379 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f93cc0b49a0 0x7f93cc09b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:01.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.728+0000 7f93d222b700 1 -- 172.21.15.98:0/304622379 >> 172.21.15.98:0/304622379 conn(0x7f93cc0997f0 msgr2=0x7f93cc069580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:01.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.728+0000 7f93d222b700 1 -- 172.21.15.98:0/304622379 shutdown_connections 2023-10-06T05:08:01.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:01.728+0000 7f93d222b700 1 -- 172.21.15.98:0/304622379 wait complete. 2023-10-06T05:08:01.732 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 17 2023-10-06T05:08:02.145 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-06T05:07:55.873515+0000', 'modified': '2023-10-06T05:07:59.511013+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5103}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5103': {'gid': 5103, 'name': 'c', 'rank': 0, 'incarnation': 13, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.98:6837/1617154626', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6835', 'nonce': 1617154626}, {'type': 'v1', 'addr': '172.21.15.98:6837', 'nonce': 1617154626}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2023-10-06T05:08:02.145 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-06T05:08:02.145 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-10-06T05:08:02.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.261+0000 7f6b3f46a700 1 Processor -- start 2023-10-06T05:08:02.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.261+0000 7f6b3f46a700 1 -- start start 2023-10-06T05:08:02.267 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.262+0000 7f6b3f46a700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f6b38159440 0x7f6b380b3090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:02.267 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.262+0000 7f6b3f46a700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b380cb7a0 con 0x7f6b380b35d0 2023-10-06T05:08:02.267 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.262+0000 7f6b3f46a700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b380b0860 con 0x7f6b380b2100 2023-10-06T05:08:02.267 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.262+0000 7f6b3f46a700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b380cb150 con 0x7f6b38159440 2023-10-06T05:08:02.267 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.262+0000 7f6b3dc67700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f6b380b35d0 0x7f6b380cbc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:38378/0 (socket says 172.21.15.98:38378) 2023-10-06T05:08:02.267 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.262+0000 7f6b3dc67700 1 -- 172.21.15.98:0/2387237703 learned_addr learned my addr 172.21.15.98:0/2387237703 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:08:02.267 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.262+0000 7f6b3e468700 1 --2- 172.21.15.98:0/2387237703 >> v2:172.21.15.174:3300/0 conn(0x7f6b38159440 0x7f6b380b3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:02.267 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.262+0000 7f6b3d466700 1 -- 172.21.15.98:0/2387237703 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2307570831 0 0) 0x7f6b380cb7a0 con 0x7f6b380b35d0 2023-10-06T05:08:02.268 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.262+0000 7f6b3d466700 1 -- 172.21.15.98:0/2387237703 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b24003570 con 0x7f6b380b35d0 2023-10-06T05:08:02.268 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.263+0000 7f6b3d466700 1 -- 172.21.15.98:0/2387237703 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2428315508 0 0) 0x7f6b24003570 con 0x7f6b380b35d0 2023-10-06T05:08:02.268 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.263+0000 7f6b3d466700 1 -- 172.21.15.98:0/2387237703 >> v2:172.21.15.174:3300/0 conn(0x7f6b38159440 msgr2=0x7f6b380b3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:02.268 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.263+0000 7f6b3d466700 1 --2- 172.21.15.98:0/2387237703 >> v2:172.21.15.174:3300/0 conn(0x7f6b38159440 0x7f6b380b3090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:02.268 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.263+0000 7f6b3d466700 1 -- 172.21.15.98:0/2387237703 >> v1:172.21.15.174:6789/0 conn(0x7f6b380b2100 legacy=0x7f6b380b2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:02.268 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.263+0000 7f6b3d466700 1 -- 172.21.15.98:0/2387237703 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b380cb430 con 0x7f6b380b35d0 2023-10-06T05:08:02.268 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.263+0000 7f6b3d466700 1 -- 172.21.15.98:0/2387237703 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1543991876 0 0) 0x7f6b28001470 con 0x7f6b380b35d0 2023-10-06T05:08:02.268 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.263+0000 7f6b3d466700 1 -- 172.21.15.98:0/2387237703 >> v1:172.21.15.98:6789/0 conn(0x7f6b380b35d0 legacy=0x7f6b380cbc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:02.268 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.263+0000 7f6b3d466700 1 --2- 172.21.15.98:0/2387237703 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6b24003a60 0x7f6b24003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:02.268 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.263+0000 7f6b3d466700 1 --2- 172.21.15.98:0/2387237703 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b24004810 0x7f6b24004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:02.268 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.263+0000 7f6b3d466700 1 --2- 172.21.15.98:0/2387237703 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b240051c0 0x7f6b2400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:02.268 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.263+0000 7f6b3d466700 1 -- 172.21.15.98:0/2387237703 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b380cb150 con 0x7f6b240051c0 2023-10-06T05:08:02.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.263+0000 7f6b3d466700 1 -- 172.21.15.98:0/2387237703 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b2400c160 con 0x7f6b24004810 2023-10-06T05:08:02.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.263+0000 7f6b3d466700 1 -- 172.21.15.98:0/2387237703 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b2400c320 con 0x7f6b24003a60 2023-10-06T05:08:02.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.263+0000 7f6b3ec69700 1 --2- 172.21.15.98:0/2387237703 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b240051c0 0x7f6b2400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:02.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.263+0000 7f6b3e468700 1 --2- 172.21.15.98:0/2387237703 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6b24003a60 0x7f6b24003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:02.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.263+0000 7f6b3dc67700 1 --2- 172.21.15.98:0/2387237703 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b24004810 0x7f6b24004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:02.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.263+0000 7f6b3ec69700 1 -- 172.21.15.98:0/2387237703 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6b24003a60 msgr2=0x7f6b24003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:02.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.263+0000 7f6b3ec69700 1 --2- 172.21.15.98:0/2387237703 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6b24003a60 0x7f6b24003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:02.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.263+0000 7f6b3ec69700 1 -- 172.21.15.98:0/2387237703 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b24004810 msgr2=0x7f6b24004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:02.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.263+0000 7f6b3ec69700 1 --2- 172.21.15.98:0/2387237703 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b24004810 0x7f6b24004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:02.269 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.263+0000 7f6b3ec69700 1 -- 172.21.15.98:0/2387237703 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b24003570 con 0x7f6b240051c0 2023-10-06T05:08:02.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.263+0000 7f6b3ec69700 1 -- 172.21.15.98:0/2387237703 reap_dead start 2023-10-06T05:08:02.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.263+0000 7f6b3ec69700 1 --2- 172.21.15.98:0/2387237703 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b240051c0 0x7f6b2400ba60 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f6b30001ff0 tx=0x7f6b30001670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6588651125269a4e server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:02.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.263+0000 7f6b3d466700 1 -- 172.21.15.98:0/2387237703 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b30001980 con 0x7f6b240051c0 2023-10-06T05:08:02.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.263+0000 7f6b3d466700 1 -- 172.21.15.98:0/2387237703 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b3000c8b0 con 0x7f6b240051c0 2023-10-06T05:08:02.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.263+0000 7f6b3d466700 1 -- 172.21.15.98:0/2387237703 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b3000cc90 con 0x7f6b240051c0 2023-10-06T05:08:02.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.264+0000 7f6b3f46a700 1 -- 172.21.15.98:0/2387237703 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b240051c0 msgr2=0x7f6b2400ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:02.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.264+0000 7f6b3f46a700 1 --2- 172.21.15.98:0/2387237703 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b240051c0 0x7f6b2400ba60 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f6b30001ff0 tx=0x7f6b30001670 comp rx=0 tx=0).stop 2023-10-06T05:08:02.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.264+0000 7f6b3f46a700 1 -- 172.21.15.98:0/2387237703 shutdown_connections 2023-10-06T05:08:02.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.264+0000 7f6b3f46a700 1 --2- 172.21.15.98:0/2387237703 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b240051c0 0x7f6b2400ba60 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:02.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.264+0000 7f6b3f46a700 1 -- 172.21.15.98:0/2387237703 >> 172.21.15.98:0/2387237703 conn(0x7f6b3806ece0 msgr2=0x7f6b3806f0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:02.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.264+0000 7f6b3f46a700 1 -- 172.21.15.98:0/2387237703 shutdown_connections 2023-10-06T05:08:02.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.264+0000 7f6b3f46a700 1 -- 172.21.15.98:0/2387237703 wait complete. 2023-10-06T05:08:02.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.265+0000 7f6b3f46a700 1 Processor -- start 2023-10-06T05:08:02.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.265+0000 7f6b3f46a700 1 -- start start 2023-10-06T05:08:02.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.266+0000 7f6b3f46a700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b240051c0 0x7f6b380c63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:02.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.266+0000 7f6b3f46a700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6b380c4570 0x7f6b380af050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:02.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.266+0000 7f6b3f46a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b380cddf0 0x7f6b380ce290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:02.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.266+0000 7f6b3f46a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b380ce7d0 con 0x7f6b380cddf0 2023-10-06T05:08:02.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.266+0000 7f6b3f46a700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b38154f50 con 0x7f6b240051c0 2023-10-06T05:08:02.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.266+0000 7f6b3f46a700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b380c4b40 con 0x7f6b380c4570 2023-10-06T05:08:02.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.266+0000 7f6b3ec69700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b380cddf0 0x7f6b380ce290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:02.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.266+0000 7f6b3ec69700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b380cddf0 0x7f6b380ce290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54846/0 (socket says 172.21.15.98:54846) 2023-10-06T05:08:02.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.266+0000 7f6b3ec69700 1 -- 172.21.15.98:0/328503858 learned_addr learned my addr 172.21.15.98:0/328503858 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:08:02.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.266+0000 7f6b3dc67700 1 --2- 172.21.15.98:0/328503858 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6b380c4570 0x7f6b380af050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:02.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.266+0000 7f6b3e468700 1 --2- 172.21.15.98:0/328503858 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b240051c0 0x7f6b380c63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:02.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.266+0000 7f6b3ec69700 1 -- 172.21.15.98:0/328503858 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6b380c4570 msgr2=0x7f6b380af050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:02.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.266+0000 7f6b3ec69700 1 --2- 172.21.15.98:0/328503858 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6b380c4570 0x7f6b380af050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:02.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.266+0000 7f6b3ec69700 1 -- 172.21.15.98:0/328503858 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b240051c0 msgr2=0x7f6b380c63d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:02.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.266+0000 7f6b3ec69700 1 --2- 172.21.15.98:0/328503858 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b240051c0 0x7f6b380c63d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:02.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.266+0000 7f6b3ec69700 1 -- 172.21.15.98:0/328503858 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b380c4e20 con 0x7f6b380cddf0 2023-10-06T05:08:02.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.266+0000 7f6b3dc67700 1 --2- 172.21.15.98:0/328503858 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6b380c4570 0x7f6b380af050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:08:02.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.266+0000 7f6b3ec69700 1 --2- 172.21.15.98:0/328503858 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b380cddf0 0x7f6b380ce290 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f6b3000c6a0 tx=0x7f6b30015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:02.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.266+0000 7f6b2e7fc700 1 -- 172.21.15.98:0/328503858 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b30001980 con 0x7f6b380cddf0 2023-10-06T05:08:02.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.267+0000 7f6b2e7fc700 1 -- 172.21.15.98:0/328503858 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b3000c6d0 con 0x7f6b380cddf0 2023-10-06T05:08:02.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.267+0000 7f6b3f46a700 1 -- 172.21.15.98:0/328503858 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b380c5100 con 0x7f6b380cddf0 2023-10-06T05:08:02.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.267+0000 7f6b2e7fc700 1 -- 172.21.15.98:0/328503858 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b3001e870 con 0x7f6b380cddf0 2023-10-06T05:08:02.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.267+0000 7f6b3f46a700 1 -- 172.21.15.98:0/328503858 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b380c6910 con 0x7f6b380cddf0 2023-10-06T05:08:02.278 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.268+0000 7f6b2e7fc700 1 -- 172.21.15.98:0/328503858 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f6b3001ebe0 con 0x7f6b380cddf0 2023-10-06T05:08:02.278 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.268+0000 7f6b2e7fc700 1 --2- 172.21.15.98:0/328503858 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f6b1c066f30 0x7f6b1c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:02.278 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.268+0000 7f6b2e7fc700 1 -- 172.21.15.98:0/328503858 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6378+0+0 (secure 0 0 0) 0x7f6b30086360 con 0x7f6b380cddf0 2023-10-06T05:08:02.278 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.268+0000 7f6b3e468700 1 --2- 172.21.15.98:0/328503858 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f6b1c066f30 0x7f6b1c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:02.278 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.269+0000 7f6b3e468700 1 --2- 172.21.15.98:0/328503858 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f6b1c066f30 0x7f6b1c0693e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f6b34006ed0 tx=0x7f6b34008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:02.278 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.270+0000 7f6b499c4b80 1 Processor -- start 2023-10-06T05:08:02.278 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.270+0000 7f6b499c4b80 1 -- start start 2023-10-06T05:08:02.278 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.271+0000 7f6b499c4b80 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x5638b9db0950 0x5638b9db0d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:02.278 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.271+0000 7f6b499c4b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638b9dfabc0 0x5638b9dd9920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:02.278 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.271+0000 7f6b499c4b80 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5638b9dd9e60 0x5638b9dac990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:02.278 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.271+0000 7f6b499c4b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638b9d30c80 con 0x5638b9dfabc0 2023-10-06T05:08:02.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.271+0000 7f6b499c4b80 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638b9d07a70 con 0x5638b9dd9e60 2023-10-06T05:08:02.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.271+0000 7f6b499c4b80 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x5638b9d05270 con 0x5638b9db0950 2023-10-06T05:08:02.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.271+0000 7f6b3dc67700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638b9dfabc0 0x5638b9dd9920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:02.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.271+0000 7f6b3dc67700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638b9dfabc0 0x5638b9dd9920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54858/0 (socket says 172.21.15.98:54858) 2023-10-06T05:08:02.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.271+0000 7f6b3dc67700 1 -- 172.21.15.98:0/3390199270 learned_addr learned my addr 172.21.15.98:0/3390199270 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:08:02.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.271+0000 7f6b3e468700 1 --2- 172.21.15.98:0/3390199270 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x5638b9db0950 0x5638b9db0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:02.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.271+0000 7f6b3ec69700 1 --2- 172.21.15.98:0/3390199270 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5638b9dd9e60 0x5638b9dac990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:02.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.271+0000 7f6b3dc67700 1 -- 172.21.15.98:0/3390199270 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x5638b9db0950 msgr2=0x5638b9db0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:02.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.271+0000 7f6b3dc67700 1 --2- 172.21.15.98:0/3390199270 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x5638b9db0950 0x5638b9db0d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:02.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.272+0000 7f6b3dc67700 1 -- 172.21.15.98:0/3390199270 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5638b9dd9e60 msgr2=0x5638b9dac990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:02.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.272+0000 7f6b3dc67700 1 --2- 172.21.15.98:0/3390199270 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5638b9dd9e60 0x5638b9dac990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:02.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.272+0000 7f6b3dc67700 1 -- 172.21.15.98:0/3390199270 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5638b9b9b270 con 0x5638b9dfabc0 2023-10-06T05:08:02.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.272+0000 7f6b3ec69700 1 --2- 172.21.15.98:0/3390199270 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5638b9dd9e60 0x5638b9dac990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:08:02.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.272+0000 7f6b3dc67700 1 --2- 172.21.15.98:0/3390199270 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638b9dfabc0 0x5638b9dd9920 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f6b28007c70 tx=0x7f6b28009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3c9ce1f9cd268cd server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:02.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.272+0000 7f6b1b7fe700 1 -- 172.21.15.98:0/3390199270 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b2801a070 con 0x5638b9dfabc0 2023-10-06T05:08:02.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.272+0000 7f6b1b7fe700 1 -- 172.21.15.98:0/3390199270 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b28007390 con 0x5638b9dfabc0 2023-10-06T05:08:02.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.272+0000 7f6b1b7fe700 1 -- 172.21.15.98:0/3390199270 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b28007710 con 0x5638b9dfabc0 2023-10-06T05:08:02.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.272+0000 7f6b499c4b80 1 -- 172.21.15.98:0/3390199270 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638b9dfabc0 msgr2=0x5638b9dd9920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:02.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.272+0000 7f6b499c4b80 1 --2- 172.21.15.98:0/3390199270 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638b9dfabc0 0x5638b9dd9920 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f6b28007c70 tx=0x7f6b28009040 comp rx=0 tx=0).stop 2023-10-06T05:08:02.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.272+0000 7f6b499c4b80 1 -- 172.21.15.98:0/3390199270 shutdown_connections 2023-10-06T05:08:02.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.272+0000 7f6b499c4b80 1 --2- 172.21.15.98:0/3390199270 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5638b9dd9e60 0x5638b9dac990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:02.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.272+0000 7f6b499c4b80 1 --2- 172.21.15.98:0/3390199270 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x5638b9db0950 0x5638b9db0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:02.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.272+0000 7f6b499c4b80 1 --2- 172.21.15.98:0/3390199270 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638b9dfabc0 0x5638b9dd9920 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:02.280 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.272+0000 7f6b499c4b80 1 -- 172.21.15.98:0/3390199270 >> 172.21.15.98:0/3390199270 conn(0x5638b9db2270 msgr2=0x5638b9db2640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:02.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.273+0000 7f6b499c4b80 1 -- 172.21.15.98:0/3390199270 shutdown_connections 2023-10-06T05:08:02.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.273+0000 7f6b499c4b80 1 -- 172.21.15.98:0/3390199270 wait complete. 2023-10-06T05:08:02.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.273+0000 7f6b499c4b80 1 -- start start 2023-10-06T05:08:02.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.273+0000 7f6b499c4b80 1 Processor -- start 2023-10-06T05:08:02.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.274+0000 7f6b499c4b80 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5638b9db0950 0x5638b9dd6320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:02.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.274+0000 7f6b499c4b80 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x5638b9dd9e60 0x5638b9e078d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:02.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.274+0000 7f6b499c4b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638b9dfabc0 0x5638b9dfb4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:02.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.274+0000 7f6b499c4b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638b9d01060 con 0x5638b9dfabc0 2023-10-06T05:08:02.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.274+0000 7f6b499c4b80 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638b9cac890 con 0x5638b9db0950 2023-10-06T05:08:02.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.274+0000 7f6b499c4b80 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x5638b9cc7ea0 con 0x5638b9dd9e60 2023-10-06T05:08:02.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.274+0000 7f6b499c4b80 10 client.0 _ll_register_callbacks cb 0x5638b9b9df30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-06T05:08:02.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.274+0000 7f6b3ec69700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638b9dfabc0 0x5638b9dfb4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:02.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.274+0000 7f6b3ec69700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638b9dfabc0 0x5638b9dfb4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54864/0 (socket says 172.21.15.98:54864) 2023-10-06T05:08:02.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.274+0000 7f6b3ec69700 1 -- 172.21.15.98:0/3107195127 learned_addr learned my addr 172.21.15.98:0/3107195127 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:08:02.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.274+0000 7f6b3dc67700 1 --2- 172.21.15.98:0/3107195127 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x5638b9dd9e60 0x5638b9e078d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:02.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.274+0000 7f6b3e468700 1 --2- 172.21.15.98:0/3107195127 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5638b9db0950 0x5638b9dd6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:02.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.274+0000 7f6b3ec69700 1 -- 172.21.15.98:0/3107195127 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x5638b9dd9e60 msgr2=0x5638b9e078d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:02.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.274+0000 7f6b3ec69700 1 --2- 172.21.15.98:0/3107195127 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x5638b9dd9e60 0x5638b9e078d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:02.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.274+0000 7f6b3ec69700 1 -- 172.21.15.98:0/3107195127 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5638b9db0950 msgr2=0x5638b9dd6320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:02.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.274+0000 7f6b3ec69700 1 --2- 172.21.15.98:0/3107195127 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5638b9db0950 0x5638b9dd6320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:02.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.274+0000 7f6b3ec69700 1 -- 172.21.15.98:0/3107195127 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b380cb430 con 0x5638b9dfabc0 2023-10-06T05:08:02.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.275+0000 7f6b3ec69700 1 --2- 172.21.15.98:0/3107195127 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638b9dfabc0 0x5638b9dfb4c0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f6b30084610 tx=0x7f6b30084540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=971cbabce1df182e server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:02.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.275+0000 7f6b18ff9700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2023-10-06T05:08:02.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.275+0000 7f6b18ff9700 1 -- 172.21.15.98:0/3107195127 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b30053330 con 0x5638b9dfabc0 2023-10-06T05:08:02.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.275+0000 7f6b18ff9700 1 -- 172.21.15.98:0/3107195127 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(43..43 src has 1..43) v4 ==== 6378+0+0 (secure 0 0 0) 0x7f6b300576f0 con 0x5638b9dfabc0 2023-10-06T05:08:02.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.275+0000 7f6b499c4b80 1 -- 172.21.15.98:0/3107195127 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5638b9da5690 con 0x5638b9dfabc0 2023-10-06T05:08:02.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.275+0000 7f6b18ff9700 1 -- 172.21.15.98:0/3107195127 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6b30056ff0 con 0x5638b9dfabc0 2023-10-06T05:08:02.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.275+0000 7f6b499c4b80 10 client.5298 fetch_fsmap learned FSMap version 17 2023-10-06T05:08:02.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.275+0000 7f6b499c4b80 1 -- 172.21.15.98:0/3107195127 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x5638b9b3dfe0 con 0x5638b9dfabc0 2023-10-06T05:08:02.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.275+0000 7f6b18ff9700 1 -- 172.21.15.98:0/3107195127 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5638b9da5690 con 0x5638b9dfabc0 2023-10-06T05:08:02.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.275+0000 7f6b18ff9700 1 -- 172.21.15.98:0/3107195127 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 17) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f6b3007ed20 con 0x5638b9dfabc0 2023-10-06T05:08:02.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.275+0000 7f6b499c4b80 10 client.5298 fetch_fsmap finished waiting for FSMap version 17 2023-10-06T05:08:02.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.275+0000 7f6b499c4b80 10 client.5298 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-06T05:08:02.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.275+0000 7f6b499c4b80 20 client.5298 populate_metadata read hostname 'smithi098' 2023-10-06T05:08:02.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.275+0000 7f6b499c4b80 1 --2- 172.21.15.98:0/3107195127 >> [v2:172.21.15.98:6835/1617154626,v1:172.21.15.98:6837/1617154626] conn(0x5638b9e094f0 0x5638b9e098c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:02.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.275+0000 7f6b499c4b80 4 client.5298 mds_command: new command op to 5103 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-06T05:08:02.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.275+0000 7f6b499c4b80 1 -- 172.21.15.98:0/3107195127 --> [v2:172.21.15.98:6835/1617154626,v1:172.21.15.98:6837/1617154626] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5638b9d825c0 con 0x5638b9e094f0 2023-10-06T05:08:02.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.275+0000 7f6b3e468700 1 --2- 172.21.15.98:0/3107195127 >> [v2:172.21.15.98:6835/1617154626,v1:172.21.15.98:6837/1617154626] conn(0x5638b9e094f0 0x5638b9e098c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:02.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.276+0000 7f6b18ff9700 10 client.5298 ms_handle_connect on v2:172.21.15.98:6835/1617154626 2023-10-06T05:08:02.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.276+0000 7f6b3e468700 1 --2- 172.21.15.98:0/3107195127 >> [v2:172.21.15.98:6835/1617154626,v1:172.21.15.98:6837/1617154626] conn(0x5638b9e094f0 0x5638b9e098c0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=703f0e20b67902d server_cookie=499209097ed7cb4f in_seq=0 out_seq=0 2023-10-06T05:08:02.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.276+0000 7f6b18ff9700 1 -- 172.21.15.98:0/3107195127 <== mds.0 v2:172.21.15.98:6835/1617154626 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+16954 (crc 0 0 0) 0x5638b9d825c0 con 0x5638b9e094f0 2023-10-06T05:08:02.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.276+0000 7f6b18ff9700 10 client.5298 handle_command_reply: tid=0 2023-10-06T05:08:02.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.276+0000 7f6b499c4b80 1 client.5298 shutdown 2023-10-06T05:08:02.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.276+0000 7f6b499c4b80 1 -- 172.21.15.98:0/3107195127 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638b9dfabc0 msgr2=0x5638b9dfb4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:02.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.276+0000 7f6b499c4b80 1 --2- 172.21.15.98:0/3107195127 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638b9dfabc0 0x5638b9dfb4c0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f6b30084610 tx=0x7f6b30084540 comp rx=0 tx=0).stop 2023-10-06T05:08:02.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.276+0000 7f6b499c4b80 1 -- 172.21.15.98:0/3107195127 shutdown_connections 2023-10-06T05:08:02.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.276+0000 7f6b499c4b80 1 --2- 172.21.15.98:0/3107195127 >> [v2:172.21.15.98:6835/1617154626,v1:172.21.15.98:6837/1617154626] conn(0x5638b9e094f0 0x5638b9e098c0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:02.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.276+0000 7f6b499c4b80 1 --2- 172.21.15.98:0/3107195127 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638b9dfabc0 0x5638b9dfb4c0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:02.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.276+0000 7f6b499c4b80 1 --2- 172.21.15.98:0/3107195127 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5638b9db0950 0x5638b9dd6320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:02.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.276+0000 7f6b18ff9700 0 client.5298 ms_handle_reset on v2:172.21.15.98:6835/1617154626 2023-10-06T05:08:02.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.276+0000 7f6b499c4b80 1 --2- 172.21.15.98:0/3107195127 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x5638b9dd9e60 0x5638b9e078d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:02.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.276+0000 7f6b499c4b80 1 -- 172.21.15.98:0/3107195127 >> 172.21.15.98:0/3107195127 conn(0x5638b9db2270 msgr2=0x5638b9da71a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:02.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.276+0000 7f6b499c4b80 1 -- 172.21.15.98:0/3107195127 shutdown_connections 2023-10-06T05:08:02.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.276+0000 7f6b499c4b80 1 -- 172.21.15.98:0/3107195127 wait complete. 2023-10-06T05:08:02.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.277+0000 7f6b499c4b80 20 client.5298 trim_cache size 0 max 16384 2023-10-06T05:08:02.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.293+0000 7f6b499c4b80 1 Processor -- start 2023-10-06T05:08:02.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.293+0000 7f6b499c4b80 1 -- start start 2023-10-06T05:08:02.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.294+0000 7f6b499c4b80 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x5638b9db0950 0x5638b9e11900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:02.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.294+0000 7f6b499c4b80 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5638b9dd9e60 0x5638b9e11e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:02.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.294+0000 7f6b499c4b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638b9dfabc0 0x5638b9de0260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:02.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.294+0000 7f6b499c4b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638b9db1a90 con 0x5638b9dfabc0 2023-10-06T05:08:02.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.294+0000 7f6b499c4b80 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638b9b83e30 con 0x5638b9dd9e60 2023-10-06T05:08:02.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.294+0000 7f6b499c4b80 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x5638b9defe60 con 0x5638b9db0950 2023-10-06T05:08:02.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.294+0000 7f6b3ec69700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638b9dfabc0 0x5638b9de0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:02.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.294+0000 7f6b3ec69700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638b9dfabc0 0x5638b9de0260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54880/0 (socket says 172.21.15.98:54880) 2023-10-06T05:08:02.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.294+0000 7f6b3ec69700 1 -- 172.21.15.98:0/4278664289 learned_addr learned my addr 172.21.15.98:0/4278664289 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:08:02.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.294+0000 7f6b3e468700 1 --2- 172.21.15.98:0/4278664289 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x5638b9db0950 0x5638b9e11900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:02.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.294+0000 7f6b3ec69700 1 -- 172.21.15.98:0/4278664289 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x5638b9db0950 msgr2=0x5638b9e11900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:02.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.294+0000 7f6b3ec69700 1 --2- 172.21.15.98:0/4278664289 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x5638b9db0950 0x5638b9e11900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:02.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.294+0000 7f6b3ec69700 1 -- 172.21.15.98:0/4278664289 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5638b9dd9e60 msgr2=0x5638b9e11e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:02.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.294+0000 7f6b3dc67700 1 --2- 172.21.15.98:0/4278664289 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5638b9dd9e60 0x5638b9e11e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:02.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.294+0000 7f6b3ec69700 1 --2- 172.21.15.98:0/4278664289 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5638b9dd9e60 0x5638b9e11e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:02.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.294+0000 7f6b3ec69700 1 -- 172.21.15.98:0/4278664289 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5638b9ccf780 con 0x5638b9dfabc0 2023-10-06T05:08:02.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.294+0000 7f6b3dc67700 1 --2- 172.21.15.98:0/4278664289 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5638b9dd9e60 0x5638b9e11e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:08:02.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.294+0000 7f6b3ec69700 1 --2- 172.21.15.98:0/4278664289 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638b9dfabc0 0x5638b9de0260 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f6b30056f80 tx=0x7f6b3002a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90da42268ef332c1 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:02.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.295+0000 7f6b1bfff700 1 -- 172.21.15.98:0/4278664289 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b30053330 con 0x5638b9dfabc0 2023-10-06T05:08:02.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.295+0000 7f6b1bfff700 1 -- 172.21.15.98:0/4278664289 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b3001ee90 con 0x5638b9dfabc0 2023-10-06T05:08:02.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.295+0000 7f6b1bfff700 1 -- 172.21.15.98:0/4278664289 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b300576f0 con 0x5638b9dfabc0 2023-10-06T05:08:02.300 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.295+0000 7f6b499c4b80 1 -- 172.21.15.98:0/4278664289 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638b9dfabc0 msgr2=0x5638b9de0260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:02.300 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.295+0000 7f6b499c4b80 1 --2- 172.21.15.98:0/4278664289 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638b9dfabc0 0x5638b9de0260 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f6b30056f80 tx=0x7f6b3002a000 comp rx=0 tx=0).stop 2023-10-06T05:08:02.300 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.295+0000 7f6b499c4b80 1 -- 172.21.15.98:0/4278664289 shutdown_connections 2023-10-06T05:08:02.300 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.295+0000 7f6b499c4b80 1 --2- 172.21.15.98:0/4278664289 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638b9dfabc0 0x5638b9de0260 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:02.300 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.295+0000 7f6b499c4b80 1 --2- 172.21.15.98:0/4278664289 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x5638b9db0950 0x5638b9e11900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:02.300 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.295+0000 7f6b499c4b80 1 --2- 172.21.15.98:0/4278664289 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5638b9dd9e60 0x5638b9e11e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:02.300 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.295+0000 7f6b499c4b80 1 -- 172.21.15.98:0/4278664289 >> 172.21.15.98:0/4278664289 conn(0x5638b9db2270 msgr2=0x5638b9db3cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:02.300 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.295+0000 7f6b499c4b80 1 -- 172.21.15.98:0/4278664289 shutdown_connections 2023-10-06T05:08:02.300 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.295+0000 7f6b499c4b80 1 -- 172.21.15.98:0/4278664289 wait complete. 2023-10-06T05:08:02.300 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.296+0000 7f6b499c4b80 1 -- start start 2023-10-06T05:08:02.300 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.296+0000 7f6b499c4b80 1 Processor -- start 2023-10-06T05:08:02.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.297+0000 7f6b499c4b80 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x5638b9db0950 0x5638b9e5bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:02.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.297+0000 7f6b499c4b80 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5638b9dd9e60 0x5638b9e11900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:02.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.297+0000 7f6b499c4b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638b9dfabc0 0x5638b9e11e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:02.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.297+0000 7f6b499c4b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638b9e03930 con 0x5638b9dfabc0 2023-10-06T05:08:02.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.297+0000 7f6b499c4b80 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638b9deeff0 con 0x5638b9dd9e60 2023-10-06T05:08:02.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.297+0000 7f6b499c4b80 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x5638b9da7e50 con 0x5638b9db0950 2023-10-06T05:08:02.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.297+0000 7f6b499c4b80 10 client.0 _ll_register_callbacks cb 0x5638b9b9df30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-06T05:08:02.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.297+0000 7f6b3ec69700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638b9dfabc0 0x5638b9e11e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:02.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.297+0000 7f6b3ec69700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638b9dfabc0 0x5638b9e11e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54884/0 (socket says 172.21.15.98:54884) 2023-10-06T05:08:02.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.297+0000 7f6b3ec69700 1 -- 172.21.15.98:0/2454610575 learned_addr learned my addr 172.21.15.98:0/2454610575 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:08:02.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.297+0000 7f6b3e468700 1 --2- 172.21.15.98:0/2454610575 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x5638b9db0950 0x5638b9e5bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:02.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.297+0000 7f6b3dc67700 1 --2- 172.21.15.98:0/2454610575 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5638b9dd9e60 0x5638b9e11900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:02.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.297+0000 7f6b3ec69700 1 -- 172.21.15.98:0/2454610575 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x5638b9db0950 msgr2=0x5638b9e5bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:02.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.297+0000 7f6b3ec69700 1 --2- 172.21.15.98:0/2454610575 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x5638b9db0950 0x5638b9e5bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:02.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.297+0000 7f6b3ec69700 1 -- 172.21.15.98:0/2454610575 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5638b9dd9e60 msgr2=0x5638b9e11900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:02.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.297+0000 7f6b3ec69700 1 --2- 172.21.15.98:0/2454610575 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5638b9dd9e60 0x5638b9e11900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:02.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.297+0000 7f6b3ec69700 1 -- 172.21.15.98:0/2454610575 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b380cb430 con 0x5638b9dfabc0 2023-10-06T05:08:02.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.297+0000 7f6b3ec69700 1 --2- 172.21.15.98:0/2454610575 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638b9dfabc0 0x5638b9e11e40 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f6b30056f80 tx=0x7f6b3006e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52c8f5e743ede6ef server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:02.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.297+0000 7f6b18ff9700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2023-10-06T05:08:02.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.297+0000 7f6b18ff9700 1 -- 172.21.15.98:0/2454610575 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b30007d70 con 0x5638b9dfabc0 2023-10-06T05:08:02.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.297+0000 7f6b18ff9700 1 -- 172.21.15.98:0/2454610575 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(43..43 src has 1..43) v4 ==== 6378+0+0 (secure 0 0 0) 0x7f6b300576f0 con 0x5638b9dfabc0 2023-10-06T05:08:02.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.297+0000 7f6b499c4b80 1 -- 172.21.15.98:0/2454610575 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5638b9dae180 con 0x5638b9dfabc0 2023-10-06T05:08:02.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.297+0000 7f6b18ff9700 1 -- 172.21.15.98:0/2454610575 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6b30072e70 con 0x5638b9dfabc0 2023-10-06T05:08:02.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.297+0000 7f6b499c4b80 10 client.5304 fetch_fsmap learned FSMap version 17 2023-10-06T05:08:02.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.297+0000 7f6b499c4b80 1 -- 172.21.15.98:0/2454610575 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x5638b9cb6470 con 0x5638b9dfabc0 2023-10-06T05:08:02.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.298+0000 7f6b18ff9700 1 -- 172.21.15.98:0/2454610575 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5638b9dae180 con 0x5638b9dfabc0 2023-10-06T05:08:02.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.298+0000 7f6b18ff9700 1 -- 172.21.15.98:0/2454610575 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 17) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f6b3000c0e0 con 0x5638b9dfabc0 2023-10-06T05:08:02.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.298+0000 7f6b499c4b80 10 client.5304 fetch_fsmap finished waiting for FSMap version 17 2023-10-06T05:08:02.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.298+0000 7f6b499c4b80 10 client.5304 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-06T05:08:02.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.298+0000 7f6b499c4b80 20 client.5304 populate_metadata read hostname 'smithi098' 2023-10-06T05:08:02.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.298+0000 7f6b499c4b80 1 --2- 172.21.15.98:0/2454610575 >> [v2:172.21.15.98:6835/1617154626,v1:172.21.15.98:6837/1617154626] conn(0x5638b9db3d20 0x5638b9e070a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:02.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.298+0000 7f6b499c4b80 4 client.5304 mds_command: new command op to 5103 tid=0[{"prefix": "status"}] 2023-10-06T05:08:02.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.298+0000 7f6b499c4b80 1 -- 172.21.15.98:0/2454610575 --> [v2:172.21.15.98:6835/1617154626,v1:172.21.15.98:6837/1617154626] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f6b28007390 con 0x5638b9db3d20 2023-10-06T05:08:02.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.298+0000 7f6b3e468700 1 --2- 172.21.15.98:0/2454610575 >> [v2:172.21.15.98:6835/1617154626,v1:172.21.15.98:6837/1617154626] conn(0x5638b9db3d20 0x5638b9e070a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:02.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.298+0000 7f6b18ff9700 10 client.5304 ms_handle_connect on v2:172.21.15.98:6835/1617154626 2023-10-06T05:08:02.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.298+0000 7f6b3e468700 1 --2- 172.21.15.98:0/2454610575 >> [v2:172.21.15.98:6835/1617154626,v1:172.21.15.98:6837/1617154626] conn(0x5638b9db3d20 0x5638b9e070a0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c30d4ecc312e8388 server_cookie=f9acfda2b1cf8c5c in_seq=0 out_seq=0 2023-10-06T05:08:02.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.298+0000 7f6b18ff9700 1 -- 172.21.15.98:0/2454610575 <== mds.0 v2:172.21.15.98:6835/1617154626 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x7f6b28007390 con 0x5638b9db3d20 2023-10-06T05:08:02.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.298+0000 7f6b18ff9700 10 client.5304 handle_command_reply: tid=0 2023-10-06T05:08:02.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.298+0000 7f6b499c4b80 1 client.5304 shutdown 2023-10-06T05:08:02.307 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.298+0000 7f6b499c4b80 1 -- 172.21.15.98:0/2454610575 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638b9dfabc0 msgr2=0x5638b9e11e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:02.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.298+0000 7f6b499c4b80 1 --2- 172.21.15.98:0/2454610575 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638b9dfabc0 0x5638b9e11e40 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f6b30056f80 tx=0x7f6b3006e000 comp rx=0 tx=0).stop 2023-10-06T05:08:02.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.298+0000 7f6b499c4b80 1 -- 172.21.15.98:0/2454610575 shutdown_connections 2023-10-06T05:08:02.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.298+0000 7f6b499c4b80 1 --2- 172.21.15.98:0/2454610575 >> [v2:172.21.15.98:6835/1617154626,v1:172.21.15.98:6837/1617154626] conn(0x5638b9db3d20 0x5638b9e070a0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:02.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.298+0000 7f6b499c4b80 1 --2- 172.21.15.98:0/2454610575 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638b9dfabc0 0x5638b9e11e40 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:02.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.298+0000 7f6b499c4b80 1 --2- 172.21.15.98:0/2454610575 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x5638b9db0950 0x5638b9e5bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:02.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.298+0000 7f6b18ff9700 0 client.5304 ms_handle_reset on v2:172.21.15.98:6835/1617154626 2023-10-06T05:08:02.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.298+0000 7f6b499c4b80 1 --2- 172.21.15.98:0/2454610575 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5638b9dd9e60 0x5638b9e11900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:02.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.298+0000 7f6b499c4b80 1 -- 172.21.15.98:0/2454610575 >> 172.21.15.98:0/2454610575 conn(0x5638b9db2270 msgr2=0x5638b9db2640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:02.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.299+0000 7f6b499c4b80 1 -- 172.21.15.98:0/2454610575 shutdown_connections 2023-10-06T05:08:02.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.299+0000 7f6b499c4b80 1 -- 172.21.15.98:0/2454610575 wait complete. 2023-10-06T05:08:02.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.299+0000 7f6b499c4b80 20 client.5304 trim_cache size 0 max 16384 2023-10-06T05:08:02.309 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-10-06T05:08:02.309 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "f50c5df5-cad5-47a4-9bb1-8c791d4311ec", 2023-10-06T05:08:02.309 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 0, 2023-10-06T05:08:02.309 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 5103, 2023-10-06T05:08:02.309 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2023-10-06T05:08:02.309 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2023-10-06T05:08:02.309 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2023-10-06T05:08:02.309 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 5.8169047569999996, 2023-10-06T05:08:02.309 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 17, 2023-10-06T05:08:02.310 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 43, 2023-10-06T05:08:02.310 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 43, 2023-10-06T05:08:02.310 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 13.918782838 2023-10-06T05:08:02.310 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-10-06T05:08:02.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.299+0000 7f6b3f46a700 1 -- 172.21.15.98:0/328503858 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f6b1c066f30 msgr2=0x7f6b1c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:02.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.299+0000 7f6b3f46a700 1 --2- 172.21.15.98:0/328503858 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f6b1c066f30 0x7f6b1c0693e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f6b34006ed0 tx=0x7f6b34008040 comp rx=0 tx=0).stop 2023-10-06T05:08:02.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.299+0000 7f6b3f46a700 1 -- 172.21.15.98:0/328503858 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b380cddf0 msgr2=0x7f6b380ce290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:02.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.299+0000 7f6b3f46a700 1 --2- 172.21.15.98:0/328503858 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b380cddf0 0x7f6b380ce290 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f6b3000c6a0 tx=0x7f6b30015000 comp rx=0 tx=0).stop 2023-10-06T05:08:02.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.299+0000 7f6b3f46a700 1 -- 172.21.15.98:0/328503858 shutdown_connections 2023-10-06T05:08:02.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.299+0000 7f6b3f46a700 1 --2- 172.21.15.98:0/328503858 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f6b1c066f30 0x7f6b1c0693e0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:02.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.299+0000 7f6b3f46a700 1 --2- 172.21.15.98:0/328503858 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b380cddf0 0x7f6b380ce290 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:02.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.299+0000 7f6b3f46a700 1 --2- 172.21.15.98:0/328503858 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b240051c0 0x7f6b380c63d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:02.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.299+0000 7f6b3f46a700 1 --2- 172.21.15.98:0/328503858 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6b380c4570 0x7f6b380af050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:02.312 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.299+0000 7f6b3f46a700 1 -- 172.21.15.98:0/328503858 >> 172.21.15.98:0/328503858 conn(0x7f6b3806ece0 msgr2=0x7f6b380b5900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:02.312 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.299+0000 7f6b3f46a700 1 -- 172.21.15.98:0/328503858 shutdown_connections 2023-10-06T05:08:02.312 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.299+0000 7f6b3f46a700 1 -- 172.21.15.98:0/328503858 wait complete. 2023-10-06T05:08:02.313 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-06T05:08:02.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.424+0000 7fb57f883700 1 Processor -- start 2023-10-06T05:08:02.427 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.424+0000 7fb57f883700 1 -- start start 2023-10-06T05:08:02.429 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.425+0000 7fb57f883700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb5780ca370 0x7fb578158270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:02.429 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.425+0000 7fb57f883700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fb5780ce8a0 0x7fb57815aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:02.429 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.425+0000 7fb57f883700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb578157de0 con 0x7fb5780adb60 2023-10-06T05:08:02.429 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.425+0000 7fb57f883700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5780cbb90 con 0x7fb5780ca370 2023-10-06T05:08:02.429 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.425+0000 7fb57f883700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5780cb5d0 con 0x7fb5780ce8a0 2023-10-06T05:08:02.429 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.425+0000 7fb57d61f700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb5780ca370 0x7fb578158270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:02.429 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.425+0000 7fb57d61f700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb5780ca370 0x7fb578158270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54900/0 (socket says 172.21.15.98:54900) 2023-10-06T05:08:02.429 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.425+0000 7fb57d61f700 1 -- 172.21.15.98:0/2034212571 learned_addr learned my addr 172.21.15.98:0/2034212571 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:08:02.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.426+0000 7fb57de20700 1 --2- 172.21.15.98:0/2034212571 >> v2:172.21.15.174:3300/0 conn(0x7fb5780ce8a0 0x7fb57815aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:02.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.426+0000 7fb57d61f700 1 -- 172.21.15.98:0/2034212571 >> v2:172.21.15.174:3300/0 conn(0x7fb5780ce8a0 msgr2=0x7fb57815aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:02.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.426+0000 7fb57d61f700 1 --2- 172.21.15.98:0/2034212571 >> v2:172.21.15.174:3300/0 conn(0x7fb5780ce8a0 0x7fb57815aed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:02.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.426+0000 7fb57d61f700 1 -- 172.21.15.98:0/2034212571 >> v1:172.21.15.174:6789/0 conn(0x7fb5780adb60 legacy=0x7fb5780ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:02.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.426+0000 7fb57d61f700 1 -- 172.21.15.98:0/2034212571 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5780cb8b0 con 0x7fb5780ca370 2023-10-06T05:08:02.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.426+0000 7fb57d61f700 1 --2- 172.21.15.98:0/2034212571 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5780ca370 0x7fb578158270 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fb5781554c0 tx=0x7fb574009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7745a2071c46e72 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:02.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.426+0000 7fb56ffff700 1 -- 172.21.15.98:0/2034212571 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb57400e070 con 0x7fb5780ca370 2023-10-06T05:08:02.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.426+0000 7fb56ffff700 1 -- 172.21.15.98:0/2034212571 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb57400ba10 con 0x7fb5780ca370 2023-10-06T05:08:02.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.426+0000 7fb56ffff700 1 -- 172.21.15.98:0/2034212571 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5740092e0 con 0x7fb5780ca370 2023-10-06T05:08:02.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.427+0000 7fb57f883700 1 -- 172.21.15.98:0/2034212571 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5780ca370 msgr2=0x7fb578158270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:02.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.427+0000 7fb57f883700 1 --2- 172.21.15.98:0/2034212571 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5780ca370 0x7fb578158270 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fb5781554c0 tx=0x7fb574009840 comp rx=0 tx=0).stop 2023-10-06T05:08:02.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.427+0000 7fb57f883700 1 -- 172.21.15.98:0/2034212571 shutdown_connections 2023-10-06T05:08:02.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.427+0000 7fb57f883700 1 --2- 172.21.15.98:0/2034212571 >> v2:172.21.15.174:3300/0 conn(0x7fb5780ce8a0 0x7fb57815aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:02.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.427+0000 7fb57f883700 1 --2- 172.21.15.98:0/2034212571 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5780ca370 0x7fb578158270 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:02.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.427+0000 7fb57f883700 1 -- 172.21.15.98:0/2034212571 >> 172.21.15.98:0/2034212571 conn(0x7fb5780707d0 msgr2=0x7fb578070bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:02.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.427+0000 7fb57f883700 1 -- 172.21.15.98:0/2034212571 shutdown_connections 2023-10-06T05:08:02.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.427+0000 7fb57f883700 1 -- 172.21.15.98:0/2034212571 wait complete. 2023-10-06T05:08:02.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.428+0000 7fb57f883700 1 Processor -- start 2023-10-06T05:08:02.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.428+0000 7fb57f883700 1 -- start start 2023-10-06T05:08:02.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.429+0000 7fb57f883700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb5780adb60 0x7fb578152da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:02.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.429+0000 7fb57f883700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5780ca370 0x7fb5780c3af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:02.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.429+0000 7fb57f883700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb5780ce8a0 0x7fb5780c72d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:02.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.429+0000 7fb57f883700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5781532e0 con 0x7fb5780ca370 2023-10-06T05:08:02.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.429+0000 7fb57f883700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5780b2050 con 0x7fb5780adb60 2023-10-06T05:08:02.435 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.429+0000 7fb57f883700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5780b22a0 con 0x7fb5780ce8a0 2023-10-06T05:08:02.435 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.429+0000 7fb57ce1e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5780ca370 0x7fb5780c3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:02.435 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.429+0000 7fb57ce1e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5780ca370 0x7fb5780c3af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54912/0 (socket says 172.21.15.98:54912) 2023-10-06T05:08:02.435 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.429+0000 7fb57ce1e700 1 -- 172.21.15.98:0/1695571652 learned_addr learned my addr 172.21.15.98:0/1695571652 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:08:02.435 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.429+0000 7fb57d61f700 1 --2- 172.21.15.98:0/1695571652 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb5780adb60 0x7fb578152da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:02.435 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.429+0000 7fb57de20700 1 --2- 172.21.15.98:0/1695571652 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb5780ce8a0 0x7fb5780c72d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:02.435 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.429+0000 7fb57ce1e700 1 -- 172.21.15.98:0/1695571652 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb5780ce8a0 msgr2=0x7fb5780c72d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:02.435 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.429+0000 7fb57ce1e700 1 --2- 172.21.15.98:0/1695571652 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb5780ce8a0 0x7fb5780c72d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:02.435 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.429+0000 7fb57ce1e700 1 -- 172.21.15.98:0/1695571652 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb5780adb60 msgr2=0x7fb578152da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:02.435 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.429+0000 7fb57ce1e700 1 --2- 172.21.15.98:0/1695571652 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb5780adb60 0x7fb578152da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:02.436 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.429+0000 7fb57ce1e700 1 -- 172.21.15.98:0/1695571652 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb574009ae0 con 0x7fb5780ca370 2023-10-06T05:08:02.436 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.429+0000 7fb57de20700 1 --2- 172.21.15.98:0/1695571652 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb5780ce8a0 0x7fb5780c72d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:08:02.436 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.429+0000 7fb57ce1e700 1 --2- 172.21.15.98:0/1695571652 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5780ca370 0x7fb5780c3af0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fb56800b5e0 tx=0x7fb5680090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:02.436 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.429+0000 7fb56d7fa700 1 -- 172.21.15.98:0/1695571652 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb56800f510 con 0x7fb5780ca370 2023-10-06T05:08:02.436 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.429+0000 7fb56d7fa700 1 -- 172.21.15.98:0/1695571652 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb56800f6b0 con 0x7fb5780ca370 2023-10-06T05:08:02.436 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.429+0000 7fb57f883700 1 -- 172.21.15.98:0/1695571652 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5780b2580 con 0x7fb5780ca370 2023-10-06T05:08:02.436 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.429+0000 7fb56d7fa700 1 -- 172.21.15.98:0/1695571652 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb56800fa70 con 0x7fb5780ca370 2023-10-06T05:08:02.436 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.429+0000 7fb57f883700 1 -- 172.21.15.98:0/1695571652 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5780b2b40 con 0x7fb5780ca370 2023-10-06T05:08:02.436 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.430+0000 7fb57f883700 1 -- 172.21.15.98:0/1695571652 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5780b2d60 con 0x7fb5780ca370 2023-10-06T05:08:02.436 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.431+0000 7fb56d7fa700 1 -- 172.21.15.98:0/1695571652 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fb56801a400 con 0x7fb5780ca370 2023-10-06T05:08:02.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.431+0000 7fb56d7fa700 1 --2- 172.21.15.98:0/1695571652 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fb55c066f30 0x7fb55c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:02.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.431+0000 7fb56d7fa700 1 -- 172.21.15.98:0/1695571652 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6378+0+0 (secure 0 0 0) 0x7fb568084230 con 0x7fb5780ca370 2023-10-06T05:08:02.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.431+0000 7fb57d61f700 1 --2- 172.21.15.98:0/1695571652 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fb55c066f30 0x7fb55c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:02.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.432+0000 7fb57d61f700 1 --2- 172.21.15.98:0/1695571652 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fb55c066f30 0x7fb55c0693e0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb574009880 tx=0x7fb574017000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:02.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.435+0000 7fb56d7fa700 1 -- 172.21.15.98:0/1695571652 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb568051280 con 0x7fb5780ca370 2023-10-06T05:08:02.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.665+0000 7fb57f883700 1 -- 172.21.15.98:0/1695571652 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fb578064a90 con 0x7fb5780ca370 2023-10-06T05:08:02.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.666+0000 7fb56d7fa700 1 -- 172.21.15.98:0/1695571652 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) v1 ==== 53+0+73 (secure 0 0 0) 0x7fb5680548a0 con 0x7fb5780ca370 2023-10-06T05:08:02.668 INFO:teuthology.orchestra.run.smithi098.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-06T05:08:02.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.668+0000 7fb57f883700 1 -- 172.21.15.98:0/1695571652 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fb55c066f30 msgr2=0x7fb55c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:02.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.668+0000 7fb57f883700 1 --2- 172.21.15.98:0/1695571652 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fb55c066f30 0x7fb55c0693e0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb574009880 tx=0x7fb574017000 comp rx=0 tx=0).stop 2023-10-06T05:08:02.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.668+0000 7fb57f883700 1 -- 172.21.15.98:0/1695571652 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5780ca370 msgr2=0x7fb5780c3af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:02.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.668+0000 7fb57f883700 1 --2- 172.21.15.98:0/1695571652 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5780ca370 0x7fb5780c3af0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fb56800b5e0 tx=0x7fb5680090f0 comp rx=0 tx=0).stop 2023-10-06T05:08:02.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.668+0000 7fb57f883700 1 -- 172.21.15.98:0/1695571652 shutdown_connections 2023-10-06T05:08:02.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.668+0000 7fb57f883700 1 --2- 172.21.15.98:0/1695571652 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb5780ce8a0 0x7fb5780c72d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:02.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.668+0000 7fb57f883700 1 --2- 172.21.15.98:0/1695571652 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb5780adb60 0x7fb578152da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:02.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.668+0000 7fb57f883700 1 --2- 172.21.15.98:0/1695571652 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fb55c066f30 0x7fb55c0693e0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:02.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.668+0000 7fb57f883700 1 --2- 172.21.15.98:0/1695571652 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5780ca370 0x7fb5780c3af0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:02.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.668+0000 7fb57f883700 1 -- 172.21.15.98:0/1695571652 >> 172.21.15.98:0/1695571652 conn(0x7fb5780707d0 msgr2=0x7fb5780ca740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:02.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.668+0000 7fb57f883700 1 -- 172.21.15.98:0/1695571652 shutdown_connections 2023-10-06T05:08:02.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:02.668+0000 7fb57f883700 1 -- 172.21.15.98:0/1695571652 wait complete. 2023-10-06T05:08:02.682 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-06T05:08:02.682 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-06T05:08:02.682 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi098.front.sepia.ceph.com 2023-10-06T05:08:02.682 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-06T05:08:02.682 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-06T05:08:02.682 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-06T05:08:02.682 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-06T05:08:02.682 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-06T05:08:02.682 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-06T05:08:02.683 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:08:02.683 DEBUG:teuthology.orchestra.run.smithi098:> ip addr 2023-10-06T05:08:02.697 INFO:teuthology.orchestra.run.smithi098.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-06T05:08:02.698 INFO:teuthology.orchestra.run.smithi098.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-06T05:08:02.698 INFO:teuthology.orchestra.run.smithi098.stdout: inet 127.0.0.1/8 scope host lo 2023-10-06T05:08:02.698 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-10-06T05:08:02.698 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 ::1/128 scope host 2023-10-06T05:08:02.698 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-10-06T05:08:02.698 INFO:teuthology.orchestra.run.smithi098.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-06T05:08:02.698 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:d8:d9:b8 brd ff:ff:ff:ff:ff:ff 2023-10-06T05:08:02.698 INFO:teuthology.orchestra.run.smithi098.stdout: altname enp7s0f0 2023-10-06T05:08:02.698 INFO:teuthology.orchestra.run.smithi098.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-06T05:08:02.699 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:d8:d9:b9 brd ff:ff:ff:ff:ff:ff 2023-10-06T05:08:02.699 INFO:teuthology.orchestra.run.smithi098.stdout: altname enp7s0f1 2023-10-06T05:08:02.699 INFO:teuthology.orchestra.run.smithi098.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-06T05:08:02.699 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:8f:d2:6a brd ff:ff:ff:ff:ff:ff 2023-10-06T05:08:02.699 INFO:teuthology.orchestra.run.smithi098.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-06T05:08:02.699 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:8f:d2:6b brd ff:ff:ff:ff:ff:ff 2023-10-06T05:08:02.699 INFO:teuthology.orchestra.run.smithi098.stdout: inet 172.21.15.98/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-10-06T05:08:02.699 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft 42442sec preferred_lft 42442sec 2023-10-06T05:08:02.699 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::ec4:7aff:fe8f:d26b/64 scope link 2023-10-06T05:08:02.699 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-10-06T05:08:02.699 INFO:teuthology.orchestra.run.smithi098.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-06T05:08:02.699 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 5e:cd:0c:56:12:cb brd ff:ff:ff:ff:ff:ff 2023-10-06T05:08:02.700 INFO:teuthology.orchestra.run.smithi098.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-06T05:08:02.700 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-10-06T05:08:02.700 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::f8f9:dcff:feb4:5363/64 scope link 2023-10-06T05:08:02.700 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-10-06T05:08:02.700 INFO:teuthology.orchestra.run.smithi098.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-06T05:08:02.700 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether fa:c2:06:14:b9:4f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-06T05:08:02.700 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::f8c2:6ff:fe14:b94f/64 scope link 2023-10-06T05:08:02.700 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-10-06T05:08:02.700 INFO:teuthology.orchestra.run.smithi098.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-06T05:08:02.700 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 5e:cd:0c:56:12:cb brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-06T05:08:02.700 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::5ccd:cff:fe56:12cb/64 scope link 2023-10-06T05:08:02.701 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-10-06T05:08:02.701 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:08:02.701 DEBUG:teuthology.orchestra.run.smithi098:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-06T05:08:02.761 INFO:teuthology.orchestra.run.smithi098.stdout:1 2023-10-06T05:08:02.762 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:08:02.762 DEBUG:teuthology.orchestra.run.smithi098:> route 2023-10-06T05:08:02.818 INFO:teuthology.orchestra.run.smithi098.stdout:Kernel IP routing table 2023-10-06T05:08:02.819 INFO:teuthology.orchestra.run.smithi098.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-06T05:08:02.819 INFO:teuthology.orchestra.run.smithi098.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-10-06T05:08:02.819 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-10-06T05:08:02.819 INFO:teuthology.orchestra.run.smithi098.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-06T05:08:02.819 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:08:02.819 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec bash -c ' 2023-10-06T05:08:02.819 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2023-10-06T05:08:02.820 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-10-06T05:08:02.820 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-10-06T05:08:02.820 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-10-06T05:08:02.820 DEBUG:teuthology.orchestra.run.smithi098:> ') 2023-10-06T05:08:02.918 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:08:02.918 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list 2023-10-06T05:08:02.933 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-06T05:08:02.933 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-06T05:08:02.933 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-06T05:08:02.933 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:08:02.933 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set brx.0 up 2023-10-06T05:08:02.997 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-06T05:08:02.997 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-06T05:08:03.052 INFO:teuthology.orchestra.run.smithi098.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-06T05:08:03.052 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-06T05:08:03.052 DEBUG:teuthology.orchestra.run.smithi098:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-06T05:08:03.106 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2023-10-06T05:08:03.106 INFO:teuthology.orchestra.run:Running command with timeout 1800 2023-10-06T05:08:03.106 DEBUG:teuthology.orchestra.run.smithi098:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-06T05:08:03.185 INFO:teuthology.orchestra.run.smithi098.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-06T05:08:03.185 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-06T05:08:03.204 INFO:teuthology.orchestra.run.smithi098.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-06T05:08:03.204 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-06T05:08:03.204 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: resolved to: "172.21.15.98:3300,172.21.15.174:3300,172.21.15.174:3301" 2023-10-06T05:08:03.204 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: trying mount with new device syntax: 0@f50c5df5-cad5-47a4-9bb1-8c791d4311ec.cephfs=/ 2023-10-06T05:08:03.204 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.98:3300/172.21.15.174:3300/172.21.15.174:3301" will pass to kernel 2023-10-06T05:08:03.204 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: v2 mount support check returned 2 2023-10-06T05:08:03.205 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: kernel does not support v2 syntax 2023-10-06T05:08:03.205 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-06T05:08:03.226 INFO:teuthology.orchestra.run.smithi098.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-06T05:08:03.226 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-06T05:08:03.227 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: resolved to: "172.21.15.98:3300,172.21.15.174:3300,172.21.15.174:3301" 2023-10-06T05:08:03.227 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: trying mount with new device syntax: 0@f50c5df5-cad5-47a4-9bb1-8c791d4311ec.cephfs=/ 2023-10-06T05:08:03.227 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.98:3300/172.21.15.174:3300/172.21.15.174:3301" will pass to kernel 2023-10-06T05:08:03.227 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: v2 mount support check returned 2 2023-10-06T05:08:03.227 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: kernel does not support v2 syntax 2023-10-06T05:08:03.227 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-06T05:08:03.227 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: resolved to: "172.21.15.98:6789,172.21.15.174:6789,172.21.15.174:6790" 2023-10-06T05:08:03.227 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.98:6789,172.21.15.174:6789,172.21.15.174:6790:/ 2023-10-06T05:08:03.227 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,key=0,mds_namespace=cephfs,fsid=f50c5df5-cad5-47a4-9bb1-8c791d4311ec" will pass to kernel 2023-10-06T05:08:03.229 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-06T05:08:03.229 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:08:03.229 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-06T05:08:03.294 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T05:08:03.294 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=foo bs=1M count=32) 2023-10-06T05:08:03.497 INFO:teuthology.orchestra.run.smithi098.stderr:32+0 records in 2023-10-06T05:08:03.498 INFO:teuthology.orchestra.run.smithi098.stderr:32+0 records out 2023-10-06T05:08:03.498 INFO:teuthology.orchestra.run.smithi098.stderr:33554432 bytes (34 MB, 32 MiB) copied, 0.141338 s, 237 MB/s 2023-10-06T05:08:03.498 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-06T05:08:03.498 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T05:08:03.498 DEBUG:teuthology.orchestra.run.smithi098:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-06T05:08:03.587 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi098.front.sepia.ceph.com 2023-10-06T05:08:03.587 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:08:03.587 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-06T05:08:03.602 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-06T05:08:03.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.754+0000 7f87ef59e700 1 Processor -- start 2023-10-06T05:08:03.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.754+0000 7f87ef59e700 1 -- start start 2023-10-06T05:08:03.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.755+0000 7f87ef59e700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f87f0159390 0x7f87f00ac8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:03.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.755+0000 7f87ef59e700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f87f00ab410 0x7f87f00aeb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:03.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.755+0000 7f87ef59e700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87f00686c0 con 0x7f87f0159760 2023-10-06T05:08:03.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.755+0000 7f87ef59e700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87f0068c80 con 0x7f87f00ab410 2023-10-06T05:08:03.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.755+0000 7f87ef59e700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87f00ace20 con 0x7f87f0159390 2023-10-06T05:08:03.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.755+0000 7f87edd9b700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f87f0159760 0x7f87f00aad00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:38388/0 (socket says 172.21.15.98:38388) 2023-10-06T05:08:03.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.755+0000 7f87edd9b700 1 -- 172.21.15.98:0/1229047406 learned_addr learned my addr 172.21.15.98:0/1229047406 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:08:03.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.755+0000 7f87ee59c700 1 --2- 172.21.15.98:0/1229047406 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f87f0159390 0x7f87f00ac8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:03.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.755+0000 7f87eed9d700 1 --2- 172.21.15.98:0/1229047406 >> v2:172.21.15.174:3300/0 conn(0x7f87f00ab410 0x7f87f00aeb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:03.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.755+0000 7f87ed59a700 1 -- 172.21.15.98:0/1229047406 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2714945150 0 0) 0x7f87f00686c0 con 0x7f87f0159760 2023-10-06T05:08:03.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.755+0000 7f87ed59a700 1 -- 172.21.15.98:0/1229047406 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87d8003570 con 0x7f87f0159760 2023-10-06T05:08:03.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.756+0000 7f87ed59a700 1 -- 172.21.15.98:0/1229047406 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 494874392 0 0) 0x7f87d8003570 con 0x7f87f0159760 2023-10-06T05:08:03.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.756+0000 7f87ed59a700 1 -- 172.21.15.98:0/1229047406 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f87f0159390 msgr2=0x7f87f00ac8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:03.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.756+0000 7f87ed59a700 1 --2- 172.21.15.98:0/1229047406 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f87f0159390 0x7f87f00ac8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:03.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.756+0000 7f87ed59a700 1 -- 172.21.15.98:0/1229047406 >> v2:172.21.15.174:3300/0 conn(0x7f87f00ab410 msgr2=0x7f87f00aeb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:03.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.756+0000 7f87ed59a700 1 --2- 172.21.15.98:0/1229047406 >> v2:172.21.15.174:3300/0 conn(0x7f87f00ab410 0x7f87f00aeb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:03.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.756+0000 7f87ed59a700 1 -- 172.21.15.98:0/1229047406 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87f00ad0f0 con 0x7f87f0159760 2023-10-06T05:08:03.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.756+0000 7f87ed59a700 1 -- 172.21.15.98:0/1229047406 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1543991876 0 0) 0x7f87e4001470 con 0x7f87f0159760 2023-10-06T05:08:03.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.756+0000 7f87ee59c700 1 --2- 172.21.15.98:0/1229047406 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f87f0159390 0x7f87f00ac8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T05:08:03.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.756+0000 7f87eed9d700 1 --2- 172.21.15.98:0/1229047406 >> v2:172.21.15.174:3300/0 conn(0x7f87f00ab410 0x7f87f00aeb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T05:08:03.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.756+0000 7f87ed59a700 1 -- 172.21.15.98:0/1229047406 >> v1:172.21.15.98:6789/0 conn(0x7f87f0159760 legacy=0x7f87f00aad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:03.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.756+0000 7f87ed59a700 1 --2- 172.21.15.98:0/1229047406 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f87f0159390 0x7f87d8003a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:03.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.756+0000 7f87ed59a700 1 --2- 172.21.15.98:0/1229047406 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f87d8003fa0 0x7f87d8004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:03.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.756+0000 7f87ed59a700 1 --2- 172.21.15.98:0/1229047406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87d8004d50 0x7f87d80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:03.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.756+0000 7f87ed59a700 1 -- 172.21.15.98:0/1229047406 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f0068c80 con 0x7f87d8004d50 2023-10-06T05:08:03.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.756+0000 7f87ed59a700 1 -- 172.21.15.98:0/1229047406 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f00ace20 con 0x7f87d8003fa0 2023-10-06T05:08:03.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.756+0000 7f87ed59a700 1 -- 172.21.15.98:0/1229047406 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87d800ba60 con 0x7f87f0159390 2023-10-06T05:08:03.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.756+0000 7f87eed9d700 1 --2- 172.21.15.98:0/1229047406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87d8004d50 0x7f87d80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:03.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.756+0000 7f87ee59c700 1 --2- 172.21.15.98:0/1229047406 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f87f0159390 0x7f87d8003a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:03.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.756+0000 7f87edd9b700 1 --2- 172.21.15.98:0/1229047406 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f87d8003fa0 0x7f87d8004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:03.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.756+0000 7f87ee59c700 1 -- 172.21.15.98:0/1229047406 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f87d8003fa0 msgr2=0x7f87d8004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:03.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.756+0000 7f87ee59c700 1 --2- 172.21.15.98:0/1229047406 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f87d8003fa0 0x7f87d8004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:03.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.756+0000 7f87ee59c700 1 -- 172.21.15.98:0/1229047406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87d8004d50 msgr2=0x7f87d80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:03.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.756+0000 7f87ee59c700 1 --2- 172.21.15.98:0/1229047406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87d8004d50 0x7f87d80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:03.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.756+0000 7f87ee59c700 1 -- 172.21.15.98:0/1229047406 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87e0009420 con 0x7f87f0159390 2023-10-06T05:08:03.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.756+0000 7f87eed9d700 1 --2- 172.21.15.98:0/1229047406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87d8004d50 0x7f87d80051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T05:08:03.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.756+0000 7f87ee59c700 1 --2- 172.21.15.98:0/1229047406 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f87f0159390 0x7f87d8003a60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f87e0009df0 tx=0x7f87e00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c9579d4308d01d62 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:03.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.756+0000 7f87ed59a700 1 -- 172.21.15.98:0/1229047406 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87e000e920 con 0x7f87f0159390 2023-10-06T05:08:03.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.756+0000 7f87ed59a700 1 -- 172.21.15.98:0/1229047406 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87e000ea80 con 0x7f87f0159390 2023-10-06T05:08:03.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.756+0000 7f87ed59a700 1 -- 172.21.15.98:0/1229047406 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87e000ee20 con 0x7f87f0159390 2023-10-06T05:08:03.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.757+0000 7f87ef59e700 1 -- 172.21.15.98:0/1229047406 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f87f0159390 msgr2=0x7f87d8003a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:03.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.757+0000 7f87ef59e700 1 --2- 172.21.15.98:0/1229047406 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f87f0159390 0x7f87d8003a60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f87e0009df0 tx=0x7f87e00091a0 comp rx=0 tx=0).stop 2023-10-06T05:08:03.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.757+0000 7f87eed9d700 1 -- 172.21.15.98:0/1229047406 reap_dead start 2023-10-06T05:08:03.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.757+0000 7f87ef59e700 1 -- 172.21.15.98:0/1229047406 shutdown_connections 2023-10-06T05:08:03.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.757+0000 7f87ef59e700 1 -- 172.21.15.98:0/1229047406 >> 172.21.15.98:0/1229047406 conn(0x7f87f0097410 msgr2=0x7f87f0067d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:03.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.757+0000 7f87ef59e700 1 -- 172.21.15.98:0/1229047406 shutdown_connections 2023-10-06T05:08:03.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.757+0000 7f87ef59e700 1 -- 172.21.15.98:0/1229047406 wait complete. 2023-10-06T05:08:03.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.758+0000 7f87ef59e700 1 Processor -- start 2023-10-06T05:08:03.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.758+0000 7f87ef59e700 1 -- start start 2023-10-06T05:08:03.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.759+0000 7f87ef59e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87f0159390 0x7f87f0152c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:03.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.759+0000 7f87ef59e700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f87f0169a30 0x7f87f0169ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:03.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.759+0000 7f87ef59e700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f87f006c120 0x7f87f006c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:03.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.759+0000 7f87ef59e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f016aa70 con 0x7f87f0159390 2023-10-06T05:08:03.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.759+0000 7f87ef59e700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f016b0e0 con 0x7f87f0169a30 2023-10-06T05:08:03.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.759+0000 7f87ef59e700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87f006cab0 con 0x7f87f006c120 2023-10-06T05:08:03.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.759+0000 7f87ee59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87f0159390 0x7f87f0152c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:03.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.759+0000 7f87ee59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87f0159390 0x7f87f0152c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54942/0 (socket says 172.21.15.98:54942) 2023-10-06T05:08:03.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.759+0000 7f87ee59c700 1 -- 172.21.15.98:0/962060851 learned_addr learned my addr 172.21.15.98:0/962060851 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:08:03.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.759+0000 7f87eed9d700 1 --2- 172.21.15.98:0/962060851 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f87f006c120 0x7f87f006c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:03.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.759+0000 7f87edd9b700 1 --2- 172.21.15.98:0/962060851 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f87f0169a30 0x7f87f0169ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:03.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.759+0000 7f87ee59c700 1 -- 172.21.15.98:0/962060851 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f87f006c120 msgr2=0x7f87f006c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:03.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.759+0000 7f87ee59c700 1 --2- 172.21.15.98:0/962060851 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f87f006c120 0x7f87f006c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:03.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.759+0000 7f87ee59c700 1 -- 172.21.15.98:0/962060851 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f87f0169a30 msgr2=0x7f87f0169ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:03.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.759+0000 7f87ee59c700 1 --2- 172.21.15.98:0/962060851 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f87f0169a30 0x7f87f0169ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:03.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.759+0000 7f87ee59c700 1 -- 172.21.15.98:0/962060851 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87f016adb0 con 0x7f87f0159390 2023-10-06T05:08:03.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.759+0000 7f87eed9d700 1 --2- 172.21.15.98:0/962060851 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f87f006c120 0x7f87f006c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:08:03.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.759+0000 7f87ee59c700 1 --2- 172.21.15.98:0/962060851 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87f0159390 0x7f87f0152c60 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f87e000e790 tx=0x7f87e001a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:03.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.760+0000 7f87de7fc700 1 -- 172.21.15.98:0/962060851 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87e0022070 con 0x7f87f0159390 2023-10-06T05:08:03.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.760+0000 7f87de7fc700 1 -- 172.21.15.98:0/962060851 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87e0013c20 con 0x7f87f0159390 2023-10-06T05:08:03.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.760+0000 7f87ef59e700 1 -- 172.21.15.98:0/962060851 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87f016a4b0 con 0x7f87f0159390 2023-10-06T05:08:03.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.760+0000 7f87de7fc700 1 -- 172.21.15.98:0/962060851 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87e001c450 con 0x7f87f0159390 2023-10-06T05:08:03.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.760+0000 7f87ef59e700 1 -- 172.21.15.98:0/962060851 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87f01531a0 con 0x7f87f0159390 2023-10-06T05:08:03.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.760+0000 7f87ef59e700 1 -- 172.21.15.98:0/962060851 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87f016ac10 con 0x7f87f0159390 2023-10-06T05:08:03.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.761+0000 7f87de7fc700 1 -- 172.21.15.98:0/962060851 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f87e0026070 con 0x7f87f0159390 2023-10-06T05:08:03.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.761+0000 7f87de7fc700 1 --2- 172.21.15.98:0/962060851 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f87d0066f30 0x7f87d00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:03.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.761+0000 7f87de7fc700 1 -- 172.21.15.98:0/962060851 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6378+0+0 (secure 0 0 0) 0x7f87e0086040 con 0x7f87f0159390 2023-10-06T05:08:03.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.761+0000 7f87edd9b700 1 --2- 172.21.15.98:0/962060851 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f87d0066f30 0x7f87d00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:03.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.762+0000 7f87edd9b700 1 --2- 172.21.15.98:0/962060851 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f87d0066f30 0x7f87d00693e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f87e4001d20 tx=0x7f87e4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:03.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.765+0000 7f87de7fc700 1 -- 172.21.15.98:0/962060851 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f87f016ac10 con 0x7f87f0159390 2023-10-06T05:08:03.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.991+0000 7f87ef59e700 1 -- 172.21.15.98:0/962060851 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f87f016ac10 con 0x7f87f0159390 2023-10-06T05:08:03.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.992+0000 7f87de7fc700 1 -- 172.21.15.98:0/962060851 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) v1 ==== 53+0+73 (secure 0 0 0) 0x7f87f016ac10 con 0x7f87f0159390 2023-10-06T05:08:03.994 INFO:teuthology.orchestra.run.smithi098.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-06T05:08:03.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.994+0000 7f87ef59e700 1 -- 172.21.15.98:0/962060851 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f87d0066f30 msgr2=0x7f87d00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:03.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.994+0000 7f87ef59e700 1 --2- 172.21.15.98:0/962060851 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f87d0066f30 0x7f87d00693e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f87e4001d20 tx=0x7f87e4009040 comp rx=0 tx=0).stop 2023-10-06T05:08:03.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.994+0000 7f87ef59e700 1 -- 172.21.15.98:0/962060851 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87f0159390 msgr2=0x7f87f0152c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:03.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.994+0000 7f87ef59e700 1 --2- 172.21.15.98:0/962060851 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87f0159390 0x7f87f0152c60 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f87e000e790 tx=0x7f87e001a000 comp rx=0 tx=0).stop 2023-10-06T05:08:03.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.994+0000 7f87ef59e700 1 -- 172.21.15.98:0/962060851 shutdown_connections 2023-10-06T05:08:03.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.994+0000 7f87ef59e700 1 --2- 172.21.15.98:0/962060851 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f87f006c120 0x7f87f006c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:03.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.994+0000 7f87ef59e700 1 --2- 172.21.15.98:0/962060851 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f87d0066f30 0x7f87d00693e0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:03.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.994+0000 7f87ef59e700 1 --2- 172.21.15.98:0/962060851 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87f0159390 0x7f87f0152c60 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:03.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.994+0000 7f87ef59e700 1 --2- 172.21.15.98:0/962060851 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f87f0169a30 0x7f87f0169ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:03.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.994+0000 7f87ef59e700 1 -- 172.21.15.98:0/962060851 >> 172.21.15.98:0/962060851 conn(0x7f87f0097410 msgr2=0x7f87f01567f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:03.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.994+0000 7f87ef59e700 1 -- 172.21.15.98:0/962060851 shutdown_connections 2023-10-06T05:08:03.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:03.994+0000 7f87ef59e700 1 -- 172.21.15.98:0/962060851 wait complete. 2023-10-06T05:08:04.008 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-06T05:08:04.008 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-06T05:08:04.008 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi098.front.sepia.ceph.com 2023-10-06T05:08:04.008 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-06T05:08:04.008 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-06T05:08:04.008 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-06T05:08:04.008 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-06T05:08:04.008 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-06T05:08:04.008 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-06T05:08:04.008 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:08:04.009 DEBUG:teuthology.orchestra.run.smithi098:> ip addr 2023-10-06T05:08:04.023 INFO:teuthology.orchestra.run.smithi098.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-06T05:08:04.023 INFO:teuthology.orchestra.run.smithi098.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-06T05:08:04.023 INFO:teuthology.orchestra.run.smithi098.stdout: inet 127.0.0.1/8 scope host lo 2023-10-06T05:08:04.024 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-10-06T05:08:04.024 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 ::1/128 scope host 2023-10-06T05:08:04.024 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-10-06T05:08:04.024 INFO:teuthology.orchestra.run.smithi098.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-06T05:08:04.024 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:d8:d9:b8 brd ff:ff:ff:ff:ff:ff 2023-10-06T05:08:04.024 INFO:teuthology.orchestra.run.smithi098.stdout: altname enp7s0f0 2023-10-06T05:08:04.024 INFO:teuthology.orchestra.run.smithi098.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-06T05:08:04.024 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:d8:d9:b9 brd ff:ff:ff:ff:ff:ff 2023-10-06T05:08:04.024 INFO:teuthology.orchestra.run.smithi098.stdout: altname enp7s0f1 2023-10-06T05:08:04.024 INFO:teuthology.orchestra.run.smithi098.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-06T05:08:04.024 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:8f:d2:6a brd ff:ff:ff:ff:ff:ff 2023-10-06T05:08:04.025 INFO:teuthology.orchestra.run.smithi098.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-06T05:08:04.025 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:8f:d2:6b brd ff:ff:ff:ff:ff:ff 2023-10-06T05:08:04.025 INFO:teuthology.orchestra.run.smithi098.stdout: inet 172.21.15.98/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-10-06T05:08:04.025 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft 42441sec preferred_lft 42441sec 2023-10-06T05:08:04.025 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::ec4:7aff:fe8f:d26b/64 scope link 2023-10-06T05:08:04.025 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-10-06T05:08:04.025 INFO:teuthology.orchestra.run.smithi098.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-06T05:08:04.025 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 5e:cd:0c:56:12:cb brd ff:ff:ff:ff:ff:ff 2023-10-06T05:08:04.025 INFO:teuthology.orchestra.run.smithi098.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-06T05:08:04.025 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-10-06T05:08:04.025 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::f8f9:dcff:feb4:5363/64 scope link 2023-10-06T05:08:04.025 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-10-06T05:08:04.026 INFO:teuthology.orchestra.run.smithi098.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-06T05:08:04.026 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether fa:c2:06:14:b9:4f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-06T05:08:04.026 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::f8c2:6ff:fe14:b94f/64 scope link 2023-10-06T05:08:04.026 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-10-06T05:08:04.026 INFO:teuthology.orchestra.run.smithi098.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-06T05:08:04.026 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 5e:cd:0c:56:12:cb brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-06T05:08:04.026 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::5ccd:cff:fe56:12cb/64 scope link 2023-10-06T05:08:04.026 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-10-06T05:08:04.026 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:08:04.027 DEBUG:teuthology.orchestra.run.smithi098:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-06T05:08:04.087 INFO:teuthology.orchestra.run.smithi098.stdout:1 2023-10-06T05:08:04.088 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:08:04.088 DEBUG:teuthology.orchestra.run.smithi098:> route 2023-10-06T05:08:04.143 INFO:teuthology.orchestra.run.smithi098.stdout:Kernel IP routing table 2023-10-06T05:08:04.143 INFO:teuthology.orchestra.run.smithi098.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-06T05:08:04.144 INFO:teuthology.orchestra.run.smithi098.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-10-06T05:08:04.144 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-10-06T05:08:04.144 INFO:teuthology.orchestra.run.smithi098.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-06T05:08:04.144 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:08:04.144 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec bash -c ' 2023-10-06T05:08:04.144 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2023-10-06T05:08:04.144 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-10-06T05:08:04.144 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-10-06T05:08:04.145 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-10-06T05:08:04.145 DEBUG:teuthology.orchestra.run.smithi098:> ') 2023-10-06T05:08:04.243 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:08:04.244 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list 2023-10-06T05:08:04.258 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-06T05:08:04.258 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-06T05:08:04.258 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-06T05:08:04.258 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:08:04.259 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set brx.0 up 2023-10-06T05:08:04.323 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-06T05:08:04.324 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-06T05:08:04.378 INFO:teuthology.orchestra.run.smithi098.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-06T05:08:04.379 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-06T05:08:04.379 DEBUG:teuthology.orchestra.run.smithi098:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-06T05:08:04.433 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2023-10-06T05:08:04.433 INFO:teuthology.orchestra.run:Running command with timeout 1800 2023-10-06T05:08:04.433 DEBUG:teuthology.orchestra.run.smithi098:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-06T05:08:04.513 INFO:teuthology.orchestra.run.smithi098.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-06T05:08:04.513 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-06T05:08:04.532 INFO:teuthology.orchestra.run.smithi098.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-06T05:08:04.533 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-06T05:08:04.533 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: resolved to: "172.21.15.98:3300,172.21.15.174:3300,172.21.15.174:3301" 2023-10-06T05:08:04.533 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: trying mount with new device syntax: 0@f50c5df5-cad5-47a4-9bb1-8c791d4311ec.cephfs=/ 2023-10-06T05:08:04.533 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.98:3300/172.21.15.174:3300/172.21.15.174:3301" will pass to kernel 2023-10-06T05:08:04.533 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: v2 mount support check returned 2 2023-10-06T05:08:04.533 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: kernel does not support v2 syntax 2023-10-06T05:08:04.533 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-06T05:08:04.555 INFO:teuthology.orchestra.run.smithi098.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-10-06T05:08:04.555 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-10-06T05:08:04.556 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: resolved to: "172.21.15.98:3300,172.21.15.174:3300,172.21.15.174:3301" 2023-10-06T05:08:04.556 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: trying mount with new device syntax: 0@f50c5df5-cad5-47a4-9bb1-8c791d4311ec.cephfs=/ 2023-10-06T05:08:04.556 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.98:3300/172.21.15.174:3300/172.21.15.174:3301" will pass to kernel 2023-10-06T05:08:04.556 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: v2 mount support check returned 2 2023-10-06T05:08:04.556 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: kernel does not support v2 syntax 2023-10-06T05:08:04.556 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-06T05:08:04.556 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: resolved to: "172.21.15.98:6789,172.21.15.174:6789,172.21.15.174:6790" 2023-10-06T05:08:04.556 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.98:6789,172.21.15.174:6789,172.21.15.174:6790:/ 2023-10-06T05:08:04.556 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,key=0,mds_namespace=cephfs,fsid=f50c5df5-cad5-47a4-9bb1-8c791d4311ec" will pass to kernel 2023-10-06T05:08:04.558 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-06T05:08:04.558 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:08:04.558 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-06T05:08:04.622 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T05:08:04.623 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-06T05:08:04.691 INFO:teuthology.orchestra.run.smithi098.stdout:client4817 2023-10-06T05:08:04.692 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T05:08:04.692 DEBUG:teuthology.orchestra.run.smithi098:> sudo stat /sys/kernel/debug/ceph/f50c5df5-cad5-47a4-9bb1-8c791d4311ec.client4817/metrics/size 2023-10-06T05:08:04.758 INFO:teuthology.orchestra.run.smithi098.stdout: File: /sys/kernel/debug/ceph/f50c5df5-cad5-47a4-9bb1-8c791d4311ec.client4817/metrics/size 2023-10-06T05:08:04.758 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 0 Blocks: 0 IO Block: 4096 regular empty file 2023-10-06T05:08:04.758 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 8h/8d Inode: 298501 Links: 1 2023-10-06T05:08:04.758 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0400/-r--------) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-06T05:08:04.758 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:debugfs_t:s0 2023-10-06T05:08:04.758 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2023-10-06 05:08:04.546821825 +0000 2023-10-06T05:08:04.758 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2023-10-06 05:08:04.546821825 +0000 2023-10-06T05:08:04.758 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2023-10-06 05:08:04.546821825 +0000 2023-10-06T05:08:04.759 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: - 2023-10-06T05:08:04.759 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T05:08:04.760 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-06T05:08:04.828 INFO:teuthology.orchestra.run.smithi098.stdout:client4817 2023-10-06T05:08:04.829 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:08:04.829 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/f50c5df5-cad5-47a4-9bb1-8c791d4311ec.client4817/metrics/size') 2023-10-06T05:08:04.904 INFO:teuthology.orchestra.run.smithi098.stdout:item total avg_sz(bytes) min_sz(bytes) max_sz(bytes) total_sz(bytes) 2023-10-06T05:08:04.905 INFO:teuthology.orchestra.run.smithi098.stdout:---------------------------------------------------------------------------------------- 2023-10-06T05:08:04.905 INFO:teuthology.orchestra.run.smithi098.stdout:read 0 0 0 0 0 2023-10-06T05:08:04.905 INFO:teuthology.orchestra.run.smithi098.stdout:write 0 0 0 0 0 2023-10-06T05:08:04.905 INFO:teuthology.orchestra.run.smithi098.stdout:copyfrom 0 0 0 0 0 2023-10-06T05:08:04.906 INFO:teuthology.orchestra.run.smithi098.stderr:0+1 records in 2023-10-06T05:08:04.906 INFO:teuthology.orchestra.run.smithi098.stderr:0+1 records out 2023-10-06T05:08:04.906 INFO:teuthology.orchestra.run.smithi098.stderr:406 bytes copied, 2.1231e-05 s, 19.1 MB/s 2023-10-06T05:08:04.906 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T05:08:04.906 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=foo of=/dev/null bs=128k count=32) 2023-10-06T05:08:04.976 INFO:teuthology.orchestra.run.smithi098.stderr:32+0 records in 2023-10-06T05:08:04.976 INFO:teuthology.orchestra.run.smithi098.stderr:32+0 records out 2023-10-06T05:08:04.976 INFO:teuthology.orchestra.run.smithi098.stderr:4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.0144486 s, 290 MB/s 2023-10-06T05:08:04.977 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T05:08:04.977 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-06T05:08:05.045 INFO:teuthology.orchestra.run.smithi098.stdout:client4817 2023-10-06T05:08:05.045 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T05:08:05.046 DEBUG:teuthology.orchestra.run.smithi098:> sudo stat /sys/kernel/debug/ceph/f50c5df5-cad5-47a4-9bb1-8c791d4311ec.client4817/metrics/size 2023-10-06T05:08:05.109 INFO:teuthology.orchestra.run.smithi098.stdout: File: /sys/kernel/debug/ceph/f50c5df5-cad5-47a4-9bb1-8c791d4311ec.client4817/metrics/size 2023-10-06T05:08:05.109 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 0 Blocks: 0 IO Block: 4096 regular empty file 2023-10-06T05:08:05.109 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 8h/8d Inode: 298501 Links: 1 2023-10-06T05:08:05.109 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0400/-r--------) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-06T05:08:05.109 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:debugfs_t:s0 2023-10-06T05:08:05.110 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2023-10-06 05:08:04.546821825 +0000 2023-10-06T05:08:05.110 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2023-10-06 05:08:04.546821825 +0000 2023-10-06T05:08:05.110 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2023-10-06 05:08:04.546821825 +0000 2023-10-06T05:08:05.110 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: - 2023-10-06T05:08:05.111 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T05:08:05.111 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-06T05:08:05.178 INFO:teuthology.orchestra.run.smithi098.stdout:client4817 2023-10-06T05:08:05.179 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:08:05.179 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/f50c5df5-cad5-47a4-9bb1-8c791d4311ec.client4817/metrics/size') 2023-10-06T05:08:05.254 INFO:teuthology.orchestra.run.smithi098.stdout:item total avg_sz(bytes) min_sz(bytes) max_sz(bytes) total_sz(bytes) 2023-10-06T05:08:05.254 INFO:teuthology.orchestra.run.smithi098.stdout:---------------------------------------------------------------------------------------- 2023-10-06T05:08:05.254 INFO:teuthology.orchestra.run.smithi098.stdout:read 8 2031616 524288 4194304 16252928 2023-10-06T05:08:05.254 INFO:teuthology.orchestra.run.smithi098.stdout:write 0 0 0 0 0 2023-10-06T05:08:05.255 INFO:teuthology.orchestra.run.smithi098.stdout:copyfrom 0 0 0 0 0 2023-10-06T05:08:05.255 INFO:teuthology.orchestra.run.smithi098.stderr:0+1 records in 2023-10-06T05:08:05.256 INFO:teuthology.orchestra.run.smithi098.stderr:0+1 records out 2023-10-06T05:08:05.256 INFO:teuthology.orchestra.run.smithi098.stderr:413 bytes copied, 2.1897e-05 s, 18.9 MB/s 2023-10-06T05:08:05.256 INFO:tasks.cephfs.test_readahead:read operations: 8 2023-10-06T05:08:05.256 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-06T05:08:05.256 DEBUG:teuthology.orchestra.run.smithi098:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-06T05:08:05.339 DEBUG:teuthology.orchestra.run.smithi174:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-06T05:08:05.381 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-06T05:08:05.382 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-06T05:08:05.382 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T05:08:05.382 DEBUG:teuthology.orchestra.run.smithi098:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-06T05:08:05.423 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi098.front.sepia.ceph.com 2023-10-06T05:08:05.423 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:08:05.423 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-06T05:08:05.478 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-06T05:08:05.633 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.631+0000 7efc4a16d700 1 Processor -- start 2023-10-06T05:08:05.633 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.631+0000 7efc4a16d700 1 -- start start 2023-10-06T05:08:05.635 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.632+0000 7efc4a16d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7efc44159440 0x7efc440b3090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:05.635 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.632+0000 7efc4a16d700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efc440b35d0 0x7efc440cbc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:05.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.632+0000 7efc4a16d700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc440cb7a0 con 0x7efc440cc170 2023-10-06T05:08:05.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.632+0000 7efc4a16d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc440b0860 con 0x7efc44159440 2023-10-06T05:08:05.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.632+0000 7efc4a16d700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc440cb150 con 0x7efc440b35d0 2023-10-06T05:08:05.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.632+0000 7efc4916b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7efc44159440 0x7efc440b3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:05.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.632+0000 7efc4916b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7efc44159440 0x7efc440b3090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54956/0 (socket says 172.21.15.98:54956) 2023-10-06T05:08:05.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.632+0000 7efc4916b700 1 -- 172.21.15.98:0/3225442237 learned_addr learned my addr 172.21.15.98:0/3225442237 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:08:05.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.632+0000 7efc4896a700 1 --2- 172.21.15.98:0/3225442237 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efc440b35d0 0x7efc440cbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:05.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.632+0000 7efc4916b700 1 -- 172.21.15.98:0/3225442237 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efc440b35d0 msgr2=0x7efc440cbc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:05.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.632+0000 7efc4916b700 1 --2- 172.21.15.98:0/3225442237 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efc440b35d0 0x7efc440cbc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:05.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.632+0000 7efc4916b700 1 -- 172.21.15.98:0/3225442237 >> v1:172.21.15.174:6789/0 conn(0x7efc440cc170 legacy=0x7efc440b2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:05.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.632+0000 7efc4916b700 1 -- 172.21.15.98:0/3225442237 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc440cb430 con 0x7efc44159440 2023-10-06T05:08:05.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.632+0000 7efc4916b700 1 --2- 172.21.15.98:0/3225442237 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc44159440 0x7efc440b3090 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7efc3c009f20 tx=0x7efc3c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6276b86d36df20f server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:05.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.632+0000 7efc3bfff700 1 -- 172.21.15.98:0/3225442237 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc3c00e070 con 0x7efc44159440 2023-10-06T05:08:05.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.632+0000 7efc3bfff700 1 -- 172.21.15.98:0/3225442237 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc3c00ba80 con 0x7efc44159440 2023-10-06T05:08:05.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.633+0000 7efc3bfff700 1 -- 172.21.15.98:0/3225442237 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc3c0092e0 con 0x7efc44159440 2023-10-06T05:08:05.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.633+0000 7efc4a16d700 1 -- 172.21.15.98:0/3225442237 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc44159440 msgr2=0x7efc440b3090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:05.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.633+0000 7efc4a16d700 1 --2- 172.21.15.98:0/3225442237 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc44159440 0x7efc440b3090 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7efc3c009f20 tx=0x7efc3c009840 comp rx=0 tx=0).stop 2023-10-06T05:08:05.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.633+0000 7efc4a16d700 1 -- 172.21.15.98:0/3225442237 shutdown_connections 2023-10-06T05:08:05.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.633+0000 7efc4a16d700 1 --2- 172.21.15.98:0/3225442237 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc44159440 0x7efc440b3090 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:05.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.633+0000 7efc4a16d700 1 --2- 172.21.15.98:0/3225442237 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efc440b35d0 0x7efc440cbc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:05.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.633+0000 7efc4a16d700 1 -- 172.21.15.98:0/3225442237 >> 172.21.15.98:0/3225442237 conn(0x7efc4406ece0 msgr2=0x7efc4406f0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:05.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.633+0000 7efc4a16d700 1 -- 172.21.15.98:0/3225442237 shutdown_connections 2023-10-06T05:08:05.638 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.633+0000 7efc4a16d700 1 -- 172.21.15.98:0/3225442237 wait complete. 2023-10-06T05:08:05.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.634+0000 7efc4a16d700 1 Processor -- start 2023-10-06T05:08:05.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.634+0000 7efc4a16d700 1 -- start start 2023-10-06T05:08:05.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.635+0000 7efc4a16d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc440b35d0 0x7efc44151410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:05.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.635+0000 7efc4a16d700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efc440cc170 0x7efc44150e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:05.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.635+0000 7efc4a16d700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efc44159440 0x7efc4416a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:05.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.635+0000 7efc4a16d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc44151ed0 con 0x7efc440b35d0 2023-10-06T05:08:05.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.635+0000 7efc4a16d700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc44152ea0 con 0x7efc440cc170 2023-10-06T05:08:05.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.635+0000 7efc4a16d700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc44152a10 con 0x7efc44159440 2023-10-06T05:08:05.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.635+0000 7efc4916b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc440b35d0 0x7efc44151410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:05.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.635+0000 7efc4916b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc440b35d0 0x7efc44151410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54960/0 (socket says 172.21.15.98:54960) 2023-10-06T05:08:05.641 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.635+0000 7efc4916b700 1 -- 172.21.15.98:0/903266996 learned_addr learned my addr 172.21.15.98:0/903266996 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:08:05.641 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.635+0000 7efc4896a700 1 --2- 172.21.15.98:0/903266996 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efc440cc170 0x7efc44150e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:05.641 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.635+0000 7efc4916b700 1 -- 172.21.15.98:0/903266996 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efc44159440 msgr2=0x7efc4416a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:05.641 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.635+0000 7efc4916b700 1 --2- 172.21.15.98:0/903266996 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efc44159440 0x7efc4416a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:05.641 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.635+0000 7efc4916b700 1 -- 172.21.15.98:0/903266996 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efc440cc170 msgr2=0x7efc44150e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:05.641 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.635+0000 7efc4916b700 1 --2- 172.21.15.98:0/903266996 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efc440cc170 0x7efc44150e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:05.641 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.635+0000 7efc4916b700 1 -- 172.21.15.98:0/903266996 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc3c009ae0 con 0x7efc440b35d0 2023-10-06T05:08:05.641 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.636+0000 7efc4916b700 1 --2- 172.21.15.98:0/903266996 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc440b35d0 0x7efc44151410 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7efc3c00b6c0 tx=0x7efc3c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:05.641 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.636+0000 7efc397fa700 1 -- 172.21.15.98:0/903266996 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc3c008bb0 con 0x7efc440b35d0 2023-10-06T05:08:05.641 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.636+0000 7efc397fa700 1 -- 172.21.15.98:0/903266996 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc3c008d10 con 0x7efc440b35d0 2023-10-06T05:08:05.641 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.636+0000 7efc4a16d700 1 -- 172.21.15.98:0/903266996 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc441520e0 con 0x7efc440b35d0 2023-10-06T05:08:05.641 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.636+0000 7efc397fa700 1 -- 172.21.15.98:0/903266996 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc3c0256b0 con 0x7efc440b35d0 2023-10-06T05:08:05.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.636+0000 7efc4a16d700 1 -- 172.21.15.98:0/903266996 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc441526a0 con 0x7efc440b35d0 2023-10-06T05:08:05.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.637+0000 7efc4a16d700 1 -- 172.21.15.98:0/903266996 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc4405f230 con 0x7efc440b35d0 2023-10-06T05:08:05.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.637+0000 7efc397fa700 1 -- 172.21.15.98:0/903266996 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7efc3c003040 con 0x7efc440b35d0 2023-10-06T05:08:05.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.637+0000 7efc397fa700 1 --2- 172.21.15.98:0/903266996 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7efc28066f30 0x7efc280693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:05.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.638+0000 7efc397fa700 1 -- 172.21.15.98:0/903266996 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6378+0+0 (secure 0 0 0) 0x7efc3c086470 con 0x7efc440b35d0 2023-10-06T05:08:05.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.638+0000 7efc4896a700 1 --2- 172.21.15.98:0/903266996 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7efc28066f30 0x7efc280693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:05.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.638+0000 7efc4896a700 1 --2- 172.21.15.98:0/903266996 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7efc28066f30 0x7efc280693e0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7efc4416aa30 tx=0x7efc34008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:05.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.642+0000 7efc397fa700 1 -- 172.21.15.98:0/903266996 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7efc3c0533a0 con 0x7efc440b35d0 2023-10-06T05:08:05.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.864+0000 7efc4a16d700 1 -- 172.21.15.98:0/903266996 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efc44159810 con 0x7efc440b35d0 2023-10-06T05:08:05.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.864+0000 7efc397fa700 1 -- 172.21.15.98:0/903266996 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4603 (secure 0 0 0) 0x7efc3c0569c0 con 0x7efc440b35d0 2023-10-06T05:08:05.867 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:08:05.867 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4966,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6834/2320896583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2320896583},{"type":"v1","addr":"172.21.15.174:6834","nonce":2320896583}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":4969,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/1423644921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6833","nonce":1423644921},{"type":"v1","addr":"172.21.15.174:6835","nonce":1423644921}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5100,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/4291295874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":4291295874},{"type":"v1","addr":"172.21.15.98:6836","nonce":4291295874}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-06T05:07:55.873515+0000","modified":"2023-10-06T05:07:59.511013+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5103},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5103":{"gid":5103,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.98:6837/1617154626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1617154626},{"type":"v1","addr":"172.21.15.98:6837","nonce":1617154626}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2023-10-06T05:08:05.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.866+0000 7efc4a16d700 1 -- 172.21.15.98:0/903266996 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7efc28066f30 msgr2=0x7efc280693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:05.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.866+0000 7efc4a16d700 1 --2- 172.21.15.98:0/903266996 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7efc28066f30 0x7efc280693e0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7efc4416aa30 tx=0x7efc34008040 comp rx=0 tx=0).stop 2023-10-06T05:08:05.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.866+0000 7efc4a16d700 1 -- 172.21.15.98:0/903266996 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc440b35d0 msgr2=0x7efc44151410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:05.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.866+0000 7efc4a16d700 1 --2- 172.21.15.98:0/903266996 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc440b35d0 0x7efc44151410 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7efc3c00b6c0 tx=0x7efc3c018000 comp rx=0 tx=0).stop 2023-10-06T05:08:05.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.866+0000 7efc4a16d700 1 -- 172.21.15.98:0/903266996 shutdown_connections 2023-10-06T05:08:05.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.866+0000 7efc4a16d700 1 --2- 172.21.15.98:0/903266996 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efc44159440 0x7efc4416a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:05.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.866+0000 7efc4a16d700 1 --2- 172.21.15.98:0/903266996 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7efc28066f30 0x7efc280693e0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:05.869 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.866+0000 7efc4a16d700 1 --2- 172.21.15.98:0/903266996 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc440b35d0 0x7efc44151410 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:05.870 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.866+0000 7efc4a16d700 1 --2- 172.21.15.98:0/903266996 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efc440cc170 0x7efc44150e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:05.870 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.866+0000 7efc4a16d700 1 -- 172.21.15.98:0/903266996 >> 172.21.15.98:0/903266996 conn(0x7efc4406ece0 msgr2=0x7efc440b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:05.870 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.866+0000 7efc4a16d700 1 -- 172.21.15.98:0/903266996 shutdown_connections 2023-10-06T05:08:05.870 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.867+0000 7efc4a16d700 1 -- 172.21.15.98:0/903266996 wait complete. 2023-10-06T05:08:05.870 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 17 2023-10-06T05:08:05.881 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T05:08:05.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.992+0000 7f9300248700 1 Processor -- start 2023-10-06T05:08:05.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.992+0000 7f9300248700 1 -- start start 2023-10-06T05:08:05.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.993+0000 7f9300248700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f92f8159560 0x7f92f80aa530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:05.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.993+0000 7f9300248700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f92f80b0760 0x7f92f80b1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:05.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.993+0000 7f9300248700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92f80b0c60 con 0x7f92f80b49a0 2023-10-06T05:08:05.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.993+0000 7f9300248700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92f80b0f40 con 0x7f92f80b0760 2023-10-06T05:08:05.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.993+0000 7f9300248700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92f80b1220 con 0x7f92f8159560 2023-10-06T05:08:05.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.993+0000 7f92fe7e5700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f92f80b0760 0x7f92f80b1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:05.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.993+0000 7f92fe7e5700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f92f80b0760 0x7f92f80b1bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54962/0 (socket says 172.21.15.98:54962) 2023-10-06T05:08:05.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.993+0000 7f92fe7e5700 1 -- 172.21.15.98:0/1609710743 learned_addr learned my addr 172.21.15.98:0/1609710743 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:08:05.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.993+0000 7f92fdfe4700 1 --2- 172.21.15.98:0/1609710743 >> v2:172.21.15.174:3300/0 conn(0x7f92f8159560 0x7f92f80aa530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:05.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.993+0000 7f92fe7e5700 1 -- 172.21.15.98:0/1609710743 >> v2:172.21.15.174:3300/0 conn(0x7f92f8159560 msgr2=0x7f92f80aa530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:05.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.993+0000 7f92fe7e5700 1 --2- 172.21.15.98:0/1609710743 >> v2:172.21.15.174:3300/0 conn(0x7f92f8159560 0x7f92f80aa530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:05.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.993+0000 7f92fe7e5700 1 -- 172.21.15.98:0/1609710743 >> v1:172.21.15.174:6789/0 conn(0x7f92f80b49a0 legacy=0x7f92f80aaa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:05.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.993+0000 7f92fe7e5700 1 -- 172.21.15.98:0/1609710743 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92f80b1500 con 0x7f92f80b0760 2023-10-06T05:08:05.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.993+0000 7f92fe7e5700 1 --2- 172.21.15.98:0/1609710743 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92f80b0760 0x7f92f80b1bc0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f92e80099b0 tx=0x7f92e8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffeee07b37efeedc server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:05.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.993+0000 7f92fcfe2700 1 -- 172.21.15.98:0/1609710743 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92e8009ea0 con 0x7f92f80b0760 2023-10-06T05:08:05.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.994+0000 7f92fcfe2700 1 -- 172.21.15.98:0/1609710743 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92e800b820 con 0x7f92f80b0760 2023-10-06T05:08:05.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.994+0000 7f92fcfe2700 1 -- 172.21.15.98:0/1609710743 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92e800bbc0 con 0x7f92f80b0760 2023-10-06T05:08:05.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.994+0000 7f9300248700 1 -- 172.21.15.98:0/1609710743 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92f80b0760 msgr2=0x7f92f80b1bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:05.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.994+0000 7f9300248700 1 --2- 172.21.15.98:0/1609710743 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92f80b0760 0x7f92f80b1bc0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f92e80099b0 tx=0x7f92e8009250 comp rx=0 tx=0).stop 2023-10-06T05:08:05.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.994+0000 7f9300248700 1 -- 172.21.15.98:0/1609710743 shutdown_connections 2023-10-06T05:08:05.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.994+0000 7f9300248700 1 --2- 172.21.15.98:0/1609710743 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92f80b0760 0x7f92f80b1bc0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:05.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.994+0000 7f9300248700 1 --2- 172.21.15.98:0/1609710743 >> v2:172.21.15.174:3300/0 conn(0x7f92f8159560 0x7f92f80aa530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:05.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.994+0000 7f9300248700 1 -- 172.21.15.98:0/1609710743 >> 172.21.15.98:0/1609710743 conn(0x7f92f80997f0 msgr2=0x7f92f806f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:05.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.994+0000 7f9300248700 1 -- 172.21.15.98:0/1609710743 shutdown_connections 2023-10-06T05:08:05.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.994+0000 7f9300248700 1 -- 172.21.15.98:0/1609710743 wait complete. 2023-10-06T05:08:06.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.995+0000 7f9300248700 1 Processor -- start 2023-10-06T05:08:06.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.995+0000 7f9300248700 1 -- start start 2023-10-06T05:08:06.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.996+0000 7f9300248700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f92f80b0760 0x7f92f80aab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:06.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.996+0000 7f9300248700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f92f80b49a0 0x7f92f80aea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:06.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.996+0000 7f9300248700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92f8159560 0x7f92f80acbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:06.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.996+0000 7f9300248700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92f81599d0 con 0x7f92f8159560 2023-10-06T05:08:06.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.996+0000 7f9300248700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92f80abcb0 con 0x7f92f80b0760 2023-10-06T05:08:06.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.996+0000 7f9300248700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92f80abf50 con 0x7f92f80b49a0 2023-10-06T05:08:06.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.996+0000 7f92fe7e5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92f8159560 0x7f92f80acbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:06.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.996+0000 7f92fe7e5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92f8159560 0x7f92f80acbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54964/0 (socket says 172.21.15.98:54964) 2023-10-06T05:08:06.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.996+0000 7f92fe7e5700 1 -- 172.21.15.98:0/726934732 learned_addr learned my addr 172.21.15.98:0/726934732 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:08:06.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.996+0000 7f92fdfe4700 1 --2- 172.21.15.98:0/726934732 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f92f80b0760 0x7f92f80aab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:06.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.996+0000 7f92fd7e3700 1 --2- 172.21.15.98:0/726934732 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f92f80b49a0 0x7f92f80aea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:06.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.996+0000 7f92fe7e5700 1 -- 172.21.15.98:0/726934732 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f92f80b49a0 msgr2=0x7f92f80aea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:06.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.996+0000 7f92fe7e5700 1 --2- 172.21.15.98:0/726934732 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f92f80b49a0 0x7f92f80aea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:06.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.996+0000 7f92fe7e5700 1 -- 172.21.15.98:0/726934732 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f92f80b0760 msgr2=0x7f92f80aab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:06.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.996+0000 7f92fe7e5700 1 --2- 172.21.15.98:0/726934732 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f92f80b0760 0x7f92f80aab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:06.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.997+0000 7f92fe7e5700 1 -- 172.21.15.98:0/726934732 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92e80094f0 con 0x7f92f8159560 2023-10-06T05:08:06.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.997+0000 7f92fdfe4700 1 --2- 172.21.15.98:0/726934732 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f92f80b0760 0x7f92f80aab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:08:06.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.997+0000 7f92fe7e5700 1 --2- 172.21.15.98:0/726934732 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92f8159560 0x7f92f80acbe0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f92e8009290 tx=0x7f92e8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:06.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.997+0000 7f92f5ffb700 1 -- 172.21.15.98:0/726934732 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92e800bb60 con 0x7f92f8159560 2023-10-06T05:08:06.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.997+0000 7f92f5ffb700 1 -- 172.21.15.98:0/726934732 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92e8011630 con 0x7f92f8159560 2023-10-06T05:08:06.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.997+0000 7f9300248700 1 -- 172.21.15.98:0/726934732 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92f8159c40 con 0x7f92f8159560 2023-10-06T05:08:06.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.997+0000 7f92f5ffb700 1 -- 172.21.15.98:0/726934732 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92e80119f0 con 0x7f92f8159560 2023-10-06T05:08:06.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.997+0000 7f9300248700 1 -- 172.21.15.98:0/726934732 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92f80aef70 con 0x7f92f8159560 2023-10-06T05:08:06.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.998+0000 7f9300248700 1 -- 172.21.15.98:0/726934732 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92f8064a90 con 0x7f92f8159560 2023-10-06T05:08:06.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.998+0000 7f92f5ffb700 1 -- 172.21.15.98:0/726934732 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f92e80253f0 con 0x7f92f8159560 2023-10-06T05:08:06.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.999+0000 7f92f5ffb700 1 --2- 172.21.15.98:0/726934732 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f92e4066f60 0x7f92e4069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:06.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.999+0000 7f92f5ffb700 1 -- 172.21.15.98:0/726934732 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6378+0+0 (secure 0 0 0) 0x7f92e8086340 con 0x7f92f8159560 2023-10-06T05:08:06.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.999+0000 7f92fdfe4700 1 --2- 172.21.15.98:0/726934732 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f92e4066f60 0x7f92e4069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:06.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:05.999+0000 7f92fdfe4700 1 --2- 172.21.15.98:0/726934732 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f92e4066f60 0x7f92e4069410 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f92f8069fa0 tx=0x7f92f000b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:06.005 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.003+0000 7f92f5ffb700 1 -- 172.21.15.98:0/726934732 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f92e8053270 con 0x7f92f8159560 2023-10-06T05:08:06.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.188+0000 7f9300248700 1 -- 172.21.15.98:0/726934732 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f92f80b17e0 con 0x7f92f8159560 2023-10-06T05:08:06.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.189+0000 7f92f5ffb700 1 -- 172.21.15.98:0/726934732 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v43) v1 ==== 74+0+17071 (secure 0 0 0) 0x7f92e8056890 con 0x7f92f8159560 2023-10-06T05:08:06.191 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:08:06.191 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":43,"fsid":"f50c5df5-cad5-47a4-9bb1-8c791d4311ec","created":"2023-10-06T05:06:32.971175+0000","modified":"2023-10-06T05:07:57.482671+0000","last_up_change":"2023-10-06T05:06:44.016489+0000","last_in_change":"2023-10-06T05:06:35.544511+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-06T05:06:44.559671+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-06T05:06:51.432161+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-10-06T05:07:54.125452+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-10-06T05:07:54.831073+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d9495d95-da75-4c3b-8927-e9bae6a8b093","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":90627},{"type":"v1","addr":"172.21.15.98:6801","nonce":90627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":90627},{"type":"v1","addr":"172.21.15.98:6803","nonce":90627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":90627},{"type":"v1","addr":"172.21.15.98:6807","nonce":90627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":90627},{"type":"v1","addr":"172.21.15.98:6805","nonce":90627}]},"public_addr":"172.21.15.98:6801/90627","cluster_addr":"172.21.15.98:6803/90627","heartbeat_back_addr":"172.21.15.98:6807/90627","heartbeat_front_addr":"172.21.15.98:6805/90627","state":["exists","up"]},{"osd":1,"uuid":"f25ba5be-d65e-4707-a134-154092deb754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":90628},{"type":"v1","addr":"172.21.15.98:6809","nonce":90628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":90628},{"type":"v1","addr":"172.21.15.98:6811","nonce":90628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":90628},{"type":"v1","addr":"172.21.15.98:6815","nonce":90628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":90628},{"type":"v1","addr":"172.21.15.98:6813","nonce":90628}]},"public_addr":"172.21.15.98:6809/90628","cluster_addr":"172.21.15.98:6811/90628","heartbeat_back_addr":"172.21.15.98:6815/90628","heartbeat_front_addr":"172.21.15.98:6813/90628","state":["exists","up"]},{"osd":2,"uuid":"5ab9e028-945c-43e2-8c95-3b0b4cf38e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":90662},{"type":"v1","addr":"172.21.15.98:6818","nonce":90662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":90662},{"type":"v1","addr":"172.21.15.98:6822","nonce":90662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":90662},{"type":"v1","addr":"172.21.15.98:6830","nonce":90662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":90662},{"type":"v1","addr":"172.21.15.98:6826","nonce":90662}]},"public_addr":"172.21.15.98:6818/90662","cluster_addr":"172.21.15.98:6822/90662","heartbeat_back_addr":"172.21.15.98:6830/90662","heartbeat_front_addr":"172.21.15.98:6826/90662","state":["exists","up"]},{"osd":3,"uuid":"3bb803ff-1738-46bf-be62-2267bf378822","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6817","nonce":90663},{"type":"v1","addr":"172.21.15.98:6819","nonce":90663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6821","nonce":90663},{"type":"v1","addr":"172.21.15.98:6823","nonce":90663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6829","nonce":90663},{"type":"v1","addr":"172.21.15.98:6831","nonce":90663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6825","nonce":90663},{"type":"v1","addr":"172.21.15.98:6827","nonce":90663}]},"public_addr":"172.21.15.98:6819/90663","cluster_addr":"172.21.15.98:6823/90663","heartbeat_back_addr":"172.21.15.98:6831/90663","heartbeat_front_addr":"172.21.15.98:6827/90663","state":["exists","up"]},{"osd":4,"uuid":"7d57a4c9-6462-43f6-b64a-7f9e9b4353d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":90847},{"type":"v1","addr":"172.21.15.174:6801","nonce":90847}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":90847},{"type":"v1","addr":"172.21.15.174:6803","nonce":90847}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":90847},{"type":"v1","addr":"172.21.15.174:6807","nonce":90847}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":90847},{"type":"v1","addr":"172.21.15.174:6805","nonce":90847}]},"public_addr":"172.21.15.174:6801/90847","cluster_addr":"172.21.15.174:6803/90847","heartbeat_back_addr":"172.21.15.174:6807/90847","heartbeat_front_addr":"172.21.15.174:6805/90847","state":["exists","up"]},{"osd":5,"uuid":"83287ea4-e44d-4af9-bc77-36d47c60d4a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":90848},{"type":"v1","addr":"172.21.15.174:6809","nonce":90848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":90848},{"type":"v1","addr":"172.21.15.174:6811","nonce":90848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":90848},{"type":"v1","addr":"172.21.15.174:6815","nonce":90848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":90848},{"type":"v1","addr":"172.21.15.174:6813","nonce":90848}]},"public_addr":"172.21.15.174:6809/90848","cluster_addr":"172.21.15.174:6811/90848","heartbeat_back_addr":"172.21.15.174:6815/90848","heartbeat_front_addr":"172.21.15.174:6813/90848","state":["exists","up"]},{"osd":6,"uuid":"496a6272-892e-43c4-a145-206623c22f99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":90877},{"type":"v1","addr":"172.21.15.174:6817","nonce":90877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":90877},{"type":"v1","addr":"172.21.15.174:6819","nonce":90877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":90877},{"type":"v1","addr":"172.21.15.174:6823","nonce":90877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":90877},{"type":"v1","addr":"172.21.15.174:6821","nonce":90877}]},"public_addr":"172.21.15.174:6817/90877","cluster_addr":"172.21.15.174:6819/90877","heartbeat_back_addr":"172.21.15.174:6823/90877","heartbeat_front_addr":"172.21.15.174:6821/90877","state":["exists","up"]},{"osd":7,"uuid":"49967e34-8b07-4bab-a13e-0c16d6a0f3a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":90878},{"type":"v1","addr":"172.21.15.174:6825","nonce":90878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":90878},{"type":"v1","addr":"172.21.15.174:6827","nonce":90878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":90878},{"type":"v1","addr":"172.21.15.174:6831","nonce":90878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":90878},{"type":"v1","addr":"172.21.15.174:6829","nonce":90878}]},"public_addr":"172.21.15.174:6825/90878","cluster_addr":"172.21.15.174:6827/90878","heartbeat_back_addr":"172.21.15.174:6831/90878","heartbeat_front_addr":"172.21.15.174:6829/90878","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.446479+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.514266+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.794108+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.810894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.731919+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.717756+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.737435+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.508682+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T05:08:06.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.190+0000 7f9300248700 1 -- 172.21.15.98:0/726934732 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f92e4066f60 msgr2=0x7f92e4069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:06.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.190+0000 7f9300248700 1 --2- 172.21.15.98:0/726934732 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f92e4066f60 0x7f92e4069410 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f92f8069fa0 tx=0x7f92f000b040 comp rx=0 tx=0).stop 2023-10-06T05:08:06.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.191+0000 7f9300248700 1 -- 172.21.15.98:0/726934732 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92f8159560 msgr2=0x7f92f80acbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:06.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.191+0000 7f9300248700 1 --2- 172.21.15.98:0/726934732 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92f8159560 0x7f92f80acbe0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f92e8009290 tx=0x7f92e8017000 comp rx=0 tx=0).stop 2023-10-06T05:08:06.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.191+0000 7f9300248700 1 -- 172.21.15.98:0/726934732 shutdown_connections 2023-10-06T05:08:06.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.191+0000 7f9300248700 1 --2- 172.21.15.98:0/726934732 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f92e4066f60 0x7f92e4069410 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:06.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.191+0000 7f9300248700 1 --2- 172.21.15.98:0/726934732 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f92f8159560 0x7f92f80acbe0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:06.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.191+0000 7f9300248700 1 --2- 172.21.15.98:0/726934732 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f92f80b0760 0x7f92f80aab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:06.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.191+0000 7f9300248700 1 --2- 172.21.15.98:0/726934732 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f92f80b49a0 0x7f92f80aea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:06.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.191+0000 7f9300248700 1 -- 172.21.15.98:0/726934732 >> 172.21.15.98:0/726934732 conn(0x7f92f80997f0 msgr2=0x7f92f80cce90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:06.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.191+0000 7f9300248700 1 -- 172.21.15.98:0/726934732 shutdown_connections 2023-10-06T05:08:06.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.191+0000 7f9300248700 1 -- 172.21.15.98:0/726934732 wait complete. 2023-10-06T05:08:06.205 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-06T05:06:44.559671+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-10-06T05:06:51.432161+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-06T05:07:54.125452+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '42', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2023-10-06T05:07:54.831073+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '43', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-06T05:08:06.205 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-06T05:08:06.322 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.320+0000 7fbb95a08700 1 Processor -- start 2023-10-06T05:08:06.322 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.320+0000 7fbb95a08700 1 -- start start 2023-10-06T05:08:06.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.321+0000 7fbb95a08700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbb901594d0 0x7fbb900cb560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:06.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.321+0000 7fbb95a08700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fbb901598a0 0x7fbb900cbaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:06.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.321+0000 7fbb95a08700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb900cf310 con 0x7fbb900ce7e0 2023-10-06T05:08:06.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.321+0000 7fbb95a08700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb900cf5d0 con 0x7fbb901598a0 2023-10-06T05:08:06.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.321+0000 7fbb95a08700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb900cf8b0 con 0x7fbb901594d0 2023-10-06T05:08:06.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.321+0000 7fbb867fc700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fbb901598a0 0x7fbb900cbaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:06.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.321+0000 7fbb8f7fe700 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fbb900ce7e0 0x7fbb900cec00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.98:42062/0 (socket says 172.21.15.98:42062) 2023-10-06T05:08:06.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.321+0000 7fbb8f7fe700 1 -- 172.21.15.98:0/3794646030 learned_addr learned my addr 172.21.15.98:0/3794646030 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:08:06.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.321+0000 7fbb8effd700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbb901594d0 0x7fbb900cb560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:06.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.322+0000 7fbb867fc700 1 -- 172.21.15.98:0/3794646030 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbb901594d0 msgr2=0x7fbb900cb560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:06.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.322+0000 7fbb867fc700 1 --2- 172.21.15.98:0/3794646030 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbb901594d0 0x7fbb900cb560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:06.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.322+0000 7fbb867fc700 1 -- 172.21.15.98:0/3794646030 >> v1:172.21.15.174:6789/0 conn(0x7fbb900ce7e0 legacy=0x7fbb900cec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:06.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.322+0000 7fbb8e7fc700 1 -- 172.21.15.98:0/3794646030 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2554799409 0 0) 0x7fbb900cf310 con 0x7fbb900ce7e0 2023-10-06T05:08:06.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.322+0000 7fbb867fc700 1 -- 172.21.15.98:0/3794646030 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb900cfb90 con 0x7fbb901598a0 2023-10-06T05:08:06.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.322+0000 7fbb867fc700 1 --2- 172.21.15.98:0/3794646030 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb901598a0 0x7fbb900cbaa0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fbb780098e0 tx=0x7fbb780091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=80c40948896d1815 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:06.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.322+0000 7fbb8e7fc700 1 -- 172.21.15.98:0/3794646030 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb7800b640 con 0x7fbb901598a0 2023-10-06T05:08:06.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.322+0000 7fbb8e7fc700 1 -- 172.21.15.98:0/3794646030 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb7800b7a0 con 0x7fbb901598a0 2023-10-06T05:08:06.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.322+0000 7fbb8e7fc700 1 -- 172.21.15.98:0/3794646030 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb7800bb40 con 0x7fbb901598a0 2023-10-06T05:08:06.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.322+0000 7fbb95a08700 1 -- 172.21.15.98:0/3794646030 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb901598a0 msgr2=0x7fbb900cbaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:06.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.322+0000 7fbb95a08700 1 --2- 172.21.15.98:0/3794646030 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb901598a0 0x7fbb900cbaa0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fbb780098e0 tx=0x7fbb780091a0 comp rx=0 tx=0).stop 2023-10-06T05:08:06.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.322+0000 7fbb95a08700 1 -- 172.21.15.98:0/3794646030 shutdown_connections 2023-10-06T05:08:06.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.322+0000 7fbb95a08700 1 --2- 172.21.15.98:0/3794646030 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbb901594d0 0x7fbb900cb560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:06.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.322+0000 7fbb95a08700 1 --2- 172.21.15.98:0/3794646030 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb901598a0 0x7fbb900cbaa0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:06.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.322+0000 7fbb95a08700 1 -- 172.21.15.98:0/3794646030 >> 172.21.15.98:0/3794646030 conn(0x7fbb90089e50 msgr2=0x7fbb90099770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:06.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.323+0000 7fbb95a08700 1 -- 172.21.15.98:0/3794646030 shutdown_connections 2023-10-06T05:08:06.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.323+0000 7fbb95a08700 1 -- 172.21.15.98:0/3794646030 wait complete. 2023-10-06T05:08:06.330 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.324+0000 7fbb95a08700 1 Processor -- start 2023-10-06T05:08:06.330 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.324+0000 7fbb95a08700 1 -- start start 2023-10-06T05:08:06.330 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.324+0000 7fbb95a08700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbb900ce7e0 0x7fbb90158400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:06.330 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.324+0000 7fbb95a08700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb901594d0 0x7fbb90159cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:06.330 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.324+0000 7fbb95a08700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb901598a0 0x7fbb901494a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:06.330 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.324+0000 7fbb95a08700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb9015ad00 con 0x7fbb901598a0 2023-10-06T05:08:06.330 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.324+0000 7fbb95a08700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb9015a200 con 0x7fbb901594d0 2023-10-06T05:08:06.331 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.324+0000 7fbb95a08700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb9015a460 con 0x7fbb900ce7e0 2023-10-06T05:08:06.331 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.325+0000 7fbb8f7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb901598a0 0x7fbb901494a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:06.331 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.325+0000 7fbb8f7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb901598a0 0x7fbb901494a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54976/0 (socket says 172.21.15.98:54976) 2023-10-06T05:08:06.331 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.325+0000 7fbb8f7fe700 1 -- 172.21.15.98:0/197208648 learned_addr learned my addr 172.21.15.98:0/197208648 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:08:06.331 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.325+0000 7fbb867fc700 1 --2- 172.21.15.98:0/197208648 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb901594d0 0x7fbb90159cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:06.331 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.325+0000 7fbb8effd700 1 --2- 172.21.15.98:0/197208648 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbb900ce7e0 0x7fbb90158400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:06.331 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.325+0000 7fbb8f7fe700 1 -- 172.21.15.98:0/197208648 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbb900ce7e0 msgr2=0x7fbb90158400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:06.331 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.325+0000 7fbb8f7fe700 1 --2- 172.21.15.98:0/197208648 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbb900ce7e0 0x7fbb90158400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:06.331 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.325+0000 7fbb8f7fe700 1 -- 172.21.15.98:0/197208648 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb901594d0 msgr2=0x7fbb90159cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:06.331 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.325+0000 7fbb8f7fe700 1 --2- 172.21.15.98:0/197208648 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb901594d0 0x7fbb90159cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:06.331 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.325+0000 7fbb8f7fe700 1 -- 172.21.15.98:0/197208648 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb78009420 con 0x7fbb901598a0 2023-10-06T05:08:06.332 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.325+0000 7fbb8f7fe700 1 --2- 172.21.15.98:0/197208648 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb901598a0 0x7fbb901494a0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fbb8800cc90 tx=0x7fbb88001a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:06.332 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.325+0000 7fbb877fe700 1 -- 172.21.15.98:0/197208648 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb8800cea0 con 0x7fbb901598a0 2023-10-06T05:08:06.332 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.325+0000 7fbb877fe700 1 -- 172.21.15.98:0/197208648 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb8800ab70 con 0x7fbb901598a0 2023-10-06T05:08:06.332 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.325+0000 7fbb95a08700 1 -- 172.21.15.98:0/197208648 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb9015a740 con 0x7fbb901598a0 2023-10-06T05:08:06.332 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.325+0000 7fbb877fe700 1 -- 172.21.15.98:0/197208648 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb880070f0 con 0x7fbb901598a0 2023-10-06T05:08:06.332 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.325+0000 7fbb95a08700 1 -- 172.21.15.98:0/197208648 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb90153670 con 0x7fbb901598a0 2023-10-06T05:08:06.332 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.326+0000 7fbb95a08700 1 -- 172.21.15.98:0/197208648 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb900cc1c0 con 0x7fbb901598a0 2023-10-06T05:08:06.332 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.326+0000 7fbb877fe700 1 -- 172.21.15.98:0/197208648 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fbb880074a0 con 0x7fbb901598a0 2023-10-06T05:08:06.332 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.327+0000 7fbb877fe700 1 --2- 172.21.15.98:0/197208648 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fbb70066f30 0x7fbb700693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:06.332 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.327+0000 7fbb877fe700 1 -- 172.21.15.98:0/197208648 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6378+0+0 (secure 0 0 0) 0x7fbb88084c80 con 0x7fbb901598a0 2023-10-06T05:08:06.332 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.327+0000 7fbb8effd700 1 --2- 172.21.15.98:0/197208648 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fbb70066f30 0x7fbb700693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:06.333 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.327+0000 7fbb8effd700 1 --2- 172.21.15.98:0/197208648 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fbb70066f30 0x7fbb700693e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fbb800093f0 tx=0x7fbb80008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:06.333 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.331+0000 7fbb877fe700 1 -- 172.21.15.98:0/197208648 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbb88051cd0 con 0x7fbb901598a0 2023-10-06T05:08:06.516 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.514+0000 7fbb95a08700 1 -- 172.21.15.98:0/197208648 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbb900c53b0 con 0x7fbb901598a0 2023-10-06T05:08:06.517 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.515+0000 7fbb877fe700 1 -- 172.21.15.98:0/197208648 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v43) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbb880552f0 con 0x7fbb901598a0 2023-10-06T05:08:06.517 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2023-10-06T05:08:06.519 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.517+0000 7fbb95a08700 1 -- 172.21.15.98:0/197208648 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fbb70066f30 msgr2=0x7fbb700693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:06.519 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.517+0000 7fbb95a08700 1 --2- 172.21.15.98:0/197208648 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fbb70066f30 0x7fbb700693e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fbb800093f0 tx=0x7fbb80008040 comp rx=0 tx=0).stop 2023-10-06T05:08:06.519 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.517+0000 7fbb95a08700 1 -- 172.21.15.98:0/197208648 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb901598a0 msgr2=0x7fbb901494a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:06.519 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.517+0000 7fbb95a08700 1 --2- 172.21.15.98:0/197208648 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb901598a0 0x7fbb901494a0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fbb8800cc90 tx=0x7fbb88001a80 comp rx=0 tx=0).stop 2023-10-06T05:08:06.519 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.517+0000 7fbb95a08700 1 -- 172.21.15.98:0/197208648 shutdown_connections 2023-10-06T05:08:06.519 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.517+0000 7fbb95a08700 1 --2- 172.21.15.98:0/197208648 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fbb70066f30 0x7fbb700693e0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:06.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.517+0000 7fbb95a08700 1 --2- 172.21.15.98:0/197208648 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbb901598a0 0x7fbb901494a0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:06.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.517+0000 7fbb95a08700 1 --2- 172.21.15.98:0/197208648 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbb900ce7e0 0x7fbb90158400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:06.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.517+0000 7fbb95a08700 1 --2- 172.21.15.98:0/197208648 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb901594d0 0x7fbb90159cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:06.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.517+0000 7fbb95a08700 1 -- 172.21.15.98:0/197208648 >> 172.21.15.98:0/197208648 conn(0x7fbb90089e50 msgr2=0x7fbb900c6420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:06.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.517+0000 7fbb95a08700 1 -- 172.21.15.98:0/197208648 shutdown_connections 2023-10-06T05:08:06.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.517+0000 7fbb95a08700 1 -- 172.21.15.98:0/197208648 wait complete. 2023-10-06T05:08:06.531 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-10-06T05:08:06.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.646+0000 7f13fce1d700 1 Processor -- start 2023-10-06T05:08:06.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.646+0000 7f13fce1d700 1 -- start start 2023-10-06T05:08:06.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.647+0000 7f13fce1d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f13f81594d0 0x7f13f80cb560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:06.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.647+0000 7f13fce1d700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f13f80cbaa0 0x7f13f80ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:06.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.647+0000 7f13fce1d700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13f80cbeb0 con 0x7f13f81598a0 2023-10-06T05:08:06.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.647+0000 7f13fce1d700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13f80cf5d0 con 0x7f13f81594d0 2023-10-06T05:08:06.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.647+0000 7f13fce1d700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13f80cf8b0 con 0x7f13f80cbaa0 2023-10-06T05:08:06.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.647+0000 7f13f659c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f13f81594d0 0x7f13f80cb560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:06.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.647+0000 7f13f5d9b700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f13f81598a0 0x7f13f80ce7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:38396/0 (socket says 172.21.15.98:38396) 2023-10-06T05:08:06.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.647+0000 7f13f5d9b700 1 -- 172.21.15.98:0/337090915 learned_addr learned my addr 172.21.15.98:0/337090915 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:08:06.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.647+0000 7f13f6d9d700 1 --2- 172.21.15.98:0/337090915 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f13f80cbaa0 0x7f13f80ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:06.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.647+0000 7f13f559a700 1 -- 172.21.15.98:0/337090915 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4210202784 0 0) 0x7f13f80cbeb0 con 0x7f13f81598a0 2023-10-06T05:08:06.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.647+0000 7f13f559a700 1 -- 172.21.15.98:0/337090915 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13e0003570 con 0x7f13f81598a0 2023-10-06T05:08:06.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.647+0000 7f13f659c700 1 -- 172.21.15.98:0/337090915 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f13f80cbaa0 msgr2=0x7f13f80ceef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:06.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.647+0000 7f13f659c700 1 --2- 172.21.15.98:0/337090915 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f13f80cbaa0 0x7f13f80ceef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:06.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.647+0000 7f13f659c700 1 -- 172.21.15.98:0/337090915 >> v1:172.21.15.98:6789/0 conn(0x7f13f81598a0 legacy=0x7f13f80ce7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:06.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.647+0000 7f13f659c700 1 -- 172.21.15.98:0/337090915 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13f80cfb90 con 0x7f13f81594d0 2023-10-06T05:08:06.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.647+0000 7f13f659c700 1 --2- 172.21.15.98:0/337090915 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13f81594d0 0x7f13f80cb560 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f13f00098e0 tx=0x7f13f00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f10aba5a2a90882 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:06.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.647+0000 7f13f559a700 1 -- 172.21.15.98:0/337090915 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13f0011630 con 0x7f13f81594d0 2023-10-06T05:08:06.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.647+0000 7f13f559a700 1 -- 172.21.15.98:0/337090915 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13f0011790 con 0x7f13f81594d0 2023-10-06T05:08:06.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.647+0000 7f13f559a700 1 -- 172.21.15.98:0/337090915 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13f0011b10 con 0x7f13f81594d0 2023-10-06T05:08:06.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.648+0000 7f13fce1d700 1 -- 172.21.15.98:0/337090915 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13f81594d0 msgr2=0x7f13f80cb560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:06.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.648+0000 7f13fce1d700 1 --2- 172.21.15.98:0/337090915 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13f81594d0 0x7f13f80cb560 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f13f00098e0 tx=0x7f13f00091a0 comp rx=0 tx=0).stop 2023-10-06T05:08:06.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.648+0000 7f13fce1d700 1 -- 172.21.15.98:0/337090915 shutdown_connections 2023-10-06T05:08:06.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.648+0000 7f13fce1d700 1 --2- 172.21.15.98:0/337090915 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f13f80cbaa0 0x7f13f80ceef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:06.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.648+0000 7f13fce1d700 1 --2- 172.21.15.98:0/337090915 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13f81594d0 0x7f13f80cb560 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:06.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.648+0000 7f13fce1d700 1 -- 172.21.15.98:0/337090915 >> 172.21.15.98:0/337090915 conn(0x7f13f8089e50 msgr2=0x7f13f8099770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:06.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.648+0000 7f13fce1d700 1 -- 172.21.15.98:0/337090915 shutdown_connections 2023-10-06T05:08:06.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.648+0000 7f13fce1d700 1 -- 172.21.15.98:0/337090915 wait complete. 2023-10-06T05:08:06.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.649+0000 7f13fce1d700 1 Processor -- start 2023-10-06T05:08:06.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.649+0000 7f13fce1d700 1 -- start start 2023-10-06T05:08:06.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.650+0000 7f13fce1d700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f13f80cbaa0 0x7f13f80c89a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:06.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.650+0000 7f13fce1d700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f13f81594d0 0x7f13f814f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:06.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.650+0000 7f13fce1d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13f81598a0 0x7f13f809d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:06.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.650+0000 7f13fce1d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13f816a1d0 con 0x7f13f81598a0 2023-10-06T05:08:06.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.650+0000 7f13fce1d700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13f816a4b0 con 0x7f13f80cbaa0 2023-10-06T05:08:06.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.650+0000 7f13fce1d700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13f816a790 con 0x7f13f81594d0 2023-10-06T05:08:06.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.650+0000 7f13f6d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13f81598a0 0x7f13f809d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:06.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.650+0000 7f13f6d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13f81598a0 0x7f13f809d980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54998/0 (socket says 172.21.15.98:54998) 2023-10-06T05:08:06.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.650+0000 7f13f6d9d700 1 -- 172.21.15.98:0/18303264 learned_addr learned my addr 172.21.15.98:0/18303264 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:08:06.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.650+0000 7f13f659c700 1 --2- 172.21.15.98:0/18303264 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f13f80cbaa0 0x7f13f80c89a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:06.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.650+0000 7f13f5d9b700 1 --2- 172.21.15.98:0/18303264 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f13f81594d0 0x7f13f814f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:06.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.650+0000 7f13f6d9d700 1 -- 172.21.15.98:0/18303264 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f13f81594d0 msgr2=0x7f13f814f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:06.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.650+0000 7f13f6d9d700 1 --2- 172.21.15.98:0/18303264 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f13f81594d0 0x7f13f814f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:06.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.650+0000 7f13f6d9d700 1 -- 172.21.15.98:0/18303264 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f13f80cbaa0 msgr2=0x7f13f80c89a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:06.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.650+0000 7f13f6d9d700 1 --2- 172.21.15.98:0/18303264 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f13f80cbaa0 0x7f13f80c89a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:06.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.650+0000 7f13f6d9d700 1 -- 172.21.15.98:0/18303264 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13f0009420 con 0x7f13f81598a0 2023-10-06T05:08:06.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.650+0000 7f13f659c700 1 --2- 172.21.15.98:0/18303264 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f13f80cbaa0 0x7f13f80c89a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:08:06.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.651+0000 7f13f6d9d700 1 --2- 172.21.15.98:0/18303264 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13f81598a0 0x7f13f809d980 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f13e800ea50 tx=0x7f13e80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:06.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.651+0000 7f13e67fc700 1 -- 172.21.15.98:0/18303264 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13e800c920 con 0x7f13f81598a0 2023-10-06T05:08:06.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.651+0000 7f13e67fc700 1 -- 172.21.15.98:0/18303264 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13e800caa0 con 0x7f13f81598a0 2023-10-06T05:08:06.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.651+0000 7f13fce1d700 1 -- 172.21.15.98:0/18303264 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13f816aa70 con 0x7f13f81598a0 2023-10-06T05:08:06.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.651+0000 7f13e67fc700 1 -- 172.21.15.98:0/18303264 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13e800ce60 con 0x7f13f81598a0 2023-10-06T05:08:06.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.651+0000 7f13fce1d700 1 -- 172.21.15.98:0/18303264 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13f8157a90 con 0x7f13f81598a0 2023-10-06T05:08:06.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.652+0000 7f13fce1d700 1 -- 172.21.15.98:0/18303264 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13f8157d40 con 0x7f13f81598a0 2023-10-06T05:08:06.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.652+0000 7f13e67fc700 1 -- 172.21.15.98:0/18303264 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f13e8013680 con 0x7f13f81598a0 2023-10-06T05:08:06.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.652+0000 7f13e67fc700 1 --2- 172.21.15.98:0/18303264 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f13d8066f30 0x7f13d80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:06.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.653+0000 7f13e67fc700 1 -- 172.21.15.98:0/18303264 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6378+0+0 (secure 0 0 0) 0x7f13e8084bf0 con 0x7f13f81598a0 2023-10-06T05:08:06.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.653+0000 7f13f659c700 1 --2- 172.21.15.98:0/18303264 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f13d8066f30 0x7f13d80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:06.659 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.653+0000 7f13f659c700 1 --2- 172.21.15.98:0/18303264 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f13d8066f30 0x7f13d80693e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f13f00098e0 tx=0x7f13f0017000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:06.659 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.657+0000 7f13e67fc700 1 -- 172.21.15.98:0/18303264 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f13f8157d40 con 0x7f13f81598a0 2023-10-06T05:08:06.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.839+0000 7f13fce1d700 1 -- 172.21.15.98:0/18303264 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f13f8157d40 con 0x7f13f81598a0 2023-10-06T05:08:06.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.839+0000 7f13e67fc700 1 -- 172.21.15.98:0/18303264 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v43) v1 ==== 92+0+10 (secure 0 0 0) 0x7f13f8157d40 con 0x7f13f81598a0 2023-10-06T05:08:06.842 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 8 2023-10-06T05:08:06.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.841+0000 7f13fce1d700 1 -- 172.21.15.98:0/18303264 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f13d8066f30 msgr2=0x7f13d80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:06.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.841+0000 7f13fce1d700 1 --2- 172.21.15.98:0/18303264 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f13d8066f30 0x7f13d80693e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f13f00098e0 tx=0x7f13f0017000 comp rx=0 tx=0).stop 2023-10-06T05:08:06.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.841+0000 7f13fce1d700 1 -- 172.21.15.98:0/18303264 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13f81598a0 msgr2=0x7f13f809d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:06.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.841+0000 7f13fce1d700 1 --2- 172.21.15.98:0/18303264 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13f81598a0 0x7f13f809d980 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f13e800ea50 tx=0x7f13e80091a0 comp rx=0 tx=0).stop 2023-10-06T05:08:06.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.841+0000 7f13fce1d700 1 -- 172.21.15.98:0/18303264 shutdown_connections 2023-10-06T05:08:06.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.841+0000 7f13fce1d700 1 --2- 172.21.15.98:0/18303264 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f13d8066f30 0x7f13d80693e0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:06.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.841+0000 7f13fce1d700 1 --2- 172.21.15.98:0/18303264 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13f81598a0 0x7f13f809d980 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:06.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.841+0000 7f13fce1d700 1 --2- 172.21.15.98:0/18303264 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f13f80cbaa0 0x7f13f80c89a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:06.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.841+0000 7f13fce1d700 1 --2- 172.21.15.98:0/18303264 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f13f81594d0 0x7f13f814f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:06.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.841+0000 7f13fce1d700 1 -- 172.21.15.98:0/18303264 >> 172.21.15.98:0/18303264 conn(0x7f13f8089e50 msgr2=0x7f13f80693f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:06.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.841+0000 7f13fce1d700 1 -- 172.21.15.98:0/18303264 shutdown_connections 2023-10-06T05:08:06.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.841+0000 7f13fce1d700 1 -- 172.21.15.98:0/18303264 wait complete. 2023-10-06T05:08:06.855 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-06T05:08:06.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.970+0000 7fcae64c5700 1 Processor -- start 2023-10-06T05:08:06.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.970+0000 7fcae64c5700 1 -- start start 2023-10-06T05:08:06.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.970+0000 7fcae64c5700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fcae01594d0 0x7fcae00cb560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:06.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.971+0000 7fcae64c5700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcae01598a0 0x7fcae00cbaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:06.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.971+0000 7fcae64c5700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcae00cf310 con 0x7fcae00ce7e0 2023-10-06T05:08:06.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.971+0000 7fcae64c5700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcae00cf5d0 con 0x7fcae01594d0 2023-10-06T05:08:06.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.971+0000 7fcae64c5700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcae00cf8b0 con 0x7fcae01598a0 2023-10-06T05:08:06.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.971+0000 7fcae4a62700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fcae00ce7e0 0x7fcae00cec00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:38398/0 (socket says 172.21.15.98:38398) 2023-10-06T05:08:06.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.971+0000 7fcae4a62700 1 -- 172.21.15.98:0/1602708087 learned_addr learned my addr 172.21.15.98:0/1602708087 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:08:06.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.971+0000 7fcadeffd700 1 -- 172.21.15.98:0/1602708087 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1842769417 0 0) 0x7fcae00cf310 con 0x7fcae00ce7e0 2023-10-06T05:08:06.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.971+0000 7fcadffff700 1 --2- 172.21.15.98:0/1602708087 >> v2:172.21.15.174:3300/0 conn(0x7fcae01594d0 0x7fcae00cb560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:06.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.971+0000 7fcadf7fe700 1 --2- 172.21.15.98:0/1602708087 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcae01598a0 0x7fcae00cbaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:06.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.971+0000 7fcadeffd700 1 -- 172.21.15.98:0/1602708087 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcac8003570 con 0x7fcae00ce7e0 2023-10-06T05:08:06.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.971+0000 7fcadeffd700 1 -- 172.21.15.98:0/1602708087 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3342952445 0 0) 0x7fcac8003570 con 0x7fcae00ce7e0 2023-10-06T05:08:06.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.971+0000 7fcadeffd700 1 -- 172.21.15.98:0/1602708087 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcae01598a0 msgr2=0x7fcae00cbaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:06.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.971+0000 7fcadeffd700 1 --2- 172.21.15.98:0/1602708087 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcae01598a0 0x7fcae00cbaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:06.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.971+0000 7fcadeffd700 1 -- 172.21.15.98:0/1602708087 >> v2:172.21.15.174:3300/0 conn(0x7fcae01594d0 msgr2=0x7fcae00cb560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:06.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.971+0000 7fcadeffd700 1 --2- 172.21.15.98:0/1602708087 >> v2:172.21.15.174:3300/0 conn(0x7fcae01594d0 0x7fcae00cb560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:06.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.971+0000 7fcadeffd700 1 -- 172.21.15.98:0/1602708087 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcae00cfb90 con 0x7fcae00ce7e0 2023-10-06T05:08:06.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.971+0000 7fcadeffd700 1 -- 172.21.15.98:0/1602708087 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1543991876 0 0) 0x7fcad80017f0 con 0x7fcae00ce7e0 2023-10-06T05:08:06.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.971+0000 7fcadeffd700 1 -- 172.21.15.98:0/1602708087 >> v1:172.21.15.98:6789/0 conn(0x7fcae00ce7e0 legacy=0x7fcae00cec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:06.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.971+0000 7fcadeffd700 1 --2- 172.21.15.98:0/1602708087 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcac8003a60 0x7fcac8003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:06.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.971+0000 7fcadeffd700 1 --2- 172.21.15.98:0/1602708087 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcac8004810 0x7fcac8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:06.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.971+0000 7fcadeffd700 1 --2- 172.21.15.98:0/1602708087 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcae01598a0 0x7fcac80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:06.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.971+0000 7fcadeffd700 1 -- 172.21.15.98:0/1602708087 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcae00cf5d0 con 0x7fcac8003a60 2023-10-06T05:08:06.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.971+0000 7fcadeffd700 1 -- 172.21.15.98:0/1602708087 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcae00cf8b0 con 0x7fcac8004810 2023-10-06T05:08:06.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.971+0000 7fcadeffd700 1 -- 172.21.15.98:0/1602708087 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcac800ba60 con 0x7fcae01598a0 2023-10-06T05:08:06.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.971+0000 7fcadffff700 1 --2- 172.21.15.98:0/1602708087 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcac8003a60 0x7fcac8003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:06.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.971+0000 7fcadf7fe700 1 --2- 172.21.15.98:0/1602708087 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcac8004810 0x7fcac8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:06.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.972+0000 7fcae4a62700 1 --2- 172.21.15.98:0/1602708087 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcae01598a0 0x7fcac80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:06.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.972+0000 7fcadffff700 1 -- 172.21.15.98:0/1602708087 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcae01598a0 msgr2=0x7fcac80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:06.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.972+0000 7fcadffff700 1 --2- 172.21.15.98:0/1602708087 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcae01598a0 0x7fcac80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:06.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.972+0000 7fcadffff700 1 -- 172.21.15.98:0/1602708087 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcac8004810 msgr2=0x7fcac8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:06.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.972+0000 7fcadffff700 1 --2- 172.21.15.98:0/1602708087 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcac8004810 0x7fcac8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:06.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.972+0000 7fcadffff700 1 -- 172.21.15.98:0/1602708087 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcac8003570 con 0x7fcac8003a60 2023-10-06T05:08:06.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.972+0000 7fcae4a62700 1 --2- 172.21.15.98:0/1602708087 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcae01598a0 0x7fcac80051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:08:06.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.972+0000 7fcadffff700 1 --2- 172.21.15.98:0/1602708087 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcac8003a60 0x7fcac8003e70 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fcad0009c30 tx=0x7fcad00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fac8e17b49773394 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:06.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.972+0000 7fcadeffd700 1 -- 172.21.15.98:0/1602708087 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcad000e710 con 0x7fcac8003a60 2023-10-06T05:08:06.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.972+0000 7fcadeffd700 1 -- 172.21.15.98:0/1602708087 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcad000e870 con 0x7fcac8003a60 2023-10-06T05:08:06.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.972+0000 7fcadeffd700 1 -- 172.21.15.98:0/1602708087 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcad000ec30 con 0x7fcac8003a60 2023-10-06T05:08:06.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.972+0000 7fcae64c5700 1 -- 172.21.15.98:0/1602708087 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcac8003a60 msgr2=0x7fcac8003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:06.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.972+0000 7fcae64c5700 1 --2- 172.21.15.98:0/1602708087 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcac8003a60 0x7fcac8003e70 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fcad0009c30 tx=0x7fcad00091a0 comp rx=0 tx=0).stop 2023-10-06T05:08:06.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.972+0000 7fcae4a62700 1 -- 172.21.15.98:0/1602708087 reap_dead start 2023-10-06T05:08:06.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.973+0000 7fcae64c5700 1 -- 172.21.15.98:0/1602708087 shutdown_connections 2023-10-06T05:08:06.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.973+0000 7fcae64c5700 1 -- 172.21.15.98:0/1602708087 >> 172.21.15.98:0/1602708087 conn(0x7fcae0089e50 msgr2=0x7fcae0099770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:06.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.973+0000 7fcae64c5700 1 -- 172.21.15.98:0/1602708087 shutdown_connections 2023-10-06T05:08:06.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.973+0000 7fcae64c5700 1 -- 172.21.15.98:0/1602708087 wait complete. 2023-10-06T05:08:06.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.974+0000 7fcae64c5700 1 Processor -- start 2023-10-06T05:08:06.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.974+0000 7fcae64c5700 1 -- start start 2023-10-06T05:08:06.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.974+0000 7fcae64c5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcac8003a60 0x7fcae00c8280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:06.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.974+0000 7fcae64c5700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcae0154f70 0x7fcae0157900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:06.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.975+0000 7fcae64c5700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcae016d490 0x7fcae0151df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:06.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.975+0000 7fcae64c5700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcae00b3730 con 0x7fcac8003a60 2023-10-06T05:08:06.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.975+0000 7fcae64c5700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcae00c87c0 con 0x7fcae0154f70 2023-10-06T05:08:06.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.975+0000 7fcae64c5700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcae01581b0 con 0x7fcae016d490 2023-10-06T05:08:06.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.975+0000 7fcadffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcac8003a60 0x7fcae00c8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:06.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.975+0000 7fcadffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcac8003a60 0x7fcae00c8280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55020/0 (socket says 172.21.15.98:55020) 2023-10-06T05:08:06.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.975+0000 7fcadffff700 1 -- 172.21.15.98:0/419627721 learned_addr learned my addr 172.21.15.98:0/419627721 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:08:06.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.975+0000 7fcadf7fe700 1 --2- 172.21.15.98:0/419627721 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcae0154f70 0x7fcae0157900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:06.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.975+0000 7fcae4a62700 1 --2- 172.21.15.98:0/419627721 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcae016d490 0x7fcae0151df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:06.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.975+0000 7fcadffff700 1 -- 172.21.15.98:0/419627721 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcae016d490 msgr2=0x7fcae0151df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:06.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.975+0000 7fcadffff700 1 --2- 172.21.15.98:0/419627721 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcae016d490 0x7fcae0151df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:06.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.975+0000 7fcadffff700 1 -- 172.21.15.98:0/419627721 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcae0154f70 msgr2=0x7fcae0157900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:06.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.975+0000 7fcadffff700 1 --2- 172.21.15.98:0/419627721 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcae0154f70 0x7fcae0157900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:06.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.975+0000 7fcadffff700 1 -- 172.21.15.98:0/419627721 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcae00b3980 con 0x7fcac8003a60 2023-10-06T05:08:06.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.975+0000 7fcadf7fe700 1 --2- 172.21.15.98:0/419627721 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcae0154f70 0x7fcae0157900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:08:06.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.975+0000 7fcadffff700 1 --2- 172.21.15.98:0/419627721 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcac8003a60 0x7fcae00c8280 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fcad001ef70 tx=0x7fcad001a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:06.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.975+0000 7fcac7fff700 1 -- 172.21.15.98:0/419627721 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcad0023070 con 0x7fcac8003a60 2023-10-06T05:08:06.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.975+0000 7fcac7fff700 1 -- 172.21.15.98:0/419627721 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcad000e9e0 con 0x7fcac8003a60 2023-10-06T05:08:06.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.975+0000 7fcae64c5700 1 -- 172.21.15.98:0/419627721 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcae00b3c60 con 0x7fcac8003a60 2023-10-06T05:08:06.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.975+0000 7fcac7fff700 1 -- 172.21.15.98:0/419627721 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcad001e030 con 0x7fcac8003a60 2023-10-06T05:08:06.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.975+0000 7fcae64c5700 1 -- 172.21.15.98:0/419627721 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcae0157e40 con 0x7fcac8003a60 2023-10-06T05:08:06.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.976+0000 7fcac7fff700 1 -- 172.21.15.98:0/419627721 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fcad001e1b0 con 0x7fcac8003a60 2023-10-06T05:08:06.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.977+0000 7fcae64c5700 1 -- 172.21.15.98:0/419627721 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcae00b42b0 con 0x7fcac8003a60 2023-10-06T05:08:06.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.977+0000 7fcac7fff700 1 --2- 172.21.15.98:0/419627721 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fcacc066f30 0x7fcacc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:06.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.977+0000 7fcac7fff700 1 -- 172.21.15.98:0/419627721 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6378+0+0 (secure 0 0 0) 0x7fcad0085fc0 con 0x7fcac8003a60 2023-10-06T05:08:06.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.977+0000 7fcadf7fe700 1 --2- 172.21.15.98:0/419627721 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fcacc066f30 0x7fcacc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:06.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.977+0000 7fcadf7fe700 1 --2- 172.21.15.98:0/419627721 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fcacc066f30 0x7fcacc0693e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fcae00cb3c0 tx=0x7fcad4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:06.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:06.981+0000 7fcac7fff700 1 -- 172.21.15.98:0/419627721 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcad00b75a0 con 0x7fcac8003a60 2023-10-06T05:08:07.169 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.167+0000 7fcae64c5700 1 -- 172.21.15.98:0/419627721 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fcae0064a90 con 0x7fcac8003a60 2023-10-06T05:08:07.169 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.167+0000 7fcac7fff700 1 -- 172.21.15.98:0/419627721 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v43) v1 ==== 104+0+11 (secure 0 0 0) 0x7fcad0052f90 con 0x7fcac8003a60 2023-10-06T05:08:07.170 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2023-10-06T05:08:07.171 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.169+0000 7fcae64c5700 1 -- 172.21.15.98:0/419627721 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fcacc066f30 msgr2=0x7fcacc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:07.172 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.169+0000 7fcae64c5700 1 --2- 172.21.15.98:0/419627721 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fcacc066f30 0x7fcacc0693e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fcae00cb3c0 tx=0x7fcad4008040 comp rx=0 tx=0).stop 2023-10-06T05:08:07.172 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.169+0000 7fcae64c5700 1 -- 172.21.15.98:0/419627721 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcac8003a60 msgr2=0x7fcae00c8280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:07.172 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.169+0000 7fcae64c5700 1 --2- 172.21.15.98:0/419627721 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcac8003a60 0x7fcae00c8280 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fcad001ef70 tx=0x7fcad001a000 comp rx=0 tx=0).stop 2023-10-06T05:08:07.172 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.169+0000 7fcae64c5700 1 -- 172.21.15.98:0/419627721 shutdown_connections 2023-10-06T05:08:07.172 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.169+0000 7fcae64c5700 1 --2- 172.21.15.98:0/419627721 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcae016d490 0x7fcae0151df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:07.172 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.169+0000 7fcae64c5700 1 --2- 172.21.15.98:0/419627721 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fcacc066f30 0x7fcacc0693e0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:07.172 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.169+0000 7fcae64c5700 1 --2- 172.21.15.98:0/419627721 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcac8003a60 0x7fcae00c8280 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:07.172 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.169+0000 7fcae64c5700 1 --2- 172.21.15.98:0/419627721 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcae0154f70 0x7fcae0157900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:07.172 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.169+0000 7fcae64c5700 1 -- 172.21.15.98:0/419627721 >> 172.21.15.98:0/419627721 conn(0x7fcae0089e50 msgr2=0x7fcae00c64f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:07.172 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.169+0000 7fcae64c5700 1 -- 172.21.15.98:0/419627721 shutdown_connections 2023-10-06T05:08:07.172 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.169+0000 7fcae64c5700 1 -- 172.21.15.98:0/419627721 wait complete. 2023-10-06T05:08:07.183 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-06T05:08:07.300 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.298+0000 7f6ba4cbb700 1 Processor -- start 2023-10-06T05:08:07.300 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.298+0000 7f6ba4cbb700 1 -- start start 2023-10-06T05:08:07.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.299+0000 7f6ba4cbb700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f6ba00cd3c0 0x7f6ba00ceaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:07.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.299+0000 7f6ba4cbb700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6ba00b29f0 0x7f6ba00cefe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:07.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.299+0000 7f6ba4cbb700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ba00cc970 con 0x7f6ba00b2620 2023-10-06T05:08:07.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.299+0000 7f6ba4cbb700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ba00cf520 con 0x7f6ba00b29f0 2023-10-06T05:08:07.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.299+0000 7f6ba4cbb700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ba00cbd60 con 0x7f6ba00cd3c0 2023-10-06T05:08:07.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.299+0000 7f6b9ffff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6ba00b29f0 0x7f6ba00cefe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:07.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.299+0000 7f6b9ffff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6ba00b29f0 0x7f6ba00cefe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55024/0 (socket says 172.21.15.98:55024) 2023-10-06T05:08:07.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.299+0000 7f6b9ffff700 1 -- 172.21.15.98:0/3079748086 learned_addr learned my addr 172.21.15.98:0/3079748086 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:08:07.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.299+0000 7f6b96ffd700 1 --2- 172.21.15.98:0/3079748086 >> v2:172.21.15.174:3300/0 conn(0x7f6ba00cd3c0 0x7f6ba00ceaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:07.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.299+0000 7f6b9ffff700 1 -- 172.21.15.98:0/3079748086 >> v2:172.21.15.174:3300/0 conn(0x7f6ba00cd3c0 msgr2=0x7f6ba00ceaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:07.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.299+0000 7f6b9ffff700 1 --2- 172.21.15.98:0/3079748086 >> v2:172.21.15.174:3300/0 conn(0x7f6ba00cd3c0 0x7f6ba00ceaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:07.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.299+0000 7f6b9ffff700 1 -- 172.21.15.98:0/3079748086 >> v1:172.21.15.174:6789/0 conn(0x7f6ba00b2620 legacy=0x7f6ba00cccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:07.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.299+0000 7f6b9ffff700 1 -- 172.21.15.98:0/3079748086 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ba00cc040 con 0x7f6ba00b29f0 2023-10-06T05:08:07.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.299+0000 7f6b96ffd700 1 --2- 172.21.15.98:0/3079748086 >> v2:172.21.15.174:3300/0 conn(0x7f6ba00cd3c0 0x7f6ba00ceaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:08:07.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.299+0000 7f6b9ffff700 1 --2- 172.21.15.98:0/3079748086 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ba00b29f0 0x7f6ba00cefe0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f6ba00b62d0 tx=0x7f6b98009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10d096fd998fb0ad server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:07.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.299+0000 7f6b9effd700 1 -- 172.21.15.98:0/3079748086 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b9800e070 con 0x7f6ba00b29f0 2023-10-06T05:08:07.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.299+0000 7f6b9effd700 1 -- 172.21.15.98:0/3079748086 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b9800ba10 con 0x7f6ba00b29f0 2023-10-06T05:08:07.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.300+0000 7f6b9effd700 1 -- 172.21.15.98:0/3079748086 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b980092e0 con 0x7f6ba00b29f0 2023-10-06T05:08:07.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.300+0000 7f6ba4cbb700 1 -- 172.21.15.98:0/3079748086 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ba00b29f0 msgr2=0x7f6ba00cefe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:07.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.300+0000 7f6ba4cbb700 1 --2- 172.21.15.98:0/3079748086 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ba00b29f0 0x7f6ba00cefe0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f6ba00b62d0 tx=0x7f6b98009840 comp rx=0 tx=0).stop 2023-10-06T05:08:07.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.300+0000 7f6ba4cbb700 1 -- 172.21.15.98:0/3079748086 shutdown_connections 2023-10-06T05:08:07.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.300+0000 7f6ba4cbb700 1 --2- 172.21.15.98:0/3079748086 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ba00b29f0 0x7f6ba00cefe0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:07.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.300+0000 7f6ba4cbb700 1 --2- 172.21.15.98:0/3079748086 >> v2:172.21.15.174:3300/0 conn(0x7f6ba00cd3c0 0x7f6ba00ceaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:07.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.300+0000 7f6ba4cbb700 1 -- 172.21.15.98:0/3079748086 >> 172.21.15.98:0/3079748086 conn(0x7f6ba0099520 msgr2=0x7f6ba006f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:07.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.300+0000 7f6ba4cbb700 1 -- 172.21.15.98:0/3079748086 shutdown_connections 2023-10-06T05:08:07.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.300+0000 7f6ba4cbb700 1 -- 172.21.15.98:0/3079748086 wait complete. 2023-10-06T05:08:07.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.301+0000 7f6ba4cbb700 1 Processor -- start 2023-10-06T05:08:07.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.301+0000 7f6ba4cbb700 1 -- start start 2023-10-06T05:08:07.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.302+0000 7f6ba4cbb700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ba00b2620 0x7f6ba0153270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:07.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.302+0000 7f6ba4cbb700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6ba00b29f0 0x7f6ba0154a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:07.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.302+0000 7f6ba4cbb700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6ba00cd3c0 0x7f6ba0068c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:07.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.302+0000 7f6ba4cbb700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ba00b40a0 con 0x7f6ba00b2620 2023-10-06T05:08:07.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.302+0000 7f6ba4cbb700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ba01554e0 con 0x7f6ba00cd3c0 2023-10-06T05:08:07.308 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.302+0000 7f6ba4cbb700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ba01557c0 con 0x7f6ba00b29f0 2023-10-06T05:08:07.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.302+0000 7f6b9f7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ba00b2620 0x7f6ba0153270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:07.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.302+0000 7f6b9f7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ba00b2620 0x7f6ba0153270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55028/0 (socket says 172.21.15.98:55028) 2023-10-06T05:08:07.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.302+0000 7f6b9f7fe700 1 -- 172.21.15.98:0/3199553834 learned_addr learned my addr 172.21.15.98:0/3199553834 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:08:07.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.302+0000 7f6b9f7fe700 1 -- 172.21.15.98:0/3199553834 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6ba00b29f0 msgr2=0x7f6ba0154a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:07.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.302+0000 7f6b9f7fe700 1 --2- 172.21.15.98:0/3199553834 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6ba00b29f0 0x7f6ba0154a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:07.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.302+0000 7f6b9ffff700 1 --2- 172.21.15.98:0/3199553834 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6ba00cd3c0 0x7f6ba0068c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:07.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.302+0000 7f6b9f7fe700 1 -- 172.21.15.98:0/3199553834 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6ba00cd3c0 msgr2=0x7f6ba0068c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:07.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.302+0000 7f6b9f7fe700 1 --2- 172.21.15.98:0/3199553834 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6ba00cd3c0 0x7f6ba0068c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:07.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.302+0000 7f6b9f7fe700 1 -- 172.21.15.98:0/3199553834 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b98009ae0 con 0x7f6ba00b2620 2023-10-06T05:08:07.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.303+0000 7f6b9f7fe700 1 --2- 172.21.15.98:0/3199553834 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ba00b2620 0x7f6ba0153270 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f6b9000b5e0 tx=0x7f6b900090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:07.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.303+0000 7f6b97fff700 1 -- 172.21.15.98:0/3199553834 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b9000f510 con 0x7f6ba00b2620 2023-10-06T05:08:07.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.303+0000 7f6b97fff700 1 -- 172.21.15.98:0/3199553834 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b9000f670 con 0x7f6ba00b2620 2023-10-06T05:08:07.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.303+0000 7f6ba4cbb700 1 -- 172.21.15.98:0/3199553834 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ba00b3d30 con 0x7f6ba00b2620 2023-10-06T05:08:07.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.303+0000 7f6b97fff700 1 -- 172.21.15.98:0/3199553834 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b9000fa30 con 0x7f6ba00b2620 2023-10-06T05:08:07.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.303+0000 7f6ba4cbb700 1 -- 172.21.15.98:0/3199553834 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ba00b39e0 con 0x7f6ba00b2620 2023-10-06T05:08:07.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.304+0000 7f6ba4cbb700 1 -- 172.21.15.98:0/3199553834 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ba005f230 con 0x7f6ba00b2620 2023-10-06T05:08:07.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.304+0000 7f6b97fff700 1 -- 172.21.15.98:0/3199553834 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f6b9001a400 con 0x7f6ba00b2620 2023-10-06T05:08:07.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.304+0000 7f6b97fff700 1 --2- 172.21.15.98:0/3199553834 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f6b80066f30 0x7f6b800693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:07.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.304+0000 7f6b97fff700 1 -- 172.21.15.98:0/3199553834 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6378+0+0 (secure 0 0 0) 0x7f6b900842a0 con 0x7f6ba00b2620 2023-10-06T05:08:07.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.304+0000 7f6b96ffd700 1 --2- 172.21.15.98:0/3199553834 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f6b80066f30 0x7f6b800693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:07.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.305+0000 7f6b96ffd700 1 --2- 172.21.15.98:0/3199553834 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f6b80066f30 0x7f6b800693e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f6ba01537b0 tx=0x7f6b88008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:07.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.309+0000 7f6b97fff700 1 -- 172.21.15.98:0/3199553834 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6b90054d80 con 0x7f6ba00b2620 2023-10-06T05:08:07.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.489+0000 7f6ba4cbb700 1 -- 172.21.15.98:0/3199553834 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6ba00cc320 con 0x7f6ba00b2620 2023-10-06T05:08:07.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.490+0000 7f6b97fff700 1 -- 172.21.15.98:0/3199553834 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v43) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6b90054910 con 0x7f6ba00b2620 2023-10-06T05:08:07.492 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2023-10-06T05:08:07.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.492+0000 7f6ba4cbb700 1 -- 172.21.15.98:0/3199553834 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f6b80066f30 msgr2=0x7f6b800693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:07.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.492+0000 7f6ba4cbb700 1 --2- 172.21.15.98:0/3199553834 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f6b80066f30 0x7f6b800693e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f6ba01537b0 tx=0x7f6b88008040 comp rx=0 tx=0).stop 2023-10-06T05:08:07.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.492+0000 7f6ba4cbb700 1 -- 172.21.15.98:0/3199553834 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ba00b2620 msgr2=0x7f6ba0153270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:07.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.492+0000 7f6ba4cbb700 1 --2- 172.21.15.98:0/3199553834 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ba00b2620 0x7f6ba0153270 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f6b9000b5e0 tx=0x7f6b900090f0 comp rx=0 tx=0).stop 2023-10-06T05:08:07.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.492+0000 7f6ba4cbb700 1 -- 172.21.15.98:0/3199553834 shutdown_connections 2023-10-06T05:08:07.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.492+0000 7f6ba4cbb700 1 --2- 172.21.15.98:0/3199553834 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6ba00cd3c0 0x7f6ba0068c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:07.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.492+0000 7f6ba4cbb700 1 --2- 172.21.15.98:0/3199553834 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f6b80066f30 0x7f6b800693e0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:07.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.492+0000 7f6ba4cbb700 1 --2- 172.21.15.98:0/3199553834 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ba00b2620 0x7f6ba0153270 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:07.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.492+0000 7f6ba4cbb700 1 --2- 172.21.15.98:0/3199553834 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6ba00b29f0 0x7f6ba0154a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:07.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.492+0000 7f6ba4cbb700 1 -- 172.21.15.98:0/3199553834 >> 172.21.15.98:0/3199553834 conn(0x7f6ba0099520 msgr2=0x7f6ba0157880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:07.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.492+0000 7f6ba4cbb700 1 -- 172.21.15.98:0/3199553834 shutdown_connections 2023-10-06T05:08:07.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.492+0000 7f6ba4cbb700 1 -- 172.21.15.98:0/3199553834 wait complete. 2023-10-06T05:08:07.506 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-06T05:08:07.620 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.618+0000 7fde9e269700 1 Processor -- start 2023-10-06T05:08:07.620 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.618+0000 7fde9e269700 1 -- start start 2023-10-06T05:08:07.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.619+0000 7fde9e269700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fde98069690 0x7fde98069a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:07.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.619+0000 7fde9e269700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fde980b11c0 0x7fde980b15d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:07.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.619+0000 7fde9e269700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde98159f60 con 0x7fde9815a3f0 2023-10-06T05:08:07.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.619+0000 7fde9e269700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde98159980 con 0x7fde980b11c0 2023-10-06T05:08:07.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.619+0000 7fde9e269700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde980b1b10 con 0x7fde98069690 2023-10-06T05:08:07.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.619+0000 7fde9d267700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fde98069690 0x7fde98069a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:07.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.619+0000 7fde9ca66700 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fde9815a3f0 0x7fde9806cbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.98:42076/0 (socket says 172.21.15.98:42076) 2023-10-06T05:08:07.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.619+0000 7fde9da68700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fde980b11c0 0x7fde980b15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:07.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.619+0000 7fde9ca66700 1 -- 172.21.15.98:0/1223980483 learned_addr learned my addr 172.21.15.98:0/1223980483 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:08:07.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.620+0000 7fde8ffff700 1 -- 172.21.15.98:0/1223980483 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4090426779 0 0) 0x7fde98159f60 con 0x7fde9815a3f0 2023-10-06T05:08:07.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.620+0000 7fde8ffff700 1 -- 172.21.15.98:0/1223980483 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde84003570 con 0x7fde9815a3f0 2023-10-06T05:08:07.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.620+0000 7fde9d267700 1 -- 172.21.15.98:0/1223980483 >> v2:172.21.15.174:3300/0 conn(0x7fde980b11c0 msgr2=0x7fde980b15d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:07.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.620+0000 7fde9d267700 1 --2- 172.21.15.98:0/1223980483 >> v2:172.21.15.174:3300/0 conn(0x7fde980b11c0 0x7fde980b15d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:07.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.620+0000 7fde9d267700 1 -- 172.21.15.98:0/1223980483 >> v1:172.21.15.174:6789/0 conn(0x7fde9815a3f0 legacy=0x7fde9806cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:07.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.620+0000 7fde9d267700 1 -- 172.21.15.98:0/1223980483 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde9806d610 con 0x7fde98069690 2023-10-06T05:08:07.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.620+0000 7fde9da68700 1 --2- 172.21.15.98:0/1223980483 >> v2:172.21.15.174:3300/0 conn(0x7fde980b11c0 0x7fde980b15d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T05:08:07.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.620+0000 7fde9d267700 1 --2- 172.21.15.98:0/1223980483 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fde98069690 0x7fde98069a60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fde94009e60 tx=0x7fde94009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=898fc324546c6ec server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:07.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.620+0000 7fde8ffff700 1 -- 172.21.15.98:0/1223980483 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde9400e070 con 0x7fde98069690 2023-10-06T05:08:07.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.620+0000 7fde8ffff700 1 -- 172.21.15.98:0/1223980483 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde9400bab0 con 0x7fde98069690 2023-10-06T05:08:07.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.620+0000 7fde8ffff700 1 -- 172.21.15.98:0/1223980483 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde9400be30 con 0x7fde98069690 2023-10-06T05:08:07.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.621+0000 7fde9e269700 1 -- 172.21.15.98:0/1223980483 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fde98069690 msgr2=0x7fde98069a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:07.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.621+0000 7fde9e269700 1 --2- 172.21.15.98:0/1223980483 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fde98069690 0x7fde98069a60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fde94009e60 tx=0x7fde94009780 comp rx=0 tx=0).stop 2023-10-06T05:08:07.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.621+0000 7fde9e269700 1 -- 172.21.15.98:0/1223980483 shutdown_connections 2023-10-06T05:08:07.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.621+0000 7fde9e269700 1 --2- 172.21.15.98:0/1223980483 >> v2:172.21.15.174:3300/0 conn(0x7fde980b11c0 0x7fde980b15d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:07.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.621+0000 7fde9e269700 1 --2- 172.21.15.98:0/1223980483 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fde98069690 0x7fde98069a60 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:07.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.621+0000 7fde9e269700 1 -- 172.21.15.98:0/1223980483 >> 172.21.15.98:0/1223980483 conn(0x7fde9806b500 msgr2=0x7fde9806b900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:07.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.621+0000 7fde9e269700 1 -- 172.21.15.98:0/1223980483 shutdown_connections 2023-10-06T05:08:07.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.621+0000 7fde9e269700 1 -- 172.21.15.98:0/1223980483 wait complete. 2023-10-06T05:08:07.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.622+0000 7fde9e269700 1 Processor -- start 2023-10-06T05:08:07.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.622+0000 7fde9e269700 1 -- start start 2023-10-06T05:08:07.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.623+0000 7fde9e269700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fde98069690 0x7fde98147bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:07.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.623+0000 7fde9e269700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fde980b11c0 0x7fde980af3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:07.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.623+0000 7fde9e269700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde9815a3f0 0x7fde980c9e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:07.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.623+0000 7fde9e269700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde980aef10 con 0x7fde9815a3f0 2023-10-06T05:08:07.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.623+0000 7fde9e269700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde981480f0 con 0x7fde98069690 2023-10-06T05:08:07.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.623+0000 7fde9e269700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde980ae300 con 0x7fde980b11c0 2023-10-06T05:08:07.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.623+0000 7fde9da68700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde9815a3f0 0x7fde980c9e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:07.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.623+0000 7fde9da68700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde9815a3f0 0x7fde980c9e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55032/0 (socket says 172.21.15.98:55032) 2023-10-06T05:08:07.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.623+0000 7fde9da68700 1 -- 172.21.15.98:0/2677160882 learned_addr learned my addr 172.21.15.98:0/2677160882 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:08:07.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.623+0000 7fde9d267700 1 --2- 172.21.15.98:0/2677160882 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fde98069690 0x7fde98147bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:07.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.623+0000 7fde9ca66700 1 --2- 172.21.15.98:0/2677160882 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fde980b11c0 0x7fde980af3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:07.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.623+0000 7fde9da68700 1 -- 172.21.15.98:0/2677160882 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fde980b11c0 msgr2=0x7fde980af3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:07.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.623+0000 7fde9da68700 1 --2- 172.21.15.98:0/2677160882 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fde980b11c0 0x7fde980af3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:07.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.623+0000 7fde9da68700 1 -- 172.21.15.98:0/2677160882 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fde98069690 msgr2=0x7fde98147bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:07.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.623+0000 7fde9da68700 1 --2- 172.21.15.98:0/2677160882 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fde98069690 0x7fde98147bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:07.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.623+0000 7fde9da68700 1 -- 172.21.15.98:0/2677160882 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde90009420 con 0x7fde9815a3f0 2023-10-06T05:08:07.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.623+0000 7fde9d267700 1 --2- 172.21.15.98:0/2677160882 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fde98069690 0x7fde98147bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:08:07.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.623+0000 7fde9da68700 1 --2- 172.21.15.98:0/2677160882 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde9815a3f0 0x7fde980c9e20 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fde9000ec80 tx=0x7fde900091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:07.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.623+0000 7fde8d7fa700 1 -- 172.21.15.98:0/2677160882 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde9000c9d0 con 0x7fde9815a3f0 2023-10-06T05:08:07.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.623+0000 7fde8d7fa700 1 -- 172.21.15.98:0/2677160882 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde9000cb30 con 0x7fde9815a3f0 2023-10-06T05:08:07.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.623+0000 7fde9e269700 1 -- 172.21.15.98:0/2677160882 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde94009a20 con 0x7fde9815a3f0 2023-10-06T05:08:07.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.623+0000 7fde8d7fa700 1 -- 172.21.15.98:0/2677160882 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde90013470 con 0x7fde9815a3f0 2023-10-06T05:08:07.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.623+0000 7fde9e269700 1 -- 172.21.15.98:0/2677160882 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde980ae8c0 con 0x7fde9815a3f0 2023-10-06T05:08:07.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.624+0000 7fde9e269700 1 -- 172.21.15.98:0/2677160882 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde9805f230 con 0x7fde9815a3f0 2023-10-06T05:08:07.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.625+0000 7fde8d7fa700 1 -- 172.21.15.98:0/2677160882 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fde90013860 con 0x7fde9815a3f0 2023-10-06T05:08:07.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.625+0000 7fde8d7fa700 1 --2- 172.21.15.98:0/2677160882 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fde7c066f30 0x7fde7c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:07.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.625+0000 7fde8d7fa700 1 -- 172.21.15.98:0/2677160882 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6378+0+0 (secure 0 0 0) 0x7fde900849a0 con 0x7fde9815a3f0 2023-10-06T05:08:07.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.625+0000 7fde9d267700 1 --2- 172.21.15.98:0/2677160882 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fde7c066f30 0x7fde7c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:07.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.625+0000 7fde9d267700 1 --2- 172.21.15.98:0/2677160882 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fde7c066f30 0x7fde7c0693e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fde940097c0 tx=0x7fde94019000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:07.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.629+0000 7fde8d7fa700 1 -- 172.21.15.98:0/2677160882 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fde90051970 con 0x7fde9815a3f0 2023-10-06T05:08:07.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.855+0000 7fde9e269700 1 -- 172.21.15.98:0/2677160882 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fde980ceeb0 con 0x7fde9815a3f0 2023-10-06T05:08:07.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.855+0000 7fde8d7fa700 1 -- 172.21.15.98:0/2677160882 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4603 (secure 0 0 0) 0x7fde90017090 con 0x7fde9815a3f0 2023-10-06T05:08:07.857 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:08:07.857 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4966,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6834/2320896583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2320896583},{"type":"v1","addr":"172.21.15.174:6834","nonce":2320896583}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":4969,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/1423644921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6833","nonce":1423644921},{"type":"v1","addr":"172.21.15.174:6835","nonce":1423644921}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5100,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/4291295874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":4291295874},{"type":"v1","addr":"172.21.15.98:6836","nonce":4291295874}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-06T05:07:55.873515+0000","modified":"2023-10-06T05:07:59.511013+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5103},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5103":{"gid":5103,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.98:6837/1617154626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1617154626},{"type":"v1","addr":"172.21.15.98:6837","nonce":1617154626}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2023-10-06T05:08:07.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.857+0000 7fde9e269700 1 -- 172.21.15.98:0/2677160882 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fde7c066f30 msgr2=0x7fde7c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:07.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.857+0000 7fde9e269700 1 --2- 172.21.15.98:0/2677160882 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fde7c066f30 0x7fde7c0693e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fde940097c0 tx=0x7fde94019000 comp rx=0 tx=0).stop 2023-10-06T05:08:07.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.857+0000 7fde9e269700 1 -- 172.21.15.98:0/2677160882 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde9815a3f0 msgr2=0x7fde980c9e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:07.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.857+0000 7fde9e269700 1 --2- 172.21.15.98:0/2677160882 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde9815a3f0 0x7fde980c9e20 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fde9000ec80 tx=0x7fde900091a0 comp rx=0 tx=0).stop 2023-10-06T05:08:07.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.857+0000 7fde9e269700 1 -- 172.21.15.98:0/2677160882 shutdown_connections 2023-10-06T05:08:07.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.857+0000 7fde9e269700 1 --2- 172.21.15.98:0/2677160882 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fde7c066f30 0x7fde7c0693e0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:07.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.857+0000 7fde9e269700 1 --2- 172.21.15.98:0/2677160882 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde9815a3f0 0x7fde980c9e20 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:07.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.857+0000 7fde9e269700 1 --2- 172.21.15.98:0/2677160882 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fde98069690 0x7fde98147bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:07.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.857+0000 7fde9e269700 1 --2- 172.21.15.98:0/2677160882 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fde980b11c0 0x7fde980af3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:07.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.857+0000 7fde9e269700 1 -- 172.21.15.98:0/2677160882 >> 172.21.15.98:0/2677160882 conn(0x7fde9806b500 msgr2=0x7fde98156df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:07.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.857+0000 7fde9e269700 1 -- 172.21.15.98:0/2677160882 shutdown_connections 2023-10-06T05:08:07.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.857+0000 7fde9e269700 1 -- 172.21.15.98:0/2677160882 wait complete. 2023-10-06T05:08:07.861 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 17 2023-10-06T05:08:07.871 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T05:08:07.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.984+0000 7f292221e700 1 Processor -- start 2023-10-06T05:08:07.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.984+0000 7f292221e700 1 -- start start 2023-10-06T05:08:07.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.985+0000 7f292221e700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f291c0b49a0 0x7f291c0aac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:07.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.985+0000 7f292221e700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f291c0b0760 0x7f291c0b1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:07.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.985+0000 7f292221e700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f291c0b0c60 con 0x7f291c159560 2023-10-06T05:08:07.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.985+0000 7f292221e700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f291c0b0f40 con 0x7f291c0b49a0 2023-10-06T05:08:07.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.985+0000 7f292221e700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f291c0b1220 con 0x7f291c0b0760 2023-10-06T05:08:07.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.985+0000 7f291b7fe700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f291c159560 0x7f291c0aa530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:38404/0 (socket says 172.21.15.98:38404) 2023-10-06T05:08:07.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.985+0000 7f291b7fe700 1 -- 172.21.15.98:0/197481200 learned_addr learned my addr 172.21.15.98:0/197481200 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:08:07.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.986+0000 7f291a7fc700 1 -- 172.21.15.98:0/197481200 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2291166716 0 0) 0x7f291c0b0c60 con 0x7f291c159560 2023-10-06T05:08:07.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.986+0000 7f291a7fc700 1 -- 172.21.15.98:0/197481200 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2904003570 con 0x7f291c159560 2023-10-06T05:08:07.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.986+0000 7f291affd700 1 --2- 172.21.15.98:0/197481200 >> v2:172.21.15.174:3300/0 conn(0x7f291c0b49a0 0x7f291c0aac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:07.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.986+0000 7f291bfff700 1 --2- 172.21.15.98:0/197481200 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f291c0b0760 0x7f291c0b1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:07.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.986+0000 7f291a7fc700 1 -- 172.21.15.98:0/197481200 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 561578031 0 0) 0x7f2904003570 con 0x7f291c159560 2023-10-06T05:08:07.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.986+0000 7f291a7fc700 1 -- 172.21.15.98:0/197481200 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f291c0b0760 msgr2=0x7f291c0b1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:07.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.986+0000 7f291a7fc700 1 --2- 172.21.15.98:0/197481200 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f291c0b0760 0x7f291c0b1bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:07.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.986+0000 7f291a7fc700 1 -- 172.21.15.98:0/197481200 >> v2:172.21.15.174:3300/0 conn(0x7f291c0b49a0 msgr2=0x7f291c0aac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:07.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.986+0000 7f291a7fc700 1 --2- 172.21.15.98:0/197481200 >> v2:172.21.15.174:3300/0 conn(0x7f291c0b49a0 0x7f291c0aac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:07.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.986+0000 7f291a7fc700 1 -- 172.21.15.98:0/197481200 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f291c0b1500 con 0x7f291c159560 2023-10-06T05:08:07.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.986+0000 7f291affd700 1 --2- 172.21.15.98:0/197481200 >> v2:172.21.15.174:3300/0 conn(0x7f291c0b49a0 0x7f291c0aac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:08:07.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.986+0000 7f291a7fc700 1 -- 172.21.15.98:0/197481200 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1543991876 0 0) 0x7f290c001970 con 0x7f291c159560 2023-10-06T05:08:07.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.986+0000 7f291a7fc700 1 -- 172.21.15.98:0/197481200 >> v1:172.21.15.98:6789/0 conn(0x7f291c159560 legacy=0x7f291c0aa530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:07.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.986+0000 7f291a7fc700 1 --2- 172.21.15.98:0/197481200 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f291c0b0760 0x7f2904003a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:07.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.986+0000 7f291a7fc700 1 --2- 172.21.15.98:0/197481200 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2904004090 0x7f2904004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:07.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.986+0000 7f291a7fc700 1 --2- 172.21.15.98:0/197481200 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2904004d50 0x7f29040051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:07.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.986+0000 7f291a7fc700 1 -- 172.21.15.98:0/197481200 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f291c0b0f40 con 0x7f2904004090 2023-10-06T05:08:07.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.986+0000 7f291a7fc700 1 -- 172.21.15.98:0/197481200 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f291c0b1220 con 0x7f2904004d50 2023-10-06T05:08:07.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.986+0000 7f291a7fc700 1 -- 172.21.15.98:0/197481200 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f290400bbf0 con 0x7f291c0b0760 2023-10-06T05:08:07.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.986+0000 7f291affd700 1 --2- 172.21.15.98:0/197481200 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2904004090 0x7f2904004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:07.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.986+0000 7f291b7fe700 1 --2- 172.21.15.98:0/197481200 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f291c0b0760 0x7f2904003a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:07.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.986+0000 7f291bfff700 1 --2- 172.21.15.98:0/197481200 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2904004d50 0x7f29040051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:07.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.986+0000 7f291affd700 1 -- 172.21.15.98:0/197481200 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f291c0b0760 msgr2=0x7f2904003a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:07.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.986+0000 7f291affd700 1 --2- 172.21.15.98:0/197481200 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f291c0b0760 0x7f2904003a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:07.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.986+0000 7f291affd700 1 -- 172.21.15.98:0/197481200 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2904004d50 msgr2=0x7f29040051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:07.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.986+0000 7f291affd700 1 --2- 172.21.15.98:0/197481200 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2904004d50 0x7f29040051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:07.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.986+0000 7f291affd700 1 -- 172.21.15.98:0/197481200 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2904003570 con 0x7f2904004090 2023-10-06T05:08:07.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.986+0000 7f291affd700 1 --2- 172.21.15.98:0/197481200 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2904004090 0x7f2904004810 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f291c099f20 tx=0x7f2910008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de3a1b2ff8b8e050 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:07.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.987+0000 7f291a7fc700 1 -- 172.21.15.98:0/197481200 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f291000c070 con 0x7f2904004090 2023-10-06T05:08:07.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.987+0000 7f291a7fc700 1 -- 172.21.15.98:0/197481200 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f291000e5b0 con 0x7f2904004090 2023-10-06T05:08:07.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.987+0000 7f291a7fc700 1 -- 172.21.15.98:0/197481200 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f291000e740 con 0x7f2904004090 2023-10-06T05:08:07.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.987+0000 7f292221e700 1 -- 172.21.15.98:0/197481200 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2904004090 msgr2=0x7f2904004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:07.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.987+0000 7f292221e700 1 --2- 172.21.15.98:0/197481200 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2904004090 0x7f2904004810 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f291c099f20 tx=0x7f2910008040 comp rx=0 tx=0).stop 2023-10-06T05:08:07.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.987+0000 7f291bfff700 1 -- 172.21.15.98:0/197481200 reap_dead start 2023-10-06T05:08:07.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.987+0000 7f292221e700 1 -- 172.21.15.98:0/197481200 shutdown_connections 2023-10-06T05:08:07.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.987+0000 7f292221e700 1 -- 172.21.15.98:0/197481200 >> 172.21.15.98:0/197481200 conn(0x7f291c0997f0 msgr2=0x7f291c06f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:07.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.987+0000 7f292221e700 1 -- 172.21.15.98:0/197481200 shutdown_connections 2023-10-06T05:08:07.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.987+0000 7f292221e700 1 -- 172.21.15.98:0/197481200 wait complete. 2023-10-06T05:08:07.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.988+0000 7f292221e700 1 Processor -- start 2023-10-06T05:08:07.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.988+0000 7f292221e700 1 -- start start 2023-10-06T05:08:07.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.989+0000 7f292221e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f291c156dc0 0x7f291c09a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:07.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.989+0000 7f292221e700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f291c09a590 0x7f291c09be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:07.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.989+0000 7f292221e700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f291c09c3d0 0x7f291c159c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:07.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.989+0000 7f292221e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f291c09c870 con 0x7f291c156dc0 2023-10-06T05:08:07.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.989+0000 7f292221e700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f291c09b690 con 0x7f291c09a590 2023-10-06T05:08:07.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.989+0000 7f292221e700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f291c09aa80 con 0x7f291c09c3d0 2023-10-06T05:08:07.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.989+0000 7f291b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f291c156dc0 0x7f291c09a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:07.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.989+0000 7f291b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f291c156dc0 0x7f291c09a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55062/0 (socket says 172.21.15.98:55062) 2023-10-06T05:08:07.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.989+0000 7f291b7fe700 1 -- 172.21.15.98:0/4290364109 learned_addr learned my addr 172.21.15.98:0/4290364109 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:08:07.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.989+0000 7f291bfff700 1 --2- 172.21.15.98:0/4290364109 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f291c09c3d0 0x7f291c159c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:07.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.990+0000 7f291affd700 1 --2- 172.21.15.98:0/4290364109 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f291c09a590 0x7f291c09be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:07.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.990+0000 7f291b7fe700 1 -- 172.21.15.98:0/4290364109 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f291c09c3d0 msgr2=0x7f291c159c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:07.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.990+0000 7f291b7fe700 1 --2- 172.21.15.98:0/4290364109 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f291c09c3d0 0x7f291c159c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:07.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.990+0000 7f291b7fe700 1 -- 172.21.15.98:0/4290364109 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f291c09a590 msgr2=0x7f291c09be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:07.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.990+0000 7f291b7fe700 1 --2- 172.21.15.98:0/4290364109 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f291c09a590 0x7f291c09be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:07.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.990+0000 7f291b7fe700 1 -- 172.21.15.98:0/4290364109 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f291c09ad60 con 0x7f291c156dc0 2023-10-06T05:08:07.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.990+0000 7f291b7fe700 1 --2- 172.21.15.98:0/4290364109 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f291c156dc0 0x7f291c09a050 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f290c00ffa0 tx=0x7f290c009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:07.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.990+0000 7f29037fe700 1 -- 172.21.15.98:0/4290364109 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f290c005e50 con 0x7f291c156dc0 2023-10-06T05:08:07.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.990+0000 7f29037fe700 1 -- 172.21.15.98:0/4290364109 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f290c00da70 con 0x7f291c156dc0 2023-10-06T05:08:07.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.990+0000 7f292221e700 1 -- 172.21.15.98:0/4290364109 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f291c09b040 con 0x7f291c156dc0 2023-10-06T05:08:07.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.990+0000 7f29037fe700 1 -- 172.21.15.98:0/4290364109 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f290c00de30 con 0x7f291c156dc0 2023-10-06T05:08:07.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.990+0000 7f292221e700 1 -- 172.21.15.98:0/4290364109 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f291c0cc0d0 con 0x7f291c156dc0 2023-10-06T05:08:07.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.991+0000 7f292221e700 1 -- 172.21.15.98:0/4290364109 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f291c064a90 con 0x7f291c156dc0 2023-10-06T05:08:08.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.991+0000 7f29037fe700 1 -- 172.21.15.98:0/4290364109 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f290c0072e0 con 0x7f291c156dc0 2023-10-06T05:08:08.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.992+0000 7f29037fe700 1 --2- 172.21.15.98:0/4290364109 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f2908066f30 0x7f29080693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:08.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.992+0000 7f29037fe700 1 -- 172.21.15.98:0/4290364109 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6378+0+0 (secure 0 0 0) 0x7f290c084f10 con 0x7f291c156dc0 2023-10-06T05:08:08.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.992+0000 7f291affd700 1 --2- 172.21.15.98:0/4290364109 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f2908066f30 0x7f29080693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:08.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.992+0000 7f291affd700 1 --2- 172.21.15.98:0/4290364109 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f2908066f30 0x7f29080693e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f291c099f20 tx=0x7f291001a000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:08.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:07.996+0000 7f29037fe700 1 -- 172.21.15.98:0/4290364109 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f290c051ee0 con 0x7f291c156dc0 2023-10-06T05:08:08.181 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.179+0000 7f292221e700 1 -- 172.21.15.98:0/4290364109 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f291c0b17e0 con 0x7f291c156dc0 2023-10-06T05:08:08.181 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.179+0000 7f29037fe700 1 -- 172.21.15.98:0/4290364109 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v43) v1 ==== 74+0+17071 (secure 0 0 0) 0x7f290c055500 con 0x7f291c156dc0 2023-10-06T05:08:08.182 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:08:08.182 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":43,"fsid":"f50c5df5-cad5-47a4-9bb1-8c791d4311ec","created":"2023-10-06T05:06:32.971175+0000","modified":"2023-10-06T05:07:57.482671+0000","last_up_change":"2023-10-06T05:06:44.016489+0000","last_in_change":"2023-10-06T05:06:35.544511+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-06T05:06:44.559671+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-06T05:06:51.432161+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-10-06T05:07:54.125452+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-10-06T05:07:54.831073+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d9495d95-da75-4c3b-8927-e9bae6a8b093","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":90627},{"type":"v1","addr":"172.21.15.98:6801","nonce":90627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":90627},{"type":"v1","addr":"172.21.15.98:6803","nonce":90627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":90627},{"type":"v1","addr":"172.21.15.98:6807","nonce":90627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":90627},{"type":"v1","addr":"172.21.15.98:6805","nonce":90627}]},"public_addr":"172.21.15.98:6801/90627","cluster_addr":"172.21.15.98:6803/90627","heartbeat_back_addr":"172.21.15.98:6807/90627","heartbeat_front_addr":"172.21.15.98:6805/90627","state":["exists","up"]},{"osd":1,"uuid":"f25ba5be-d65e-4707-a134-154092deb754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":90628},{"type":"v1","addr":"172.21.15.98:6809","nonce":90628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":90628},{"type":"v1","addr":"172.21.15.98:6811","nonce":90628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":90628},{"type":"v1","addr":"172.21.15.98:6815","nonce":90628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":90628},{"type":"v1","addr":"172.21.15.98:6813","nonce":90628}]},"public_addr":"172.21.15.98:6809/90628","cluster_addr":"172.21.15.98:6811/90628","heartbeat_back_addr":"172.21.15.98:6815/90628","heartbeat_front_addr":"172.21.15.98:6813/90628","state":["exists","up"]},{"osd":2,"uuid":"5ab9e028-945c-43e2-8c95-3b0b4cf38e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":90662},{"type":"v1","addr":"172.21.15.98:6818","nonce":90662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":90662},{"type":"v1","addr":"172.21.15.98:6822","nonce":90662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":90662},{"type":"v1","addr":"172.21.15.98:6830","nonce":90662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":90662},{"type":"v1","addr":"172.21.15.98:6826","nonce":90662}]},"public_addr":"172.21.15.98:6818/90662","cluster_addr":"172.21.15.98:6822/90662","heartbeat_back_addr":"172.21.15.98:6830/90662","heartbeat_front_addr":"172.21.15.98:6826/90662","state":["exists","up"]},{"osd":3,"uuid":"3bb803ff-1738-46bf-be62-2267bf378822","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6817","nonce":90663},{"type":"v1","addr":"172.21.15.98:6819","nonce":90663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6821","nonce":90663},{"type":"v1","addr":"172.21.15.98:6823","nonce":90663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6829","nonce":90663},{"type":"v1","addr":"172.21.15.98:6831","nonce":90663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6825","nonce":90663},{"type":"v1","addr":"172.21.15.98:6827","nonce":90663}]},"public_addr":"172.21.15.98:6819/90663","cluster_addr":"172.21.15.98:6823/90663","heartbeat_back_addr":"172.21.15.98:6831/90663","heartbeat_front_addr":"172.21.15.98:6827/90663","state":["exists","up"]},{"osd":4,"uuid":"7d57a4c9-6462-43f6-b64a-7f9e9b4353d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":90847},{"type":"v1","addr":"172.21.15.174:6801","nonce":90847}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":90847},{"type":"v1","addr":"172.21.15.174:6803","nonce":90847}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":90847},{"type":"v1","addr":"172.21.15.174:6807","nonce":90847}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":90847},{"type":"v1","addr":"172.21.15.174:6805","nonce":90847}]},"public_addr":"172.21.15.174:6801/90847","cluster_addr":"172.21.15.174:6803/90847","heartbeat_back_addr":"172.21.15.174:6807/90847","heartbeat_front_addr":"172.21.15.174:6805/90847","state":["exists","up"]},{"osd":5,"uuid":"83287ea4-e44d-4af9-bc77-36d47c60d4a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":90848},{"type":"v1","addr":"172.21.15.174:6809","nonce":90848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":90848},{"type":"v1","addr":"172.21.15.174:6811","nonce":90848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":90848},{"type":"v1","addr":"172.21.15.174:6815","nonce":90848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":90848},{"type":"v1","addr":"172.21.15.174:6813","nonce":90848}]},"public_addr":"172.21.15.174:6809/90848","cluster_addr":"172.21.15.174:6811/90848","heartbeat_back_addr":"172.21.15.174:6815/90848","heartbeat_front_addr":"172.21.15.174:6813/90848","state":["exists","up"]},{"osd":6,"uuid":"496a6272-892e-43c4-a145-206623c22f99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":90877},{"type":"v1","addr":"172.21.15.174:6817","nonce":90877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":90877},{"type":"v1","addr":"172.21.15.174:6819","nonce":90877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":90877},{"type":"v1","addr":"172.21.15.174:6823","nonce":90877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":90877},{"type":"v1","addr":"172.21.15.174:6821","nonce":90877}]},"public_addr":"172.21.15.174:6817/90877","cluster_addr":"172.21.15.174:6819/90877","heartbeat_back_addr":"172.21.15.174:6823/90877","heartbeat_front_addr":"172.21.15.174:6821/90877","state":["exists","up"]},{"osd":7,"uuid":"49967e34-8b07-4bab-a13e-0c16d6a0f3a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":90878},{"type":"v1","addr":"172.21.15.174:6825","nonce":90878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":90878},{"type":"v1","addr":"172.21.15.174:6827","nonce":90878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":90878},{"type":"v1","addr":"172.21.15.174:6831","nonce":90878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":90878},{"type":"v1","addr":"172.21.15.174:6829","nonce":90878}]},"public_addr":"172.21.15.174:6825/90878","cluster_addr":"172.21.15.174:6827/90878","heartbeat_back_addr":"172.21.15.174:6831/90878","heartbeat_front_addr":"172.21.15.174:6829/90878","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.446479+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.514266+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.794108+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.810894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.731919+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.717756+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.737435+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.508682+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T05:08:08.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.182+0000 7f292221e700 1 -- 172.21.15.98:0/4290364109 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f2908066f30 msgr2=0x7f29080693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:08.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.182+0000 7f292221e700 1 --2- 172.21.15.98:0/4290364109 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f2908066f30 0x7f29080693e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f291c099f20 tx=0x7f291001a000 comp rx=0 tx=0).stop 2023-10-06T05:08:08.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.182+0000 7f292221e700 1 -- 172.21.15.98:0/4290364109 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f291c156dc0 msgr2=0x7f291c09a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:08.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.182+0000 7f292221e700 1 --2- 172.21.15.98:0/4290364109 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f291c156dc0 0x7f291c09a050 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f290c00ffa0 tx=0x7f290c009040 comp rx=0 tx=0).stop 2023-10-06T05:08:08.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.182+0000 7f292221e700 1 -- 172.21.15.98:0/4290364109 shutdown_connections 2023-10-06T05:08:08.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.182+0000 7f292221e700 1 --2- 172.21.15.98:0/4290364109 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f291c09c3d0 0x7f291c159c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:08.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.182+0000 7f292221e700 1 --2- 172.21.15.98:0/4290364109 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f2908066f30 0x7f29080693e0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:08.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.182+0000 7f292221e700 1 --2- 172.21.15.98:0/4290364109 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f291c156dc0 0x7f291c09a050 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:08.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.182+0000 7f292221e700 1 --2- 172.21.15.98:0/4290364109 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f291c09a590 0x7f291c09be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:08.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.182+0000 7f292221e700 1 -- 172.21.15.98:0/4290364109 >> 172.21.15.98:0/4290364109 conn(0x7f291c0997f0 msgr2=0x7f291c0699d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:08.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.182+0000 7f292221e700 1 -- 172.21.15.98:0/4290364109 shutdown_connections 2023-10-06T05:08:08.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.182+0000 7f292221e700 1 -- 172.21.15.98:0/4290364109 wait complete. 2023-10-06T05:08:08.196 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-06T05:08:08.196 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-06T05:08:08.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.308+0000 7f705af50700 1 Processor -- start 2023-10-06T05:08:08.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.308+0000 7f705af50700 1 -- start start 2023-10-06T05:08:08.312 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.309+0000 7f705af50700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f7054159440 0x7f70540b3090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:08.313 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.309+0000 7f705af50700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f70540b35d0 0x7f70540cbc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:08.313 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.309+0000 7f705af50700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70540cb7a0 con 0x7f70540cc170 2023-10-06T05:08:08.313 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.309+0000 7f705af50700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70540b0860 con 0x7f70540b35d0 2023-10-06T05:08:08.313 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.309+0000 7f705af50700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70540cb150 con 0x7f7054159440 2023-10-06T05:08:08.313 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.309+0000 7f705974d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f70540b35d0 0x7f70540cbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:08.313 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.309+0000 7f705a74f700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f70540cc170 0x7f70540b2100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:38420/0 (socket says 172.21.15.98:38420) 2023-10-06T05:08:08.313 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.309+0000 7f705a74f700 1 -- 172.21.15.98:0/3180370993 learned_addr learned my addr 172.21.15.98:0/3180370993 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:08:08.313 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.309+0000 7f7059f4e700 1 --2- 172.21.15.98:0/3180370993 >> v2:172.21.15.174:3300/0 conn(0x7f7054159440 0x7f70540b3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:08.313 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.309+0000 7f705974d700 1 -- 172.21.15.98:0/3180370993 >> v2:172.21.15.174:3300/0 conn(0x7f7054159440 msgr2=0x7f70540b3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:08.313 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.309+0000 7f705974d700 1 --2- 172.21.15.98:0/3180370993 >> v2:172.21.15.174:3300/0 conn(0x7f7054159440 0x7f70540b3090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:08.313 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.309+0000 7f705974d700 1 -- 172.21.15.98:0/3180370993 >> v1:172.21.15.98:6789/0 conn(0x7f70540cc170 legacy=0x7f70540b2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:08.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.309+0000 7f705974d700 1 -- 172.21.15.98:0/3180370993 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70540cb430 con 0x7f70540b35d0 2023-10-06T05:08:08.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.310+0000 7f705974d700 1 --2- 172.21.15.98:0/3180370993 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70540b35d0 0x7f70540cbc30 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f705406f9f0 tx=0x7f70440091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98871fdabc5eae4e server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:08.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.310+0000 7f7058f4c700 1 -- 172.21.15.98:0/3180370993 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f704400b6b0 con 0x7f70540b35d0 2023-10-06T05:08:08.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.310+0000 7f7058f4c700 1 -- 172.21.15.98:0/3180370993 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f704400b810 con 0x7f70540b35d0 2023-10-06T05:08:08.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.310+0000 7f7058f4c700 1 -- 172.21.15.98:0/3180370993 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f704400bbb0 con 0x7f70540b35d0 2023-10-06T05:08:08.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.310+0000 7f705af50700 1 -- 172.21.15.98:0/3180370993 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70540b35d0 msgr2=0x7f70540cbc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:08.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.310+0000 7f705af50700 1 --2- 172.21.15.98:0/3180370993 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70540b35d0 0x7f70540cbc30 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f705406f9f0 tx=0x7f70440091a0 comp rx=0 tx=0).stop 2023-10-06T05:08:08.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.310+0000 7f705af50700 1 -- 172.21.15.98:0/3180370993 shutdown_connections 2023-10-06T05:08:08.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.310+0000 7f705af50700 1 --2- 172.21.15.98:0/3180370993 >> v2:172.21.15.174:3300/0 conn(0x7f7054159440 0x7f70540b3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:08.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.310+0000 7f705af50700 1 --2- 172.21.15.98:0/3180370993 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70540b35d0 0x7f70540cbc30 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:08.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.310+0000 7f705af50700 1 -- 172.21.15.98:0/3180370993 >> 172.21.15.98:0/3180370993 conn(0x7f705406ece0 msgr2=0x7f705406f0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:08.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.310+0000 7f705af50700 1 -- 172.21.15.98:0/3180370993 shutdown_connections 2023-10-06T05:08:08.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.310+0000 7f705af50700 1 -- 172.21.15.98:0/3180370993 wait complete. 2023-10-06T05:08:08.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.311+0000 7f705af50700 1 Processor -- start 2023-10-06T05:08:08.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.311+0000 7f705af50700 1 -- start start 2023-10-06T05:08:08.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.312+0000 7f705af50700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f70540b35d0 0x7f70540c40f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:08.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.312+0000 7f705af50700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f70540cc170 0x7f705416d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:08.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.312+0000 7f705af50700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7054159440 0x7f70541518f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:08.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.312+0000 7f705af50700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70540cf910 con 0x7f7054159440 2023-10-06T05:08:08.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.312+0000 7f705af50700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f705416dca0 con 0x7f70540cc170 2023-10-06T05:08:08.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.312+0000 7f705af50700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7054154ae0 con 0x7f70540b35d0 2023-10-06T05:08:08.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.312+0000 7f705a74f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7054159440 0x7f70541518f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:08.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.312+0000 7f705a74f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7054159440 0x7f70541518f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55080/0 (socket says 172.21.15.98:55080) 2023-10-06T05:08:08.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.312+0000 7f705a74f700 1 -- 172.21.15.98:0/1666055640 learned_addr learned my addr 172.21.15.98:0/1666055640 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:08:08.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.312+0000 7f705974d700 1 --2- 172.21.15.98:0/1666055640 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f70540cc170 0x7f705416d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:08.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.312+0000 7f7059f4e700 1 --2- 172.21.15.98:0/1666055640 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f70540b35d0 0x7f70540c40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:08.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.312+0000 7f705a74f700 1 -- 172.21.15.98:0/1666055640 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f70540b35d0 msgr2=0x7f70540c40f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:08.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.312+0000 7f705a74f700 1 --2- 172.21.15.98:0/1666055640 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f70540b35d0 0x7f70540c40f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:08.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.312+0000 7f705a74f700 1 -- 172.21.15.98:0/1666055640 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f70540cc170 msgr2=0x7f705416d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:08.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.312+0000 7f705a74f700 1 --2- 172.21.15.98:0/1666055640 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f70540cc170 0x7f705416d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:08.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.312+0000 7f705a74f700 1 -- 172.21.15.98:0/1666055640 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7044009420 con 0x7f7054159440 2023-10-06T05:08:08.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.312+0000 7f705974d700 1 --2- 172.21.15.98:0/1666055640 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f70540cc170 0x7f705416d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:08:08.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.313+0000 7f705a74f700 1 --2- 172.21.15.98:0/1666055640 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7054159440 0x7f70541518f0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f704c00cc40 tx=0x7f704c001a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:08.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.313+0000 7f7049ffb700 1 -- 172.21.15.98:0/1666055640 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f704c00a9c0 con 0x7f7054159440 2023-10-06T05:08:08.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.313+0000 7f7049ffb700 1 -- 172.21.15.98:0/1666055640 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f704c00ab20 con 0x7f7054159440 2023-10-06T05:08:08.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.313+0000 7f705af50700 1 -- 172.21.15.98:0/1666055640 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7054154dc0 con 0x7f7054159440 2023-10-06T05:08:08.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.313+0000 7f7049ffb700 1 -- 172.21.15.98:0/1666055640 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f704c0070f0 con 0x7f7054159440 2023-10-06T05:08:08.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.313+0000 7f705af50700 1 -- 172.21.15.98:0/1666055640 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70540cf250 con 0x7f7054159440 2023-10-06T05:08:08.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.314+0000 7f705af50700 1 -- 172.21.15.98:0/1666055640 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f705405f230 con 0x7f7054159440 2023-10-06T05:08:08.320 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.314+0000 7f7049ffb700 1 -- 172.21.15.98:0/1666055640 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f704c0074e0 con 0x7f7054159440 2023-10-06T05:08:08.320 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.315+0000 7f7049ffb700 1 --2- 172.21.15.98:0/1666055640 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f7038066f30 0x7f70380693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:08.320 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.315+0000 7f7049ffb700 1 -- 172.21.15.98:0/1666055640 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6378+0+0 (secure 0 0 0) 0x7f704c084bf0 con 0x7f7054159440 2023-10-06T05:08:08.320 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.315+0000 7f7059f4e700 1 --2- 172.21.15.98:0/1666055640 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f7038066f30 0x7f70380693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:08.320 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.315+0000 7f7059f4e700 1 --2- 172.21.15.98:0/1666055640 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f7038066f30 0x7f70380693e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f7050006fd0 tx=0x7f7050008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:08.321 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.319+0000 7f7049ffb700 1 -- 172.21.15.98:0/1666055640 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f704c051c40 con 0x7f7054159440 2023-10-06T05:08:08.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.547+0000 7f705af50700 1 -- 172.21.15.98:0/1666055640 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7054159810 con 0x7f7054159440 2023-10-06T05:08:08.550 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.548+0000 7f7049ffb700 1 -- 172.21.15.98:0/1666055640 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4603 (secure 0 0 0) 0x7f704c055260 con 0x7f7054159440 2023-10-06T05:08:08.553 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:08:08.553 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4966,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6834/2320896583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2320896583},{"type":"v1","addr":"172.21.15.174:6834","nonce":2320896583}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":4969,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/1423644921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6833","nonce":1423644921},{"type":"v1","addr":"172.21.15.174:6835","nonce":1423644921}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5100,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/4291295874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":4291295874},{"type":"v1","addr":"172.21.15.98:6836","nonce":4291295874}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-06T05:07:55.873515+0000","modified":"2023-10-06T05:07:59.511013+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5103},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5103":{"gid":5103,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.98:6837/1617154626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1617154626},{"type":"v1","addr":"172.21.15.98:6837","nonce":1617154626}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2023-10-06T05:08:08.554 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.550+0000 7f705af50700 1 -- 172.21.15.98:0/1666055640 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f7038066f30 msgr2=0x7f70380693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:08.555 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.550+0000 7f705af50700 1 --2- 172.21.15.98:0/1666055640 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f7038066f30 0x7f70380693e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f7050006fd0 tx=0x7f7050008040 comp rx=0 tx=0).stop 2023-10-06T05:08:08.555 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.550+0000 7f705af50700 1 -- 172.21.15.98:0/1666055640 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7054159440 msgr2=0x7f70541518f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:08.555 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.550+0000 7f705af50700 1 --2- 172.21.15.98:0/1666055640 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7054159440 0x7f70541518f0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f704c00cc40 tx=0x7f704c001a50 comp rx=0 tx=0).stop 2023-10-06T05:08:08.555 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.550+0000 7f705af50700 1 -- 172.21.15.98:0/1666055640 shutdown_connections 2023-10-06T05:08:08.555 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.550+0000 7f705af50700 1 --2- 172.21.15.98:0/1666055640 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f7038066f30 0x7f70380693e0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:08.555 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.550+0000 7f705af50700 1 --2- 172.21.15.98:0/1666055640 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7054159440 0x7f70541518f0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:08.555 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.550+0000 7f705af50700 1 --2- 172.21.15.98:0/1666055640 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f70540b35d0 0x7f70540c40f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:08.555 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.550+0000 7f705af50700 1 --2- 172.21.15.98:0/1666055640 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f70540cc170 0x7f705416d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:08.555 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.550+0000 7f705af50700 1 -- 172.21.15.98:0/1666055640 >> 172.21.15.98:0/1666055640 conn(0x7f705406ece0 msgr2=0x7f7054155c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:08.555 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.550+0000 7f705af50700 1 -- 172.21.15.98:0/1666055640 shutdown_connections 2023-10-06T05:08:08.556 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.550+0000 7f705af50700 1 -- 172.21.15.98:0/1666055640 wait complete. 2023-10-06T05:08:08.556 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 17 2023-10-06T05:08:08.564 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-06T05:08:08.678 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.676+0000 7fbaa5bc6700 1 Processor -- start 2023-10-06T05:08:08.678 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.676+0000 7fbaa5bc6700 1 -- start start 2023-10-06T05:08:08.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.677+0000 7fbaa5bc6700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fbaa0159440 0x7fbaa00b3090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:08.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.677+0000 7fbaa5bc6700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbaa00b35d0 0x7fbaa00cbc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:08.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.677+0000 7fbaa5bc6700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbaa00cb7a0 con 0x7fbaa00cc170 2023-10-06T05:08:08.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.677+0000 7fbaa5bc6700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbaa00b0860 con 0x7fbaa0159440 2023-10-06T05:08:08.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.677+0000 7fbaa5bc6700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa00cb150 con 0x7fbaa00b35d0 2023-10-06T05:08:08.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.677+0000 7fbaa53c5700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fbaa00cc170 0x7fbaa00b2100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:38432/0 (socket says 172.21.15.98:38432) 2023-10-06T05:08:08.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.677+0000 7fbaa53c5700 1 -- 172.21.15.98:0/2521251711 learned_addr learned my addr 172.21.15.98:0/2521251711 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:08:08.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.677+0000 7fbaa4bc4700 1 --2- 172.21.15.98:0/2521251711 >> v2:172.21.15.174:3300/0 conn(0x7fbaa0159440 0x7fbaa00b3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:08.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.677+0000 7fba9ffff700 1 --2- 172.21.15.98:0/2521251711 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbaa00b35d0 0x7fbaa00cbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:08.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.677+0000 7fba9f7fe700 1 -- 172.21.15.98:0/2521251711 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 452851265 0 0) 0x7fbaa00cb7a0 con 0x7fbaa00cc170 2023-10-06T05:08:08.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.677+0000 7fba9f7fe700 1 -- 172.21.15.98:0/2521251711 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba8c003570 con 0x7fbaa00cc170 2023-10-06T05:08:08.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.678+0000 7fba9f7fe700 1 -- 172.21.15.98:0/2521251711 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3975676683 0 0) 0x7fba8c003570 con 0x7fbaa00cc170 2023-10-06T05:08:08.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.678+0000 7fba9f7fe700 1 -- 172.21.15.98:0/2521251711 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbaa00b35d0 msgr2=0x7fbaa00cbc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:08.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.678+0000 7fba9f7fe700 1 --2- 172.21.15.98:0/2521251711 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbaa00b35d0 0x7fbaa00cbc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:08.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.678+0000 7fba9f7fe700 1 -- 172.21.15.98:0/2521251711 >> v2:172.21.15.174:3300/0 conn(0x7fbaa0159440 msgr2=0x7fbaa00b3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:08.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.678+0000 7fba9f7fe700 1 --2- 172.21.15.98:0/2521251711 >> v2:172.21.15.174:3300/0 conn(0x7fbaa0159440 0x7fbaa00b3090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:08.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.678+0000 7fba9f7fe700 1 -- 172.21.15.98:0/2521251711 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaa00cb430 con 0x7fbaa00cc170 2023-10-06T05:08:08.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.678+0000 7fba9f7fe700 1 -- 172.21.15.98:0/2521251711 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1543991876 0 0) 0x7fba88001470 con 0x7fbaa00cc170 2023-10-06T05:08:08.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.678+0000 7fbaa4bc4700 1 --2- 172.21.15.98:0/2521251711 >> v2:172.21.15.174:3300/0 conn(0x7fbaa0159440 0x7fbaa00b3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T05:08:08.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.678+0000 7fba9f7fe700 1 -- 172.21.15.98:0/2521251711 >> v1:172.21.15.98:6789/0 conn(0x7fbaa00cc170 legacy=0x7fbaa00b2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:08.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.678+0000 7fba9f7fe700 1 --2- 172.21.15.98:0/2521251711 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fba8c003a60 0x7fba8c003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:08.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.678+0000 7fba9f7fe700 1 --2- 172.21.15.98:0/2521251711 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbaa00b35d0 0x7fba8c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:08.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.678+0000 7fba9f7fe700 1 --2- 172.21.15.98:0/2521251711 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba8c004d50 0x7fba8c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:08.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.678+0000 7fba9f7fe700 1 -- 172.21.15.98:0/2521251711 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa00b0860 con 0x7fba8c004d50 2023-10-06T05:08:08.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.678+0000 7fba9f7fe700 1 -- 172.21.15.98:0/2521251711 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa00cb150 con 0x7fba8c003a60 2023-10-06T05:08:08.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.678+0000 7fba9f7fe700 1 -- 172.21.15.98:0/2521251711 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba8c00ba60 con 0x7fbaa00b35d0 2023-10-06T05:08:08.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.678+0000 7fbaa53c5700 1 --2- 172.21.15.98:0/2521251711 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba8c004d50 0x7fba8c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:08.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.678+0000 7fbaa4bc4700 1 --2- 172.21.15.98:0/2521251711 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fba8c003a60 0x7fba8c003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:08.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.678+0000 7fba9ffff700 1 --2- 172.21.15.98:0/2521251711 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbaa00b35d0 0x7fba8c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:08.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.678+0000 7fbaa53c5700 1 -- 172.21.15.98:0/2521251711 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbaa00b35d0 msgr2=0x7fba8c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:08.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.678+0000 7fbaa53c5700 1 --2- 172.21.15.98:0/2521251711 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbaa00b35d0 0x7fba8c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:08.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.678+0000 7fbaa53c5700 1 -- 172.21.15.98:0/2521251711 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fba8c003a60 msgr2=0x7fba8c003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:08.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.678+0000 7fbaa53c5700 1 --2- 172.21.15.98:0/2521251711 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fba8c003a60 0x7fba8c003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:08.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.678+0000 7fbaa53c5700 1 -- 172.21.15.98:0/2521251711 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba8c003570 con 0x7fba8c004d50 2023-10-06T05:08:08.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.678+0000 7fbaa53c5700 1 --2- 172.21.15.98:0/2521251711 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba8c004d50 0x7fba8c0051c0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fba8800c3c0 tx=0x7fba88001e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef54210abb3db429 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:08.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.678+0000 7fba9f7fe700 1 -- 172.21.15.98:0/2521251711 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba8800cc30 con 0x7fba8c004d50 2023-10-06T05:08:08.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.678+0000 7fba9f7fe700 1 -- 172.21.15.98:0/2521251711 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba8800cd90 con 0x7fba8c004d50 2023-10-06T05:08:08.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.678+0000 7fba9f7fe700 1 -- 172.21.15.98:0/2521251711 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba88005b70 con 0x7fba8c004d50 2023-10-06T05:08:08.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.679+0000 7fbaa5bc6700 1 -- 172.21.15.98:0/2521251711 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba8c004d50 msgr2=0x7fba8c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:08.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.679+0000 7fbaa5bc6700 1 --2- 172.21.15.98:0/2521251711 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba8c004d50 0x7fba8c0051c0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fba8800c3c0 tx=0x7fba88001e90 comp rx=0 tx=0).stop 2023-10-06T05:08:08.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.679+0000 7fbaa53c5700 1 -- 172.21.15.98:0/2521251711 reap_dead start 2023-10-06T05:08:08.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.679+0000 7fbaa5bc6700 1 -- 172.21.15.98:0/2521251711 shutdown_connections 2023-10-06T05:08:08.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.679+0000 7fbaa5bc6700 1 -- 172.21.15.98:0/2521251711 >> 172.21.15.98:0/2521251711 conn(0x7fbaa006ece0 msgr2=0x7fbaa006f0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:08.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.679+0000 7fbaa5bc6700 1 -- 172.21.15.98:0/2521251711 shutdown_connections 2023-10-06T05:08:08.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.679+0000 7fbaa5bc6700 1 -- 172.21.15.98:0/2521251711 wait complete. 2023-10-06T05:08:08.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.680+0000 7fbaa5bc6700 1 Processor -- start 2023-10-06T05:08:08.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.680+0000 7fbaa5bc6700 1 -- start start 2023-10-06T05:08:08.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.681+0000 7fbaa5bc6700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fba8c004d50 0x7fbaa00c8510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:08.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.681+0000 7fbaa5bc6700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbaa00c8a50 0x7fbaa016a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:08.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.681+0000 7fbaa5bc6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbaa016a8a0 0x7fbaa015b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:08.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.681+0000 7fbaa5bc6700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa00c8080 con 0x7fbaa016a8a0 2023-10-06T05:08:08.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.681+0000 7fbaa5bc6700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa016ad20 con 0x7fba8c004d50 2023-10-06T05:08:08.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.681+0000 7fbaa5bc6700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa00c7470 con 0x7fbaa00c8a50 2023-10-06T05:08:08.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.681+0000 7fbaa53c5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbaa016a8a0 0x7fbaa015b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:08.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.681+0000 7fbaa53c5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbaa016a8a0 0x7fbaa015b420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55100/0 (socket says 172.21.15.98:55100) 2023-10-06T05:08:08.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.681+0000 7fbaa53c5700 1 -- 172.21.15.98:0/291406112 learned_addr learned my addr 172.21.15.98:0/291406112 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:08:08.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.681+0000 7fba9ffff700 1 --2- 172.21.15.98:0/291406112 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbaa00c8a50 0x7fbaa016a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:08.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.681+0000 7fbaa4bc4700 1 --2- 172.21.15.98:0/291406112 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fba8c004d50 0x7fbaa00c8510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:08.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.681+0000 7fbaa53c5700 1 -- 172.21.15.98:0/291406112 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbaa00c8a50 msgr2=0x7fbaa016a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:08.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.681+0000 7fbaa53c5700 1 --2- 172.21.15.98:0/291406112 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbaa00c8a50 0x7fbaa016a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:08.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.681+0000 7fbaa53c5700 1 -- 172.21.15.98:0/291406112 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fba8c004d50 msgr2=0x7fbaa00c8510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:08.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.681+0000 7fbaa53c5700 1 --2- 172.21.15.98:0/291406112 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fba8c004d50 0x7fbaa00c8510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:08.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.681+0000 7fbaa53c5700 1 -- 172.21.15.98:0/291406112 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaa00c7750 con 0x7fbaa016a8a0 2023-10-06T05:08:08.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.682+0000 7fbaa53c5700 1 --2- 172.21.15.98:0/291406112 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbaa016a8a0 0x7fbaa015b420 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fba88009020 tx=0x7fba88015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:08.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.682+0000 7fba9cff9700 1 -- 172.21.15.98:0/291406112 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba8800cc30 con 0x7fbaa016a8a0 2023-10-06T05:08:08.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.682+0000 7fba9cff9700 1 -- 172.21.15.98:0/291406112 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba88000fe0 con 0x7fbaa016a8a0 2023-10-06T05:08:08.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.682+0000 7fba9cff9700 1 -- 172.21.15.98:0/291406112 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba88012670 con 0x7fbaa016a8a0 2023-10-06T05:08:08.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.682+0000 7fbaa5bc6700 1 -- 172.21.15.98:0/291406112 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaa00c7a30 con 0x7fbaa016a8a0 2023-10-06T05:08:08.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.682+0000 7fbaa5bc6700 1 -- 172.21.15.98:0/291406112 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaa015b960 con 0x7fbaa016a8a0 2023-10-06T05:08:08.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.683+0000 7fbaa5bc6700 1 -- 172.21.15.98:0/291406112 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbaa005f230 con 0x7fbaa016a8a0 2023-10-06T05:08:08.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.683+0000 7fba9cff9700 1 -- 172.21.15.98:0/291406112 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fba88019030 con 0x7fbaa016a8a0 2023-10-06T05:08:08.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.684+0000 7fba9cff9700 1 --2- 172.21.15.98:0/291406112 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fba80066f30 0x7fba800693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:08.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.684+0000 7fbaa4bc4700 1 --2- 172.21.15.98:0/291406112 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fba80066f30 0x7fba800693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:08.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.684+0000 7fba9cff9700 1 -- 172.21.15.98:0/291406112 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6378+0+0 (secure 0 0 0) 0x7fba88085ba0 con 0x7fbaa016a8a0 2023-10-06T05:08:08.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.684+0000 7fbaa4bc4700 1 --2- 172.21.15.98:0/291406112 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fba80066f30 0x7fba800693e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fbaa00d1120 tx=0x7fba98008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:08.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.688+0000 7fba9cff9700 1 -- 172.21.15.98:0/291406112 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fba88052b70 con 0x7fbaa016a8a0 2023-10-06T05:08:08.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.914+0000 7fbaa5bc6700 1 -- 172.21.15.98:0/291406112 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbaa0159810 con 0x7fbaa016a8a0 2023-10-06T05:08:08.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.914+0000 7fba9cff9700 1 -- 172.21.15.98:0/291406112 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4603 (secure 0 0 0) 0x7fba88001430 con 0x7fbaa016a8a0 2023-10-06T05:08:08.917 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:08:08.917 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4966,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6834/2320896583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2320896583},{"type":"v1","addr":"172.21.15.174:6834","nonce":2320896583}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":4969,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/1423644921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6833","nonce":1423644921},{"type":"v1","addr":"172.21.15.174:6835","nonce":1423644921}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5100,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/4291295874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":4291295874},{"type":"v1","addr":"172.21.15.98:6836","nonce":4291295874}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-06T05:07:55.873515+0000","modified":"2023-10-06T05:07:59.511013+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5103},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5103":{"gid":5103,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":4,"addr":"172.21.15.98:6837/1617154626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":1617154626},{"type":"v1","addr":"172.21.15.98:6837","nonce":1617154626}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2023-10-06T05:08:08.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.916+0000 7fbaa5bc6700 1 -- 172.21.15.98:0/291406112 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fba80066f30 msgr2=0x7fba800693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:08.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.916+0000 7fbaa5bc6700 1 --2- 172.21.15.98:0/291406112 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fba80066f30 0x7fba800693e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fbaa00d1120 tx=0x7fba98008040 comp rx=0 tx=0).stop 2023-10-06T05:08:08.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.916+0000 7fbaa5bc6700 1 -- 172.21.15.98:0/291406112 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbaa016a8a0 msgr2=0x7fbaa015b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:08.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.916+0000 7fbaa5bc6700 1 --2- 172.21.15.98:0/291406112 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbaa016a8a0 0x7fbaa015b420 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fba88009020 tx=0x7fba88015000 comp rx=0 tx=0).stop 2023-10-06T05:08:08.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.916+0000 7fbaa5bc6700 1 -- 172.21.15.98:0/291406112 shutdown_connections 2023-10-06T05:08:08.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.916+0000 7fbaa5bc6700 1 --2- 172.21.15.98:0/291406112 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fba80066f30 0x7fba800693e0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:08.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.916+0000 7fbaa5bc6700 1 --2- 172.21.15.98:0/291406112 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbaa016a8a0 0x7fbaa015b420 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:08.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.916+0000 7fbaa5bc6700 1 --2- 172.21.15.98:0/291406112 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fba8c004d50 0x7fbaa00c8510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:08.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.916+0000 7fbaa5bc6700 1 --2- 172.21.15.98:0/291406112 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbaa00c8a50 0x7fbaa016a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:08.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.916+0000 7fbaa5bc6700 1 -- 172.21.15.98:0/291406112 >> 172.21.15.98:0/291406112 conn(0x7fbaa006ece0 msgr2=0x7fbaa0155800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:08.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.917+0000 7fbaa5bc6700 1 -- 172.21.15.98:0/291406112 shutdown_connections 2023-10-06T05:08:08.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:08.917+0000 7fbaa5bc6700 1 -- 172.21.15.98:0/291406112 wait complete. 2023-10-06T05:08:08.920 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 17 2023-10-06T05:08:08.930 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T05:08:09.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.043+0000 7f2df0961700 1 Processor -- start 2023-10-06T05:08:09.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.043+0000 7f2df0961700 1 -- start start 2023-10-06T05:08:09.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.044+0000 7f2df0961700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2dec0b25e0 0x7f2dec0b0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:09.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.044+0000 7f2df0961700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f2dec0b29b0 0x7f2dec159120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:09.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.044+0000 7f2df0961700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2dec159660 0x7f2dec068540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:09.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.044+0000 7f2df0961700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dec0cbfa0 con 0x7f2dec159660 2023-10-06T05:08:09.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.044+0000 7f2df0961700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dec0cb390 con 0x7f2dec0b29b0 2023-10-06T05:08:09.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.044+0000 7f2df0961700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dec0cb670 con 0x7f2dec0b25e0 2023-10-06T05:08:09.049 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.044+0000 7f2dead9d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2dec159660 0x7f2dec068540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:09.049 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.044+0000 7f2dead9d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2dec159660 0x7f2dec068540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55102/0 (socket says 172.21.15.98:55102) 2023-10-06T05:08:09.049 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.044+0000 7f2dead9d700 1 -- 172.21.15.98:0/3791775419 learned_addr learned my addr 172.21.15.98:0/3791775419 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:08:09.049 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.044+0000 7f2dead9d700 1 -- 172.21.15.98:0/3791775419 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2dec0b25e0 msgr2=0x7f2dec0b0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:09.049 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.044+0000 7f2dea59c700 1 --2- 172.21.15.98:0/3791775419 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2dec0b25e0 0x7f2dec0b0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:09.049 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.044+0000 7f2de9d9b700 1 --2- 172.21.15.98:0/3791775419 >> v2:172.21.15.174:3300/0 conn(0x7f2dec0b29b0 0x7f2dec159120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:09.049 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.044+0000 7f2dead9d700 1 --2- 172.21.15.98:0/3791775419 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2dec0b25e0 0x7f2dec0b0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:09.049 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.044+0000 7f2dead9d700 1 -- 172.21.15.98:0/3791775419 >> v2:172.21.15.174:3300/0 conn(0x7f2dec0b29b0 msgr2=0x7f2dec159120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:09.049 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.044+0000 7f2dead9d700 1 --2- 172.21.15.98:0/3791775419 >> v2:172.21.15.174:3300/0 conn(0x7f2dec0b29b0 0x7f2dec159120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:09.049 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.044+0000 7f2dead9d700 1 -- 172.21.15.98:0/3791775419 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dec0cb950 con 0x7f2dec159660 2023-10-06T05:08:09.049 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.044+0000 7f2dea59c700 1 --2- 172.21.15.98:0/3791775419 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2dec0b25e0 0x7f2dec0b0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:08:09.050 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.044+0000 7f2dead9d700 1 --2- 172.21.15.98:0/3791775419 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2dec159660 0x7f2dec068540 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f2dec070560 tx=0x7f2de4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8faf5d72c5853b2c server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:09.050 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.044+0000 7f2de959a700 1 -- 172.21.15.98:0/3791775419 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de400e070 con 0x7f2dec159660 2023-10-06T05:08:09.050 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.044+0000 7f2de959a700 1 -- 172.21.15.98:0/3791775419 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2de400ba50 con 0x7f2dec159660 2023-10-06T05:08:09.050 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.044+0000 7f2de959a700 1 -- 172.21.15.98:0/3791775419 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de40092e0 con 0x7f2dec159660 2023-10-06T05:08:09.050 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.045+0000 7f2df0961700 1 -- 172.21.15.98:0/3791775419 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2dec159660 msgr2=0x7f2dec068540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:09.050 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.045+0000 7f2df0961700 1 --2- 172.21.15.98:0/3791775419 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2dec159660 0x7f2dec068540 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f2dec070560 tx=0x7f2de4009840 comp rx=0 tx=0).stop 2023-10-06T05:08:09.050 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.045+0000 7f2df0961700 1 -- 172.21.15.98:0/3791775419 shutdown_connections 2023-10-06T05:08:09.050 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.045+0000 7f2df0961700 1 --2- 172.21.15.98:0/3791775419 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2dec159660 0x7f2dec068540 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:09.050 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.045+0000 7f2df0961700 1 --2- 172.21.15.98:0/3791775419 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2dec0b25e0 0x7f2dec0b0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:09.050 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.045+0000 7f2df0961700 1 --2- 172.21.15.98:0/3791775419 >> v2:172.21.15.174:3300/0 conn(0x7f2dec0b29b0 0x7f2dec159120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:09.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.045+0000 7f2df0961700 1 -- 172.21.15.98:0/3791775419 >> 172.21.15.98:0/3791775419 conn(0x7f2dec06ece0 msgr2=0x7f2dec06f0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:09.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.045+0000 7f2df0961700 1 -- 172.21.15.98:0/3791775419 shutdown_connections 2023-10-06T05:08:09.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.045+0000 7f2df0961700 1 -- 172.21.15.98:0/3791775419 wait complete. 2023-10-06T05:08:09.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.046+0000 7f2df0961700 1 Processor -- start 2023-10-06T05:08:09.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.046+0000 7f2df0961700 1 -- start start 2023-10-06T05:08:09.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.047+0000 7f2df0961700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2dec0b25e0 0x7f2dec153a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:09.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.047+0000 7f2df0961700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2dec0b29b0 0x7f2dec15f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:09.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.047+0000 7f2df0961700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2dec159660 0x7f2dec15ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:09.054 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.047+0000 7f2df0961700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dec159a60 con 0x7f2dec0b25e0 2023-10-06T05:08:09.054 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.047+0000 7f2df0961700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dec15a8c0 con 0x7f2dec159660 2023-10-06T05:08:09.054 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.047+0000 7f2df0961700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dec159cb0 con 0x7f2dec0b29b0 2023-10-06T05:08:09.054 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.047+0000 7f2dea59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2dec0b25e0 0x7f2dec153a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:09.054 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.047+0000 7f2dea59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2dec0b25e0 0x7f2dec153a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55114/0 (socket says 172.21.15.98:55114) 2023-10-06T05:08:09.054 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.047+0000 7f2dea59c700 1 -- 172.21.15.98:0/2885626662 learned_addr learned my addr 172.21.15.98:0/2885626662 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:08:09.054 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.047+0000 7f2dea59c700 1 -- 172.21.15.98:0/2885626662 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2dec0b29b0 msgr2=0x7f2dec15f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:09.054 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.047+0000 7f2dea59c700 1 --2- 172.21.15.98:0/2885626662 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2dec0b29b0 0x7f2dec15f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:09.054 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.047+0000 7f2dea59c700 1 -- 172.21.15.98:0/2885626662 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2dec159660 msgr2=0x7f2dec15ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:09.054 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.047+0000 7f2dead9d700 1 --2- 172.21.15.98:0/2885626662 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2dec159660 0x7f2dec15ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:09.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.047+0000 7f2dea59c700 1 --2- 172.21.15.98:0/2885626662 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2dec159660 0x7f2dec15ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:09.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.047+0000 7f2dea59c700 1 -- 172.21.15.98:0/2885626662 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2de4009ae0 con 0x7f2dec0b25e0 2023-10-06T05:08:09.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.047+0000 7f2dea59c700 1 --2- 172.21.15.98:0/2885626662 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2dec0b25e0 0x7f2dec153a90 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f2ddc00c830 tx=0x7f2ddc008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:09.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.047+0000 7f2dda7fc700 1 -- 172.21.15.98:0/2885626662 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ddc005a00 con 0x7f2dec0b25e0 2023-10-06T05:08:09.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.048+0000 7f2dda7fc700 1 -- 172.21.15.98:0/2885626662 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ddc005b60 con 0x7f2dec0b25e0 2023-10-06T05:08:09.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.048+0000 7f2df0961700 1 -- 172.21.15.98:0/2885626662 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dec159f90 con 0x7f2dec0b25e0 2023-10-06T05:08:09.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.048+0000 7f2dda7fc700 1 -- 172.21.15.98:0/2885626662 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ddc011600 con 0x7f2dec0b25e0 2023-10-06T05:08:09.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.048+0000 7f2df0961700 1 -- 172.21.15.98:0/2885626662 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dec15a550 con 0x7f2dec0b25e0 2023-10-06T05:08:09.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.049+0000 7f2df0961700 1 -- 172.21.15.98:0/2885626662 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dec15aba0 con 0x7f2dec0b25e0 2023-10-06T05:08:09.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.049+0000 7f2dda7fc700 1 -- 172.21.15.98:0/2885626662 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f2ddc0119f0 con 0x7f2dec0b25e0 2023-10-06T05:08:09.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.049+0000 7f2dda7fc700 1 --2- 172.21.15.98:0/2885626662 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f2dcc066f30 0x7f2dcc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:09.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.049+0000 7f2dda7fc700 1 -- 172.21.15.98:0/2885626662 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6378+0+0 (secure 0 0 0) 0x7f2ddc084720 con 0x7f2dec0b25e0 2023-10-06T05:08:09.056 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.049+0000 7f2de9d9b700 1 --2- 172.21.15.98:0/2885626662 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f2dcc066f30 0x7f2dcc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:09.056 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.050+0000 7f2de9d9b700 1 --2- 172.21.15.98:0/2885626662 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f2dcc066f30 0x7f2dcc0693e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f2dec15e360 tx=0x7f2de0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:09.056 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.054+0000 7f2dda7fc700 1 -- 172.21.15.98:0/2885626662 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2ddc055180 con 0x7f2dec0b25e0 2023-10-06T05:08:09.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.237+0000 7f2df0961700 1 -- 172.21.15.98:0/2885626662 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2dec064a90 con 0x7f2dec0b25e0 2023-10-06T05:08:09.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.238+0000 7f2dda7fc700 1 -- 172.21.15.98:0/2885626662 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v43) v1 ==== 74+0+17071 (secure 0 0 0) 0x7f2ddc054d10 con 0x7f2dec0b25e0 2023-10-06T05:08:09.241 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:08:09.241 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":43,"fsid":"f50c5df5-cad5-47a4-9bb1-8c791d4311ec","created":"2023-10-06T05:06:32.971175+0000","modified":"2023-10-06T05:07:57.482671+0000","last_up_change":"2023-10-06T05:06:44.016489+0000","last_in_change":"2023-10-06T05:06:35.544511+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-06T05:06:44.559671+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-06T05:06:51.432161+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-10-06T05:07:54.125452+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-10-06T05:07:54.831073+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d9495d95-da75-4c3b-8927-e9bae6a8b093","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":90627},{"type":"v1","addr":"172.21.15.98:6801","nonce":90627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":90627},{"type":"v1","addr":"172.21.15.98:6803","nonce":90627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":90627},{"type":"v1","addr":"172.21.15.98:6807","nonce":90627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":90627},{"type":"v1","addr":"172.21.15.98:6805","nonce":90627}]},"public_addr":"172.21.15.98:6801/90627","cluster_addr":"172.21.15.98:6803/90627","heartbeat_back_addr":"172.21.15.98:6807/90627","heartbeat_front_addr":"172.21.15.98:6805/90627","state":["exists","up"]},{"osd":1,"uuid":"f25ba5be-d65e-4707-a134-154092deb754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":90628},{"type":"v1","addr":"172.21.15.98:6809","nonce":90628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":90628},{"type":"v1","addr":"172.21.15.98:6811","nonce":90628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":90628},{"type":"v1","addr":"172.21.15.98:6815","nonce":90628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":90628},{"type":"v1","addr":"172.21.15.98:6813","nonce":90628}]},"public_addr":"172.21.15.98:6809/90628","cluster_addr":"172.21.15.98:6811/90628","heartbeat_back_addr":"172.21.15.98:6815/90628","heartbeat_front_addr":"172.21.15.98:6813/90628","state":["exists","up"]},{"osd":2,"uuid":"5ab9e028-945c-43e2-8c95-3b0b4cf38e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":90662},{"type":"v1","addr":"172.21.15.98:6818","nonce":90662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":90662},{"type":"v1","addr":"172.21.15.98:6822","nonce":90662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":90662},{"type":"v1","addr":"172.21.15.98:6830","nonce":90662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":90662},{"type":"v1","addr":"172.21.15.98:6826","nonce":90662}]},"public_addr":"172.21.15.98:6818/90662","cluster_addr":"172.21.15.98:6822/90662","heartbeat_back_addr":"172.21.15.98:6830/90662","heartbeat_front_addr":"172.21.15.98:6826/90662","state":["exists","up"]},{"osd":3,"uuid":"3bb803ff-1738-46bf-be62-2267bf378822","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6817","nonce":90663},{"type":"v1","addr":"172.21.15.98:6819","nonce":90663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6821","nonce":90663},{"type":"v1","addr":"172.21.15.98:6823","nonce":90663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6829","nonce":90663},{"type":"v1","addr":"172.21.15.98:6831","nonce":90663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6825","nonce":90663},{"type":"v1","addr":"172.21.15.98:6827","nonce":90663}]},"public_addr":"172.21.15.98:6819/90663","cluster_addr":"172.21.15.98:6823/90663","heartbeat_back_addr":"172.21.15.98:6831/90663","heartbeat_front_addr":"172.21.15.98:6827/90663","state":["exists","up"]},{"osd":4,"uuid":"7d57a4c9-6462-43f6-b64a-7f9e9b4353d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":90847},{"type":"v1","addr":"172.21.15.174:6801","nonce":90847}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":90847},{"type":"v1","addr":"172.21.15.174:6803","nonce":90847}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":90847},{"type":"v1","addr":"172.21.15.174:6807","nonce":90847}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":90847},{"type":"v1","addr":"172.21.15.174:6805","nonce":90847}]},"public_addr":"172.21.15.174:6801/90847","cluster_addr":"172.21.15.174:6803/90847","heartbeat_back_addr":"172.21.15.174:6807/90847","heartbeat_front_addr":"172.21.15.174:6805/90847","state":["exists","up"]},{"osd":5,"uuid":"83287ea4-e44d-4af9-bc77-36d47c60d4a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":90848},{"type":"v1","addr":"172.21.15.174:6809","nonce":90848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":90848},{"type":"v1","addr":"172.21.15.174:6811","nonce":90848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":90848},{"type":"v1","addr":"172.21.15.174:6815","nonce":90848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":90848},{"type":"v1","addr":"172.21.15.174:6813","nonce":90848}]},"public_addr":"172.21.15.174:6809/90848","cluster_addr":"172.21.15.174:6811/90848","heartbeat_back_addr":"172.21.15.174:6815/90848","heartbeat_front_addr":"172.21.15.174:6813/90848","state":["exists","up"]},{"osd":6,"uuid":"496a6272-892e-43c4-a145-206623c22f99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":90877},{"type":"v1","addr":"172.21.15.174:6817","nonce":90877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":90877},{"type":"v1","addr":"172.21.15.174:6819","nonce":90877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":90877},{"type":"v1","addr":"172.21.15.174:6823","nonce":90877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":90877},{"type":"v1","addr":"172.21.15.174:6821","nonce":90877}]},"public_addr":"172.21.15.174:6817/90877","cluster_addr":"172.21.15.174:6819/90877","heartbeat_back_addr":"172.21.15.174:6823/90877","heartbeat_front_addr":"172.21.15.174:6821/90877","state":["exists","up"]},{"osd":7,"uuid":"49967e34-8b07-4bab-a13e-0c16d6a0f3a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":90878},{"type":"v1","addr":"172.21.15.174:6825","nonce":90878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":90878},{"type":"v1","addr":"172.21.15.174:6827","nonce":90878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":90878},{"type":"v1","addr":"172.21.15.174:6831","nonce":90878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":90878},{"type":"v1","addr":"172.21.15.174:6829","nonce":90878}]},"public_addr":"172.21.15.174:6825/90878","cluster_addr":"172.21.15.174:6827/90878","heartbeat_back_addr":"172.21.15.174:6831/90878","heartbeat_front_addr":"172.21.15.174:6829/90878","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.446479+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.514266+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.794108+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.810894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.731919+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.717756+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.737435+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.508682+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T05:08:09.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.240+0000 7f2df0961700 1 -- 172.21.15.98:0/2885626662 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f2dcc066f30 msgr2=0x7f2dcc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:09.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.240+0000 7f2df0961700 1 --2- 172.21.15.98:0/2885626662 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f2dcc066f30 0x7f2dcc0693e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f2dec15e360 tx=0x7f2de0008040 comp rx=0 tx=0).stop 2023-10-06T05:08:09.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.240+0000 7f2df0961700 1 -- 172.21.15.98:0/2885626662 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2dec0b25e0 msgr2=0x7f2dec153a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:09.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.240+0000 7f2df0961700 1 --2- 172.21.15.98:0/2885626662 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2dec0b25e0 0x7f2dec153a90 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f2ddc00c830 tx=0x7f2ddc008040 comp rx=0 tx=0).stop 2023-10-06T05:08:09.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.240+0000 7f2df0961700 1 -- 172.21.15.98:0/2885626662 shutdown_connections 2023-10-06T05:08:09.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.240+0000 7f2df0961700 1 --2- 172.21.15.98:0/2885626662 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2dec159660 0x7f2dec15ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:09.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.240+0000 7f2df0961700 1 --2- 172.21.15.98:0/2885626662 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f2dcc066f30 0x7f2dcc0693e0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:09.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.240+0000 7f2df0961700 1 --2- 172.21.15.98:0/2885626662 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2dec0b25e0 0x7f2dec153a90 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:09.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.240+0000 7f2df0961700 1 --2- 172.21.15.98:0/2885626662 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2dec0b29b0 0x7f2dec15f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:09.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.240+0000 7f2df0961700 1 -- 172.21.15.98:0/2885626662 >> 172.21.15.98:0/2885626662 conn(0x7f2dec06ece0 msgr2=0x7f2dec156d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:09.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.240+0000 7f2df0961700 1 -- 172.21.15.98:0/2885626662 shutdown_connections 2023-10-06T05:08:09.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.240+0000 7f2df0961700 1 -- 172.21.15.98:0/2885626662 wait complete. 2023-10-06T05:08:09.254 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-06T05:08:09.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.368+0000 7fedb359e700 1 Processor -- start 2023-10-06T05:08:09.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.368+0000 7fedb359e700 1 -- start start 2023-10-06T05:08:09.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.369+0000 7fedb359e700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fedb4159440 0x7fedb40b3090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:09.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.369+0000 7fedb359e700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fedb40b35d0 0x7fedb40cbc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:09.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.369+0000 7fedb359e700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fedb40cc170 0x7fedb40b2100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:09.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.369+0000 7fedb359e700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedb40cb7a0 con 0x7fedb40cc170 2023-10-06T05:08:09.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.369+0000 7fedb359e700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedb40b0860 con 0x7fedb4159440 2023-10-06T05:08:09.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.369+0000 7fedb359e700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedb40cb150 con 0x7fedb40b35d0 2023-10-06T05:08:09.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.369+0000 7fedb2d9d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fedb40cc170 0x7fedb40b2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:09.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.369+0000 7fedb2d9d700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fedb40cc170 0x7fedb40b2100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49700/0 (socket says 172.21.15.98:49700) 2023-10-06T05:08:09.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.369+0000 7fedb2d9d700 1 -- 172.21.15.98:0/2970811562 learned_addr learned my addr 172.21.15.98:0/2970811562 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:08:09.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.369+0000 7fedb259c700 1 --2- 172.21.15.98:0/2970811562 >> v2:172.21.15.174:3300/0 conn(0x7fedb4159440 0x7fedb40b3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:09.373 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.369+0000 7feda9d9b700 1 --2- 172.21.15.98:0/2970811562 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fedb40b35d0 0x7fedb40cbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:09.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.369+0000 7fedb2d9d700 1 -- 172.21.15.98:0/2970811562 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fedb40b35d0 msgr2=0x7fedb40cbc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:09.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.369+0000 7fedb2d9d700 1 --2- 172.21.15.98:0/2970811562 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fedb40b35d0 0x7fedb40cbc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:09.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.369+0000 7fedb2d9d700 1 -- 172.21.15.98:0/2970811562 >> v2:172.21.15.174:3300/0 conn(0x7fedb4159440 msgr2=0x7fedb40b3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:09.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.369+0000 7fedb2d9d700 1 --2- 172.21.15.98:0/2970811562 >> v2:172.21.15.174:3300/0 conn(0x7fedb4159440 0x7fedb40b3090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:09.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.369+0000 7fedb2d9d700 1 -- 172.21.15.98:0/2970811562 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedb40cb430 con 0x7fedb40cc170 2023-10-06T05:08:09.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.369+0000 7fedb2d9d700 1 --2- 172.21.15.98:0/2970811562 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fedb40cc170 0x7fedb40b2100 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fedb406f9f0 tx=0x7fedac009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59d88f714a01ee11 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:09.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.369+0000 7fedb1d9b700 1 -- 172.21.15.98:0/2970811562 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedac00e070 con 0x7fedb40cc170 2023-10-06T05:08:09.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.370+0000 7fedb1d9b700 1 -- 172.21.15.98:0/2970811562 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedac00ba50 con 0x7fedb40cc170 2023-10-06T05:08:09.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.370+0000 7fedb1d9b700 1 -- 172.21.15.98:0/2970811562 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedac0092e0 con 0x7fedb40cc170 2023-10-06T05:08:09.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.370+0000 7fedb359e700 1 -- 172.21.15.98:0/2970811562 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fedb40cc170 msgr2=0x7fedb40b2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:09.374 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.370+0000 7fedb359e700 1 --2- 172.21.15.98:0/2970811562 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fedb40cc170 0x7fedb40b2100 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fedb406f9f0 tx=0x7fedac009840 comp rx=0 tx=0).stop 2023-10-06T05:08:09.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.370+0000 7fedb359e700 1 -- 172.21.15.98:0/2970811562 shutdown_connections 2023-10-06T05:08:09.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.370+0000 7fedb359e700 1 --2- 172.21.15.98:0/2970811562 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fedb40cc170 0x7fedb40b2100 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:09.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.370+0000 7fedb359e700 1 --2- 172.21.15.98:0/2970811562 >> v2:172.21.15.174:3300/0 conn(0x7fedb4159440 0x7fedb40b3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:09.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.370+0000 7fedb359e700 1 --2- 172.21.15.98:0/2970811562 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fedb40b35d0 0x7fedb40cbc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:09.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.370+0000 7fedb359e700 1 -- 172.21.15.98:0/2970811562 >> 172.21.15.98:0/2970811562 conn(0x7fedb406ece0 msgr2=0x7fedb406f0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:09.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.370+0000 7fedb359e700 1 -- 172.21.15.98:0/2970811562 shutdown_connections 2023-10-06T05:08:09.375 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.370+0000 7fedb359e700 1 -- 172.21.15.98:0/2970811562 wait complete. 2023-10-06T05:08:09.377 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.371+0000 7fedb359e700 1 Processor -- start 2023-10-06T05:08:09.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.371+0000 7fedb359e700 1 -- start start 2023-10-06T05:08:09.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.372+0000 7fedb359e700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fedb40b35d0 0x7fedb40ce800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:09.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.372+0000 7fedb359e700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fedb40cc170 0x7fedb40c8580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:09.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.372+0000 7fedb359e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fedb4159440 0x7fedb40c8ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:09.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.372+0000 7fedb359e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedb40cf7a0 con 0x7fedb4159440 2023-10-06T05:08:09.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.372+0000 7fedb359e700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedb4159fc0 con 0x7fedb40cc170 2023-10-06T05:08:09.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.372+0000 7fedb359e700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedb40cf1e0 con 0x7fedb40b35d0 2023-10-06T05:08:09.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.372+0000 7fedb2d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fedb4159440 0x7fedb40c8ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:09.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.372+0000 7fedb2d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fedb4159440 0x7fedb40c8ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49714/0 (socket says 172.21.15.98:49714) 2023-10-06T05:08:09.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.372+0000 7fedb2d9d700 1 -- 172.21.15.98:0/4076850975 learned_addr learned my addr 172.21.15.98:0/4076850975 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:08:09.378 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.372+0000 7fedb259c700 1 --2- 172.21.15.98:0/4076850975 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fedb40b35d0 0x7fedb40ce800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:09.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.372+0000 7feda9d9b700 1 --2- 172.21.15.98:0/4076850975 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fedb40cc170 0x7fedb40c8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:09.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.372+0000 7fedb2d9d700 1 -- 172.21.15.98:0/4076850975 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fedb40b35d0 msgr2=0x7fedb40ce800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:09.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.372+0000 7fedb2d9d700 1 --2- 172.21.15.98:0/4076850975 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fedb40b35d0 0x7fedb40ce800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:09.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.372+0000 7fedb2d9d700 1 -- 172.21.15.98:0/4076850975 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fedb40cc170 msgr2=0x7fedb40c8580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:09.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.372+0000 7fedb2d9d700 1 --2- 172.21.15.98:0/4076850975 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fedb40cc170 0x7fedb40c8580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:09.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.372+0000 7fedb2d9d700 1 -- 172.21.15.98:0/4076850975 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedac009ae0 con 0x7fedb4159440 2023-10-06T05:08:09.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.373+0000 7fedb2d9d700 1 --2- 172.21.15.98:0/4076850975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fedb4159440 0x7fedb40c8ac0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fedb4069150 tx=0x7fedac018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:09.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.373+0000 7fedaaffd700 1 -- 172.21.15.98:0/4076850975 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedac0094e0 con 0x7fedb4159440 2023-10-06T05:08:09.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.373+0000 7fedaaffd700 1 -- 172.21.15.98:0/4076850975 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedac0032f0 con 0x7fedb4159440 2023-10-06T05:08:09.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.373+0000 7fedb359e700 1 -- 172.21.15.98:0/4076850975 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedb40cf4c0 con 0x7fedb4159440 2023-10-06T05:08:09.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.373+0000 7fedaaffd700 1 -- 172.21.15.98:0/4076850975 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedac0266b0 con 0x7fedb4159440 2023-10-06T05:08:09.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.373+0000 7fedb359e700 1 -- 172.21.15.98:0/4076850975 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedb40d0a20 con 0x7fedb4159440 2023-10-06T05:08:09.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.374+0000 7fedb359e700 1 -- 172.21.15.98:0/4076850975 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedb4159ab0 con 0x7fedb4159440 2023-10-06T05:08:09.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.374+0000 7fedaaffd700 1 -- 172.21.15.98:0/4076850975 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fedac003040 con 0x7fedb4159440 2023-10-06T05:08:09.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.375+0000 7fedaaffd700 1 --2- 172.21.15.98:0/4076850975 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fed94066f30 0x7fed940693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:09.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.375+0000 7fedb259c700 1 --2- 172.21.15.98:0/4076850975 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fed94066f30 0x7fed940693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:09.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.375+0000 7fedaaffd700 1 -- 172.21.15.98:0/4076850975 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6378+0+0 (secure 0 0 0) 0x7fedac086900 con 0x7fedb4159440 2023-10-06T05:08:09.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.375+0000 7fedb259c700 1 --2- 172.21.15.98:0/4076850975 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fed94066f30 0x7fed940693e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7feda4006fd0 tx=0x7feda4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:09.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.379+0000 7fedaaffd700 1 -- 172.21.15.98:0/4076850975 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fedb4159ab0 con 0x7fedb4159440 2023-10-06T05:08:09.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.605+0000 7fedb359e700 1 -- 172.21.15.98:0/4076850975 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fedb4159ab0 con 0x7fedb4159440 2023-10-06T05:08:09.608 INFO:tasks.ceph.mon.a.smithi098.stderr:2023-10-06T05:08:09.606+0000 7fd5c817d700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-06T05:08:09.611 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.609+0000 7fedaaffd700 1 -- 172.21.15.98:0/4076850975 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v18) v1 ==== 161+0+0 (secure 0 0 0) 0x7fedb4159ab0 con 0x7fedb4159440 2023-10-06T05:08:09.612 INFO:tasks.ceph.mds.c.smithi098.stderr: -2312> 2023-10-06T05:07:59.256+0000 7f4a2ed1c700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-06T05:08:09.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.611+0000 7fedb359e700 1 -- 172.21.15.98:0/4076850975 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fed94066f30 msgr2=0x7fed940693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:09.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.611+0000 7fedb359e700 1 --2- 172.21.15.98:0/4076850975 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fed94066f30 0x7fed940693e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7feda4006fd0 tx=0x7feda4008040 comp rx=0 tx=0).stop 2023-10-06T05:08:09.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.611+0000 7fedb359e700 1 -- 172.21.15.98:0/4076850975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fedb4159440 msgr2=0x7fedb40c8ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:09.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.611+0000 7fedb359e700 1 --2- 172.21.15.98:0/4076850975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fedb4159440 0x7fedb40c8ac0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fedb4069150 tx=0x7fedac018000 comp rx=0 tx=0).stop 2023-10-06T05:08:09.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.612+0000 7fedb359e700 1 -- 172.21.15.98:0/4076850975 shutdown_connections 2023-10-06T05:08:09.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.612+0000 7fedb359e700 1 --2- 172.21.15.98:0/4076850975 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fed94066f30 0x7fed940693e0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:09.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.612+0000 7fedb359e700 1 --2- 172.21.15.98:0/4076850975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fedb4159440 0x7fedb40c8ac0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:09.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.612+0000 7fedb359e700 1 --2- 172.21.15.98:0/4076850975 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fedb40b35d0 0x7fedb40ce800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:09.615 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.612+0000 7fedb359e700 1 --2- 172.21.15.98:0/4076850975 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fedb40cc170 0x7fedb40c8580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:09.615 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.612+0000 7fedb359e700 1 -- 172.21.15.98:0/4076850975 >> 172.21.15.98:0/4076850975 conn(0x7fedb406ece0 msgr2=0x7fedb4154fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:09.615 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.612+0000 7fedb359e700 1 -- 172.21.15.98:0/4076850975 shutdown_connections 2023-10-06T05:08:09.615 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.612+0000 7fedb359e700 1 -- 172.21.15.98:0/4076850975 wait complete. 2023-10-06T05:08:09.615 INFO:teuthology.orchestra.run.smithi098.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-06T05:08:09.626 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-06T05:08:09.635 INFO:tasks.ceph.mds.c.smithi098.stdout:starting mds.c at 2023-10-06T05:08:09.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.741+0000 7f172a9d6700 1 Processor -- start 2023-10-06T05:08:09.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.741+0000 7f172a9d6700 1 -- start start 2023-10-06T05:08:09.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.742+0000 7f172a9d6700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f17241595b0 0x7f17240af7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:09.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.742+0000 7f172a9d6700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1724069d10 con 0x7f17240c6f60 2023-10-06T05:08:09.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.742+0000 7f172a9d6700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f172406a890 con 0x7f1724068b80 2023-10-06T05:08:09.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.742+0000 7f172a9d6700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17240678f0 con 0x7f17241595b0 2023-10-06T05:08:09.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.742+0000 7f1723fff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f17241595b0 0x7f17240af7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:09.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.742+0000 7f17237fe700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f17240c6f60 0x7f1724068470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:47040/0 (socket says 172.21.15.98:47040) 2023-10-06T05:08:09.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.742+0000 7f17237fe700 1 -- 172.21.15.98:0/3218162395 learned_addr learned my addr 172.21.15.98:0/3218162395 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:08:09.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.742+0000 7f1723fff700 1 -- 172.21.15.98:0/3218162395 >> v1:172.21.15.174:6789/0 conn(0x7f1724068b80 legacy=0x7f172406ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:09.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.742+0000 7f1723fff700 1 -- 172.21.15.98:0/3218162395 >> v1:172.21.15.98:6789/0 conn(0x7f17240c6f60 legacy=0x7f1724068470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:09.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.742+0000 7f1723fff700 1 -- 172.21.15.98:0/3218162395 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1724069f60 con 0x7f17241595b0 2023-10-06T05:08:09.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.742+0000 7f1722ffd700 1 -- 172.21.15.98:0/3218162395 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2725442548 0 0) 0x7f1724069d10 con 0x7f17240c6f60 2023-10-06T05:08:09.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.742+0000 7f1723fff700 1 --2- 172.21.15.98:0/3218162395 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17241595b0 0x7f17240af7b0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f17140098e0 tx=0x7f17140091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=960c063506a94fc6 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:09.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.743+0000 7f1722ffd700 1 -- 172.21.15.98:0/3218162395 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1714009e80 con 0x7f17241595b0 2023-10-06T05:08:09.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.743+0000 7f1722ffd700 1 -- 172.21.15.98:0/3218162395 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f171400b860 con 0x7f17241595b0 2023-10-06T05:08:09.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.743+0000 7f1722ffd700 1 -- 172.21.15.98:0/3218162395 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f171400bbe0 con 0x7f17241595b0 2023-10-06T05:08:09.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.743+0000 7f172a9d6700 1 -- 172.21.15.98:0/3218162395 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17241595b0 msgr2=0x7f17240af7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:09.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.743+0000 7f172a9d6700 1 --2- 172.21.15.98:0/3218162395 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17241595b0 0x7f17240af7b0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f17140098e0 tx=0x7f17140091a0 comp rx=0 tx=0).stop 2023-10-06T05:08:09.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.743+0000 7f172a9d6700 1 -- 172.21.15.98:0/3218162395 shutdown_connections 2023-10-06T05:08:09.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.743+0000 7f172a9d6700 1 --2- 172.21.15.98:0/3218162395 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17241595b0 0x7f17240af7b0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:09.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.743+0000 7f172a9d6700 1 -- 172.21.15.98:0/3218162395 >> 172.21.15.98:0/3218162395 conn(0x7f17240cd3f0 msgr2=0x7f17240c9390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:09.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.743+0000 7f172a9d6700 1 -- 172.21.15.98:0/3218162395 shutdown_connections 2023-10-06T05:08:09.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.743+0000 7f172a9d6700 1 -- 172.21.15.98:0/3218162395 wait complete. 2023-10-06T05:08:09.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.744+0000 7f172a9d6700 1 Processor -- start 2023-10-06T05:08:09.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.745+0000 7f172a9d6700 1 -- start start 2023-10-06T05:08:09.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.745+0000 7f172a9d6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1724068b80 0x7f17240ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:09.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.745+0000 7f172a9d6700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f17240c6f60 0x7f1724149ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:09.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.745+0000 7f172a9d6700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f17241595b0 0x7f172416d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:09.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.745+0000 7f172a9d6700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17240ae930 con 0x7f1724068b80 2023-10-06T05:08:09.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.745+0000 7f172a9d6700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1724148e20 con 0x7f17240c6f60 2023-10-06T05:08:09.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.745+0000 7f172a9d6700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1724149100 con 0x7f17241595b0 2023-10-06T05:08:09.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.745+0000 7f1723fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1724068b80 0x7f17240ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:09.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.745+0000 7f1723fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1724068b80 0x7f17240ae3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49748/0 (socket says 172.21.15.98:49748) 2023-10-06T05:08:09.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.745+0000 7f1723fff700 1 -- 172.21.15.98:0/82154667 learned_addr learned my addr 172.21.15.98:0/82154667 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:08:09.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.746+0000 7f1728f73700 1 --2- 172.21.15.98:0/82154667 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f17241595b0 0x7f172416d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:09.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.746+0000 7f17237fe700 1 --2- 172.21.15.98:0/82154667 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f17240c6f60 0x7f1724149ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:09.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.746+0000 7f1723fff700 1 -- 172.21.15.98:0/82154667 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f17241595b0 msgr2=0x7f172416d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:09.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.746+0000 7f1723fff700 1 --2- 172.21.15.98:0/82154667 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f17241595b0 0x7f172416d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:09.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.746+0000 7f1723fff700 1 -- 172.21.15.98:0/82154667 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f17240c6f60 msgr2=0x7f1724149ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:09.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.746+0000 7f1723fff700 1 --2- 172.21.15.98:0/82154667 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f17240c6f60 0x7f1724149ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:09.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.746+0000 7f1723fff700 1 -- 172.21.15.98:0/82154667 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1714009420 con 0x7f1724068b80 2023-10-06T05:08:09.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.746+0000 7f1723fff700 1 --2- 172.21.15.98:0/82154667 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1724068b80 0x7f17240ae3f0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f17140091e0 tx=0x7f1714016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:09.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.746+0000 7f170bfff700 1 -- 172.21.15.98:0/82154667 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f171400bbe0 con 0x7f1724068b80 2023-10-06T05:08:09.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.746+0000 7f170bfff700 1 -- 172.21.15.98:0/82154667 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f171401b470 con 0x7f1724068b80 2023-10-06T05:08:09.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.746+0000 7f172a9d6700 1 -- 172.21.15.98:0/82154667 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17241493e0 con 0x7f1724068b80 2023-10-06T05:08:09.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.746+0000 7f170bfff700 1 -- 172.21.15.98:0/82154667 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f171401b810 con 0x7f1724068b80 2023-10-06T05:08:09.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.746+0000 7f172a9d6700 1 -- 172.21.15.98:0/82154667 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17241499a0 con 0x7f1724068b80 2023-10-06T05:08:09.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.747+0000 7f172a9d6700 1 -- 172.21.15.98:0/82154667 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1724149b90 con 0x7f1724068b80 2023-10-06T05:08:09.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.747+0000 7f170bfff700 1 -- 172.21.15.98:0/82154667 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f171401bb80 con 0x7f1724068b80 2023-10-06T05:08:09.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.748+0000 7f170bfff700 1 --2- 172.21.15.98:0/82154667 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f1710066f30 0x7f17100693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:09.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.748+0000 7f170bfff700 1 -- 172.21.15.98:0/82154667 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6464+0+0 (secure 0 0 0) 0x7f1714086470 con 0x7f1724068b80 2023-10-06T05:08:09.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.748+0000 7f17237fe700 1 --2- 172.21.15.98:0/82154667 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f1710066f30 0x7f17100693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:09.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.748+0000 7f17237fe700 1 --2- 172.21.15.98:0/82154667 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f1710066f30 0x7f17100693e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f1724068650 tx=0x7f1718007d30 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:09.754 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.752+0000 7f170bfff700 1 -- 172.21.15.98:0/82154667 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f17140533f0 con 0x7f1724068b80 2023-10-06T05:08:09.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:09.982+0000 7f172a9d6700 1 -- 172.21.15.98:0/82154667 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f1724064a90 con 0x7f1724068b80 2023-10-06T05:08:10.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.610+0000 7f170bfff700 1 -- 172.21.15.98:0/82154667 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v19) v1 ==== 104+0+0 (secure 0 0 0) 0x7f1714056a10 con 0x7f1724068b80 2023-10-06T05:08:10.615 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.613+0000 7f172a9d6700 1 -- 172.21.15.98:0/82154667 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f1710066f30 msgr2=0x7f17100693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:10.615 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.613+0000 7f172a9d6700 1 --2- 172.21.15.98:0/82154667 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f1710066f30 0x7f17100693e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f1724068650 tx=0x7f1718007d30 comp rx=0 tx=0).stop 2023-10-06T05:08:10.616 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.613+0000 7f172a9d6700 1 -- 172.21.15.98:0/82154667 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1724068b80 msgr2=0x7f17240ae3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:10.616 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.613+0000 7f172a9d6700 1 --2- 172.21.15.98:0/82154667 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1724068b80 0x7f17240ae3f0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f17140091e0 tx=0x7f1714016000 comp rx=0 tx=0).stop 2023-10-06T05:08:10.616 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.613+0000 7f172a9d6700 1 -- 172.21.15.98:0/82154667 shutdown_connections 2023-10-06T05:08:10.616 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.613+0000 7f172a9d6700 1 --2- 172.21.15.98:0/82154667 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f17241595b0 0x7f172416d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:10.616 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.613+0000 7f172a9d6700 1 --2- 172.21.15.98:0/82154667 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f1710066f30 0x7f17100693e0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:10.616 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.613+0000 7f172a9d6700 1 --2- 172.21.15.98:0/82154667 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1724068b80 0x7f17240ae3f0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:10.616 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.613+0000 7f172a9d6700 1 --2- 172.21.15.98:0/82154667 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f17240c6f60 0x7f1724149ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:10.616 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.613+0000 7f172a9d6700 1 -- 172.21.15.98:0/82154667 >> 172.21.15.98:0/82154667 conn(0x7f17240cd3f0 msgr2=0x7f17240afdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:10.616 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.613+0000 7f172a9d6700 1 -- 172.21.15.98:0/82154667 shutdown_connections 2023-10-06T05:08:10.616 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.613+0000 7f172a9d6700 1 -- 172.21.15.98:0/82154667 wait complete. 2023-10-06T05:08:10.629 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-06T05:08:10.792 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.789+0000 7f6d3ef93700 1 Processor -- start 2023-10-06T05:08:10.792 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.789+0000 7f6d3ef93700 1 -- start start 2023-10-06T05:08:10.795 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.790+0000 7f6d3ef93700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6d380c3090 0x7f6d380c45c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:10.795 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.790+0000 7f6d3ef93700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d380c4130 con 0x7f6d380b0720 2023-10-06T05:08:10.795 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.790+0000 7f6d3ef93700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d380b17c0 con 0x7f6d380c4b00 2023-10-06T05:08:10.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.790+0000 7f6d3ef93700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d380c3520 con 0x7f6d380c3090 2023-10-06T05:08:10.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.790+0000 7f6d3cd2f700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f6d380b0720 0x7f6d380b10b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:47042/0 (socket says 172.21.15.98:47042) 2023-10-06T05:08:10.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.790+0000 7f6d3cd2f700 1 -- 172.21.15.98:0/325919974 learned_addr learned my addr 172.21.15.98:0/325919974 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:08:10.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.791+0000 7f6d2f7fe700 1 -- 172.21.15.98:0/325919974 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 886316321 0 0) 0x7f6d380c4130 con 0x7f6d380b0720 2023-10-06T05:08:10.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.791+0000 7f6d2f7fe700 1 -- 172.21.15.98:0/325919974 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d20003570 con 0x7f6d380b0720 2023-10-06T05:08:10.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.791+0000 7f6d2ffff700 1 --2- 172.21.15.98:0/325919974 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6d380c3090 0x7f6d380c45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:10.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.791+0000 7f6d2f7fe700 1 -- 172.21.15.98:0/325919974 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1704759321 0 0) 0x7f6d20003570 con 0x7f6d380b0720 2023-10-06T05:08:10.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.791+0000 7f6d2f7fe700 1 -- 172.21.15.98:0/325919974 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6d380c3090 msgr2=0x7f6d380c45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:10.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.791+0000 7f6d2f7fe700 1 --2- 172.21.15.98:0/325919974 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6d380c3090 0x7f6d380c45c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:10.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.791+0000 7f6d2f7fe700 1 -- 172.21.15.98:0/325919974 >> v1:172.21.15.174:6789/0 conn(0x7f6d380c4b00 legacy=0x7f6d380cc170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:10.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.791+0000 7f6d2f7fe700 1 -- 172.21.15.98:0/325919974 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d380c3800 con 0x7f6d380b0720 2023-10-06T05:08:10.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.791+0000 7f6d2f7fe700 1 -- 172.21.15.98:0/325919974 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1543991876 0 0) 0x7f6d340015f0 con 0x7f6d380b0720 2023-10-06T05:08:10.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.791+0000 7f6d2f7fe700 1 -- 172.21.15.98:0/325919974 >> v1:172.21.15.98:6789/0 conn(0x7f6d380b0720 legacy=0x7f6d380b10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:10.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.791+0000 7f6d2f7fe700 1 --2- 172.21.15.98:0/325919974 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6d20003a90 0x7f6d20003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:10.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.791+0000 7f6d2f7fe700 1 --2- 172.21.15.98:0/325919974 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d20004810 0x7f6d20004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:10.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.791+0000 7f6d2f7fe700 1 --2- 172.21.15.98:0/325919974 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6d380c3090 0x7f6d20005290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:10.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.791+0000 7f6d2f7fe700 1 -- 172.21.15.98:0/325919974 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d380c3520 con 0x7f6d20004810 2023-10-06T05:08:10.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.791+0000 7f6d2f7fe700 1 -- 172.21.15.98:0/325919974 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d2000bc20 con 0x7f6d20003a90 2023-10-06T05:08:10.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.791+0000 7f6d2f7fe700 1 -- 172.21.15.98:0/325919974 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d2000bde0 con 0x7f6d380c3090 2023-10-06T05:08:10.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.791+0000 7f6d2ffff700 1 --2- 172.21.15.98:0/325919974 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d20004810 0x7f6d20004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:10.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.791+0000 7f6d3cd2f700 1 --2- 172.21.15.98:0/325919974 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6d20003a90 0x7f6d20003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:10.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.791+0000 7f6d3d530700 1 --2- 172.21.15.98:0/325919974 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6d380c3090 0x7f6d20005290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:10.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.791+0000 7f6d2ffff700 1 -- 172.21.15.98:0/325919974 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6d380c3090 msgr2=0x7f6d20005290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:10.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.791+0000 7f6d2ffff700 1 --2- 172.21.15.98:0/325919974 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6d380c3090 0x7f6d20005290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:10.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.791+0000 7f6d2ffff700 1 -- 172.21.15.98:0/325919974 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6d20003a90 msgr2=0x7f6d20003ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:10.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.791+0000 7f6d2ffff700 1 --2- 172.21.15.98:0/325919974 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6d20003a90 0x7f6d20003ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:10.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.791+0000 7f6d2ffff700 1 -- 172.21.15.98:0/325919974 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d20003570 con 0x7f6d20004810 2023-10-06T05:08:10.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.792+0000 7f6d2ffff700 1 --2- 172.21.15.98:0/325919974 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d20004810 0x7f6d20004c80 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f6d2800bb60 tx=0x7f6d28008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27fb3e489f7fc6c4 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:10.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.792+0000 7f6d2f7fe700 1 -- 172.21.15.98:0/325919974 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d2800d5d0 con 0x7f6d20004810 2023-10-06T05:08:10.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.792+0000 7f6d2f7fe700 1 -- 172.21.15.98:0/325919974 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d2800d730 con 0x7f6d20004810 2023-10-06T05:08:10.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.792+0000 7f6d2f7fe700 1 -- 172.21.15.98:0/325919974 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d2800daf0 con 0x7f6d20004810 2023-10-06T05:08:10.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.792+0000 7f6d3ef93700 1 -- 172.21.15.98:0/325919974 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d20004810 msgr2=0x7f6d20004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:10.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.792+0000 7f6d3ef93700 1 --2- 172.21.15.98:0/325919974 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d20004810 0x7f6d20004c80 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f6d2800bb60 tx=0x7f6d28008040 comp rx=0 tx=0).stop 2023-10-06T05:08:10.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.792+0000 7f6d3d530700 1 -- 172.21.15.98:0/325919974 reap_dead start 2023-10-06T05:08:10.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.792+0000 7f6d3ef93700 1 -- 172.21.15.98:0/325919974 shutdown_connections 2023-10-06T05:08:10.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.792+0000 7f6d3ef93700 1 -- 172.21.15.98:0/325919974 >> 172.21.15.98:0/325919974 conn(0x7f6d38087fc0 msgr2=0x7f6d380ce2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:10.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.792+0000 7f6d3ef93700 1 -- 172.21.15.98:0/325919974 shutdown_connections 2023-10-06T05:08:10.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.792+0000 7f6d3ef93700 1 -- 172.21.15.98:0/325919974 wait complete. 2023-10-06T05:08:10.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.793+0000 7f6d3ef93700 1 Processor -- start 2023-10-06T05:08:10.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.794+0000 7f6d3ef93700 1 -- start start 2023-10-06T05:08:10.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.794+0000 7f6d3ef93700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d20004810 0x7f6d38156020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:10.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.794+0000 7f6d3ef93700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6d3809b490 0x7f6d3809b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:10.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.794+0000 7f6d3ef93700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6d380b1970 0x7f6d380b1da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:10.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.794+0000 7f6d3ef93700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d380c72d0 con 0x7f6d20004810 2023-10-06T05:08:10.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.794+0000 7f6d3ef93700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d3809bfa0 con 0x7f6d380b1970 2023-10-06T05:08:10.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.794+0000 7f6d3ef93700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d380c6e40 con 0x7f6d3809b490 2023-10-06T05:08:10.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.794+0000 7f6d3cd2f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d20004810 0x7f6d38156020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:10.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.794+0000 7f6d3cd2f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d20004810 0x7f6d38156020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49760/0 (socket says 172.21.15.98:49760) 2023-10-06T05:08:10.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.794+0000 7f6d3cd2f700 1 -- 172.21.15.98:0/3071835603 learned_addr learned my addr 172.21.15.98:0/3071835603 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:08:10.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.794+0000 7f6d3d530700 1 --2- 172.21.15.98:0/3071835603 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6d380b1970 0x7f6d380b1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:10.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.794+0000 7f6d2ffff700 1 --2- 172.21.15.98:0/3071835603 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6d3809b490 0x7f6d3809b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:10.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.795+0000 7f6d3cd2f700 1 -- 172.21.15.98:0/3071835603 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6d3809b490 msgr2=0x7f6d3809b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:10.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.795+0000 7f6d3cd2f700 1 --2- 172.21.15.98:0/3071835603 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6d3809b490 0x7f6d3809b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:10.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.795+0000 7f6d3cd2f700 1 -- 172.21.15.98:0/3071835603 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6d380b1970 msgr2=0x7f6d380b1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:10.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.795+0000 7f6d3cd2f700 1 --2- 172.21.15.98:0/3071835603 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6d380b1970 0x7f6d380b1da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:10.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.795+0000 7f6d3cd2f700 1 -- 172.21.15.98:0/3071835603 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d380b22e0 con 0x7f6d20004810 2023-10-06T05:08:10.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.795+0000 7f6d3d530700 1 --2- 172.21.15.98:0/3071835603 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6d380b1970 0x7f6d380b1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:08:10.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.795+0000 7f6d3cd2f700 1 --2- 172.21.15.98:0/3071835603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d20004810 0x7f6d38156020 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f6d3400fc00 tx=0x7f6d34009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:10.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.795+0000 7f6d2cff9700 1 -- 172.21.15.98:0/3071835603 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d340073b0 con 0x7f6d20004810 2023-10-06T05:08:10.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.795+0000 7f6d2cff9700 1 -- 172.21.15.98:0/3071835603 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d34007530 con 0x7f6d20004810 2023-10-06T05:08:10.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.795+0000 7f6d3ef93700 1 -- 172.21.15.98:0/3071835603 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d380c8400 con 0x7f6d20004810 2023-10-06T05:08:10.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.795+0000 7f6d2cff9700 1 -- 172.21.15.98:0/3071835603 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d34005a40 con 0x7f6d20004810 2023-10-06T05:08:10.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.795+0000 7f6d3ef93700 1 -- 172.21.15.98:0/3071835603 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d380c67f0 con 0x7f6d20004810 2023-10-06T05:08:10.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.796+0000 7f6d3ef93700 1 -- 172.21.15.98:0/3071835603 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d38064a90 con 0x7f6d20004810 2023-10-06T05:08:10.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.796+0000 7f6d2cff9700 1 -- 172.21.15.98:0/3071835603 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f6d3400d6b0 con 0x7f6d20004810 2023-10-06T05:08:10.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.796+0000 7f6d2cff9700 1 --2- 172.21.15.98:0/3071835603 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f6d24066f60 0x7f6d24069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:10.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.797+0000 7f6d2cff9700 1 -- 172.21.15.98:0/3071835603 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6464+0+0 (secure 0 0 0) 0x7f6d34084af0 con 0x7f6d20004810 2023-10-06T05:08:10.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.797+0000 7f6d2ffff700 1 --2- 172.21.15.98:0/3071835603 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f6d24066f60 0x7f6d24069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:10.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.797+0000 7f6d2ffff700 1 --2- 172.21.15.98:0/3071835603 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f6d24066f60 0x7f6d24069410 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f6d2800bb60 tx=0x7f6d2801b000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:10.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.801+0000 7f6d2cff9700 1 -- 172.21.15.98:0/3071835603 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6d34051a70 con 0x7f6d20004810 2023-10-06T05:08:10.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:10.982+0000 7f6d3ef93700 1 -- 172.21.15.98:0/3071835603 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f6d380c3ae0 con 0x7f6d20004810 2023-10-06T05:08:11.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.753+0000 7f6d2cff9700 1 -- 172.21.15.98:0/3071835603 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v45) v1 ==== 181+0+0 (secure 0 0 0) 0x7f6d34005db0 con 0x7f6d20004810 2023-10-06T05:08:11.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.756+0000 7f6d3ef93700 1 -- 172.21.15.98:0/3071835603 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f6d24066f60 msgr2=0x7f6d24069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:11.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.756+0000 7f6d3ef93700 1 --2- 172.21.15.98:0/3071835603 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f6d24066f60 0x7f6d24069410 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f6d2800bb60 tx=0x7f6d2801b000 comp rx=0 tx=0).stop 2023-10-06T05:08:11.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.756+0000 7f6d3ef93700 1 -- 172.21.15.98:0/3071835603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d20004810 msgr2=0x7f6d38156020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:11.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.756+0000 7f6d3ef93700 1 --2- 172.21.15.98:0/3071835603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d20004810 0x7f6d38156020 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f6d3400fc00 tx=0x7f6d34009040 comp rx=0 tx=0).stop 2023-10-06T05:08:11.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.756+0000 7f6d3ef93700 1 -- 172.21.15.98:0/3071835603 shutdown_connections 2023-10-06T05:08:11.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.756+0000 7f6d3ef93700 1 --2- 172.21.15.98:0/3071835603 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6d380b1970 0x7f6d380b1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:11.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.756+0000 7f6d3ef93700 1 --2- 172.21.15.98:0/3071835603 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f6d24066f60 0x7f6d24069410 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:11.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.756+0000 7f6d3ef93700 1 --2- 172.21.15.98:0/3071835603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d20004810 0x7f6d38156020 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:11.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.756+0000 7f6d3ef93700 1 --2- 172.21.15.98:0/3071835603 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6d3809b490 0x7f6d3809b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:11.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.756+0000 7f6d3ef93700 1 -- 172.21.15.98:0/3071835603 >> 172.21.15.98:0/3071835603 conn(0x7f6d38087fc0 msgr2=0x7f6d380b2fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:11.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.756+0000 7f6d3ef93700 1 -- 172.21.15.98:0/3071835603 shutdown_connections 2023-10-06T05:08:11.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.756+0000 7f6d3ef93700 1 -- 172.21.15.98:0/3071835603 wait complete. 2023-10-06T05:08:11.760 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'cephfs_metadata' removed 2023-10-06T05:08:11.772 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-06T05:08:11.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.892+0000 7ff1168bb700 1 Processor -- start 2023-10-06T05:08:11.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.892+0000 7ff1168bb700 1 -- start start 2023-10-06T05:08:11.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.893+0000 7ff1168bb700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ff110155f20 0x7ff1100af630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:11.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.893+0000 7ff1168bb700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff11006b2f0 con 0x7ff1100c6cc0 2023-10-06T05:08:11.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.893+0000 7ff1168bb700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1100c7780 con 0x7ff110068cd0 2023-10-06T05:08:11.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.893+0000 7ff1168bb700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1100c7a60 con 0x7ff110155f20 2023-10-06T05:08:11.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.893+0000 7ff1150b8700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7ff1100c6cc0 0x7ff1100685c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:47058/0 (socket says 172.21.15.98:47058) 2023-10-06T05:08:11.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.893+0000 7ff1150b8700 1 -- 172.21.15.98:0/1933988791 learned_addr learned my addr 172.21.15.98:0/1933988791 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:08:11.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.894+0000 7ff1148b7700 1 -- 172.21.15.98:0/1933988791 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1008675451 0 0) 0x7ff11006b2f0 con 0x7ff1100c6cc0 2023-10-06T05:08:11.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.894+0000 7ff1148b7700 1 -- 172.21.15.98:0/1933988791 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0fc003570 con 0x7ff1100c6cc0 2023-10-06T05:08:11.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.894+0000 7ff1158b9700 1 --2- 172.21.15.98:0/1933988791 >> v2:172.21.15.174:3300/0 conn(0x7ff110155f20 0x7ff1100af630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:11.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.894+0000 7ff1148b7700 1 -- 172.21.15.98:0/1933988791 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4134121875 0 0) 0x7ff0fc003570 con 0x7ff1100c6cc0 2023-10-06T05:08:11.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.894+0000 7ff1148b7700 1 -- 172.21.15.98:0/1933988791 >> v2:172.21.15.174:3300/0 conn(0x7ff110155f20 msgr2=0x7ff1100af630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:11.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.894+0000 7ff1148b7700 1 --2- 172.21.15.98:0/1933988791 >> v2:172.21.15.174:3300/0 conn(0x7ff110155f20 0x7ff1100af630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:11.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.894+0000 7ff1148b7700 1 -- 172.21.15.98:0/1933988791 >> v1:172.21.15.174:6789/0 conn(0x7ff110068cd0 legacy=0x7ff11006b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:11.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.894+0000 7ff1148b7700 1 -- 172.21.15.98:0/1933988791 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1100c7d40 con 0x7ff1100c6cc0 2023-10-06T05:08:11.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.894+0000 7ff1148b7700 1 -- 172.21.15.98:0/1933988791 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1543991876 0 0) 0x7ff100001470 con 0x7ff1100c6cc0 2023-10-06T05:08:11.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.894+0000 7ff1148b7700 1 -- 172.21.15.98:0/1933988791 >> v1:172.21.15.98:6789/0 conn(0x7ff1100c6cc0 legacy=0x7ff1100685c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:11.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.894+0000 7ff1148b7700 1 --2- 172.21.15.98:0/1933988791 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0fc003a60 0x7ff0fc003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:11.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.894+0000 7ff1148b7700 1 --2- 172.21.15.98:0/1933988791 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff0fc004810 0x7ff0fc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:11.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.894+0000 7ff1148b7700 1 --2- 172.21.15.98:0/1933988791 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff0fc0051c0 0x7ff0fc00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:11.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.894+0000 7ff1148b7700 1 -- 172.21.15.98:0/1933988791 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1100c7a60 con 0x7ff0fc003a60 2023-10-06T05:08:11.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.894+0000 7ff1148b7700 1 -- 172.21.15.98:0/1933988791 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0fc00c0d0 con 0x7ff0fc004810 2023-10-06T05:08:11.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.894+0000 7ff1148b7700 1 -- 172.21.15.98:0/1933988791 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0fc00c290 con 0x7ff0fc0051c0 2023-10-06T05:08:11.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.894+0000 7ff1158b9700 1 --2- 172.21.15.98:0/1933988791 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0fc003a60 0x7ff0fc003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:11.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.894+0000 7ff1150b8700 1 --2- 172.21.15.98:0/1933988791 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff0fc004810 0x7ff0fc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:11.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.894+0000 7ff1160ba700 1 --2- 172.21.15.98:0/1933988791 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff0fc0051c0 0x7ff0fc00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:11.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.894+0000 7ff1158b9700 1 -- 172.21.15.98:0/1933988791 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff0fc0051c0 msgr2=0x7ff0fc00ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:11.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.894+0000 7ff1158b9700 1 --2- 172.21.15.98:0/1933988791 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff0fc0051c0 0x7ff0fc00ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:11.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.894+0000 7ff1158b9700 1 -- 172.21.15.98:0/1933988791 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff0fc004810 msgr2=0x7ff0fc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:11.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.894+0000 7ff1158b9700 1 --2- 172.21.15.98:0/1933988791 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff0fc004810 0x7ff0fc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:11.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.894+0000 7ff1158b9700 1 -- 172.21.15.98:0/1933988791 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0fc003570 con 0x7ff0fc003a60 2023-10-06T05:08:11.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.894+0000 7ff1150b8700 1 --2- 172.21.15.98:0/1933988791 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff0fc004810 0x7ff0fc004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:08:11.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.894+0000 7ff1160ba700 1 -- 172.21.15.98:0/1933988791 reap_dead start 2023-10-06T05:08:11.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.894+0000 7ff1158b9700 1 --2- 172.21.15.98:0/1933988791 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0fc003a60 0x7ff0fc003e70 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7ff10c009c00 tx=0x7ff10c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9c180e616e8b665 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:11.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.894+0000 7ff1148b7700 1 -- 172.21.15.98:0/1933988791 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff10c00e710 con 0x7ff0fc003a60 2023-10-06T05:08:11.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.895+0000 7ff1148b7700 1 -- 172.21.15.98:0/1933988791 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff10c00e870 con 0x7ff0fc003a60 2023-10-06T05:08:11.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.895+0000 7ff1148b7700 1 -- 172.21.15.98:0/1933988791 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff10c00ec30 con 0x7ff0fc003a60 2023-10-06T05:08:11.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.895+0000 7ff1168bb700 1 -- 172.21.15.98:0/1933988791 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0fc003a60 msgr2=0x7ff0fc003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:11.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.895+0000 7ff1168bb700 1 --2- 172.21.15.98:0/1933988791 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0fc003a60 0x7ff0fc003e70 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7ff10c009c00 tx=0x7ff10c0091a0 comp rx=0 tx=0).stop 2023-10-06T05:08:11.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.895+0000 7ff1168bb700 1 -- 172.21.15.98:0/1933988791 shutdown_connections 2023-10-06T05:08:11.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.895+0000 7ff1168bb700 1 --2- 172.21.15.98:0/1933988791 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0fc003a60 0x7ff0fc003e70 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:11.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.895+0000 7ff1168bb700 1 -- 172.21.15.98:0/1933988791 >> 172.21.15.98:0/1933988791 conn(0x7ff1100c9c60 msgr2=0x7ff11006da50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:11.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.895+0000 7ff1168bb700 1 -- 172.21.15.98:0/1933988791 shutdown_connections 2023-10-06T05:08:11.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.895+0000 7ff1168bb700 1 -- 172.21.15.98:0/1933988791 wait complete. 2023-10-06T05:08:11.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.896+0000 7ff1168bb700 1 Processor -- start 2023-10-06T05:08:11.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.896+0000 7ff1168bb700 1 -- start start 2023-10-06T05:08:11.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.897+0000 7ff1168bb700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0fc003a60 0x7ff1100b1780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:11.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.897+0000 7ff1168bb700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1100c2810 0x7ff110147d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:11.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.897+0000 7ff1168bb700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff1100c2c80 0x7ff1101482d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:11.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.897+0000 7ff1168bb700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff110067fc0 con 0x7ff0fc003a60 2023-10-06T05:08:11.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.897+0000 7ff1168bb700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1100ac7e0 con 0x7ff1100c2810 2023-10-06T05:08:11.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.897+0000 7ff1168bb700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff110157790 con 0x7ff1100c2c80 2023-10-06T05:08:11.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.897+0000 7ff1158b9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0fc003a60 0x7ff1100b1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:11.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.897+0000 7ff1158b9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0fc003a60 0x7ff1100b1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49776/0 (socket says 172.21.15.98:49776) 2023-10-06T05:08:11.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.897+0000 7ff1158b9700 1 -- 172.21.15.98:0/3794842335 learned_addr learned my addr 172.21.15.98:0/3794842335 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:08:11.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.898+0000 7ff1150b8700 1 --2- 172.21.15.98:0/3794842335 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1100c2810 0x7ff110147d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:11.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.898+0000 7ff1160ba700 1 --2- 172.21.15.98:0/3794842335 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff1100c2c80 0x7ff1101482d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:11.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.898+0000 7ff1158b9700 1 -- 172.21.15.98:0/3794842335 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff1100c2c80 msgr2=0x7ff1101482d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:11.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.898+0000 7ff1158b9700 1 --2- 172.21.15.98:0/3794842335 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff1100c2c80 0x7ff1101482d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:11.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.898+0000 7ff1158b9700 1 -- 172.21.15.98:0/3794842335 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1100c2810 msgr2=0x7ff110147d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:11.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.898+0000 7ff1158b9700 1 --2- 172.21.15.98:0/3794842335 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1100c2810 0x7ff110147d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:11.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.898+0000 7ff1158b9700 1 -- 172.21.15.98:0/3794842335 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff110157a70 con 0x7ff0fc003a60 2023-10-06T05:08:11.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.898+0000 7ff1150b8700 1 --2- 172.21.15.98:0/3794842335 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1100c2810 0x7ff110147d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:08:11.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.898+0000 7ff1158b9700 1 --2- 172.21.15.98:0/3794842335 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0fc003a60 0x7ff1100b1780 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7ff10c00e4b0 tx=0x7ff10c01a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:11.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.898+0000 7ff105ffb700 1 -- 172.21.15.98:0/3794842335 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff10c022070 con 0x7ff0fc003a60 2023-10-06T05:08:11.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.898+0000 7ff105ffb700 1 -- 172.21.15.98:0/3794842335 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff10c00e9e0 con 0x7ff0fc003a60 2023-10-06T05:08:11.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.898+0000 7ff1168bb700 1 -- 172.21.15.98:0/3794842335 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff110157d50 con 0x7ff0fc003a60 2023-10-06T05:08:11.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.898+0000 7ff105ffb700 1 -- 172.21.15.98:0/3794842335 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff10c027470 con 0x7ff0fc003a60 2023-10-06T05:08:11.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.898+0000 7ff1168bb700 1 -- 172.21.15.98:0/3794842335 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1100b1cc0 con 0x7ff0fc003a60 2023-10-06T05:08:11.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.900+0000 7ff105ffb700 1 -- 172.21.15.98:0/3794842335 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7ff10c01c450 con 0x7ff0fc003a60 2023-10-06T05:08:11.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.901+0000 7ff1168bb700 1 -- 172.21.15.98:0/3794842335 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1100ac950 con 0x7ff0fc003a60 2023-10-06T05:08:11.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.901+0000 7ff105ffb700 1 --2- 172.21.15.98:0/3794842335 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7ff0f4066f30 0x7ff0f40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:11.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.901+0000 7ff105ffb700 1 -- 172.21.15.98:0/3794842335 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6026+0+0 (secure 0 0 0) 0x7ff10c086170 con 0x7ff0fc003a60 2023-10-06T05:08:11.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.901+0000 7ff1150b8700 1 --2- 172.21.15.98:0/3794842335 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7ff0f4066f30 0x7ff0f40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:11.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.901+0000 7ff1150b8700 1 --2- 172.21.15.98:0/3794842335 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7ff0f4066f30 0x7ff0f40693e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ff100000e10 tx=0x7ff100009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:11.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:11.906+0000 7ff105ffb700 1 -- 172.21.15.98:0/3794842335 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff1100ac950 con 0x7ff0fc003a60 2023-10-06T05:08:12.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.094+0000 7ff1168bb700 1 -- 172.21.15.98:0/3794842335 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7ff1100ac950 con 0x7ff0fc003a60 2023-10-06T05:08:12.759 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.757+0000 7ff105ffb700 1 -- 172.21.15.98:0/3794842335 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v46) v1 ==== 169+0+0 (secure 0 0 0) 0x7ff1100ac950 con 0x7ff0fc003a60 2023-10-06T05:08:12.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.759+0000 7ff1168bb700 1 -- 172.21.15.98:0/3794842335 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7ff0f4066f30 msgr2=0x7ff0f40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:12.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.759+0000 7ff1168bb700 1 --2- 172.21.15.98:0/3794842335 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7ff0f4066f30 0x7ff0f40693e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ff100000e10 tx=0x7ff100009040 comp rx=0 tx=0).stop 2023-10-06T05:08:12.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.759+0000 7ff1168bb700 1 -- 172.21.15.98:0/3794842335 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0fc003a60 msgr2=0x7ff1100b1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:12.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.759+0000 7ff1168bb700 1 --2- 172.21.15.98:0/3794842335 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0fc003a60 0x7ff1100b1780 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7ff10c00e4b0 tx=0x7ff10c01a000 comp rx=0 tx=0).stop 2023-10-06T05:08:12.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.759+0000 7ff1168bb700 1 -- 172.21.15.98:0/3794842335 shutdown_connections 2023-10-06T05:08:12.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.759+0000 7ff1168bb700 1 --2- 172.21.15.98:0/3794842335 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff1100c2c80 0x7ff1101482d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:12.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.759+0000 7ff1168bb700 1 --2- 172.21.15.98:0/3794842335 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7ff0f4066f30 0x7ff0f40693e0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:12.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.759+0000 7ff1168bb700 1 --2- 172.21.15.98:0/3794842335 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0fc003a60 0x7ff1100b1780 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:12.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.759+0000 7ff1168bb700 1 --2- 172.21.15.98:0/3794842335 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1100c2810 0x7ff110147d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:12.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.759+0000 7ff1168bb700 1 -- 172.21.15.98:0/3794842335 >> 172.21.15.98:0/3794842335 conn(0x7ff1100c9c60 msgr2=0x7ff1100c5740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:12.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.759+0000 7ff1168bb700 1 -- 172.21.15.98:0/3794842335 shutdown_connections 2023-10-06T05:08:12.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.759+0000 7ff1168bb700 1 -- 172.21.15.98:0/3794842335 wait complete. 2023-10-06T05:08:12.763 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'cephfs_data' removed 2023-10-06T05:08:12.774 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-06T05:08:12.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.899+0000 7efcaffff700 1 Processor -- start 2023-10-06T05:08:12.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.899+0000 7efcaffff700 1 -- start start 2023-10-06T05:08:12.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.900+0000 7efcaffff700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7efcb00cd3c0 0x7efcb00ceaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:12.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.900+0000 7efcaffff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcb00cc970 con 0x7efcb00b2620 2023-10-06T05:08:12.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.900+0000 7efcaffff700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcb00cbd60 con 0x7efcb00b29f0 2023-10-06T05:08:12.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.900+0000 7efcaffff700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcb00cc040 con 0x7efcb00cd3c0 2023-10-06T05:08:12.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.900+0000 7efcaeffd700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7efcb00b2620 0x7efcb00cccb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:47062/0 (socket says 172.21.15.98:47062) 2023-10-06T05:08:12.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.900+0000 7efcaeffd700 1 -- 172.21.15.98:0/153621780 learned_addr learned my addr 172.21.15.98:0/153621780 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:08:12.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.900+0000 7efca67fc700 1 --2- 172.21.15.98:0/153621780 >> v2:172.21.15.174:3300/0 conn(0x7efcb00cd3c0 0x7efcb00ceaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:12.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.900+0000 7efcae7fc700 1 -- 172.21.15.98:0/153621780 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3225708550 0 0) 0x7efcb00cc970 con 0x7efcb00b2620 2023-10-06T05:08:12.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.900+0000 7efcae7fc700 1 -- 172.21.15.98:0/153621780 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc9c003570 con 0x7efcb00b2620 2023-10-06T05:08:12.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.901+0000 7efcae7fc700 1 -- 172.21.15.98:0/153621780 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1492924064 0 0) 0x7efc9c003570 con 0x7efcb00b2620 2023-10-06T05:08:12.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.901+0000 7efcae7fc700 1 -- 172.21.15.98:0/153621780 >> v2:172.21.15.174:3300/0 conn(0x7efcb00cd3c0 msgr2=0x7efcb00ceaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:12.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.901+0000 7efcae7fc700 1 --2- 172.21.15.98:0/153621780 >> v2:172.21.15.174:3300/0 conn(0x7efcb00cd3c0 0x7efcb00ceaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:12.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.901+0000 7efcae7fc700 1 -- 172.21.15.98:0/153621780 >> v1:172.21.15.174:6789/0 conn(0x7efcb00b29f0 legacy=0x7efcb00c5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:12.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.901+0000 7efcae7fc700 1 -- 172.21.15.98:0/153621780 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcb00cc320 con 0x7efcb00b2620 2023-10-06T05:08:12.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.901+0000 7efcae7fc700 1 -- 172.21.15.98:0/153621780 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1543991876 0 0) 0x7efca00015f0 con 0x7efcb00b2620 2023-10-06T05:08:12.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.901+0000 7efcae7fc700 1 -- 172.21.15.98:0/153621780 >> v1:172.21.15.98:6789/0 conn(0x7efcb00b2620 legacy=0x7efcb00cccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:12.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.901+0000 7efcae7fc700 1 --2- 172.21.15.98:0/153621780 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc9c003a60 0x7efc9c003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:12.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.901+0000 7efcae7fc700 1 --2- 172.21.15.98:0/153621780 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efc9c004810 0x7efc9c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:12.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.901+0000 7efcae7fc700 1 --2- 172.21.15.98:0/153621780 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efc9c0051c0 0x7efc9c00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:12.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.901+0000 7efcae7fc700 1 -- 172.21.15.98:0/153621780 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb00cc040 con 0x7efc9c003a60 2023-10-06T05:08:12.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.901+0000 7efcae7fc700 1 -- 172.21.15.98:0/153621780 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc9c00c050 con 0x7efc9c004810 2023-10-06T05:08:12.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.901+0000 7efcae7fc700 1 -- 172.21.15.98:0/153621780 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc9c00c210 con 0x7efc9c0051c0 2023-10-06T05:08:12.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.901+0000 7efcaeffd700 1 --2- 172.21.15.98:0/153621780 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc9c003a60 0x7efc9c003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:12.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.901+0000 7efca67fc700 1 --2- 172.21.15.98:0/153621780 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efc9c004810 0x7efc9c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:12.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.901+0000 7efcaf7fe700 1 --2- 172.21.15.98:0/153621780 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efc9c0051c0 0x7efc9c00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:12.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.901+0000 7efcaeffd700 1 -- 172.21.15.98:0/153621780 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efc9c0051c0 msgr2=0x7efc9c00ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:12.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.901+0000 7efcaeffd700 1 --2- 172.21.15.98:0/153621780 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efc9c0051c0 0x7efc9c00ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:12.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.901+0000 7efcaeffd700 1 -- 172.21.15.98:0/153621780 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efc9c004810 msgr2=0x7efc9c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:12.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.901+0000 7efcaeffd700 1 --2- 172.21.15.98:0/153621780 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efc9c004810 0x7efc9c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:12.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.901+0000 7efcaeffd700 1 -- 172.21.15.98:0/153621780 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc9c003570 con 0x7efc9c003a60 2023-10-06T05:08:12.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.901+0000 7efcaf7fe700 1 -- 172.21.15.98:0/153621780 reap_dead start 2023-10-06T05:08:12.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.901+0000 7efcaeffd700 1 --2- 172.21.15.98:0/153621780 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc9c003a60 0x7efc9c003e30 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7efca000c300 tx=0x7efca0007920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90cf6534fcbfc067 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:12.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.901+0000 7efcae7fc700 1 -- 172.21.15.98:0/153621780 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca000cc30 con 0x7efc9c003a60 2023-10-06T05:08:12.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.901+0000 7efcae7fc700 1 -- 172.21.15.98:0/153621780 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efca000cdb0 con 0x7efc9c003a60 2023-10-06T05:08:12.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.901+0000 7efcae7fc700 1 -- 172.21.15.98:0/153621780 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca0001de0 con 0x7efc9c003a60 2023-10-06T05:08:12.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.902+0000 7efcaffff700 1 -- 172.21.15.98:0/153621780 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc9c003a60 msgr2=0x7efc9c003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:12.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.902+0000 7efcaffff700 1 --2- 172.21.15.98:0/153621780 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc9c003a60 0x7efc9c003e30 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7efca000c300 tx=0x7efca0007920 comp rx=0 tx=0).stop 2023-10-06T05:08:12.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.902+0000 7efcaffff700 1 -- 172.21.15.98:0/153621780 shutdown_connections 2023-10-06T05:08:12.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.902+0000 7efcaffff700 1 --2- 172.21.15.98:0/153621780 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc9c003a60 0x7efc9c003e30 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:12.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.902+0000 7efcaffff700 1 -- 172.21.15.98:0/153621780 >> 172.21.15.98:0/153621780 conn(0x7efcb0099520 msgr2=0x7efcb006f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:12.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.902+0000 7efcaffff700 1 -- 172.21.15.98:0/153621780 shutdown_connections 2023-10-06T05:08:12.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.902+0000 7efcaffff700 1 -- 172.21.15.98:0/153621780 wait complete. 2023-10-06T05:08:12.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.903+0000 7efcaffff700 1 Processor -- start 2023-10-06T05:08:12.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.903+0000 7efcaffff700 1 -- start start 2023-10-06T05:08:12.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.904+0000 7efcaffff700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efc9c003a60 0x7efcb015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:12.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.904+0000 7efcaffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efcb0155390 0x7efcb00c7b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:12.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.904+0000 7efcaffff700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efcb00c80c0 0x7efcb00ce950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:12.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.904+0000 7efcaffff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb015ae40 con 0x7efcb0155390 2023-10-06T05:08:12.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.904+0000 7efcaffff700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb015b120 con 0x7efcb00c80c0 2023-10-06T05:08:12.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.904+0000 7efcaffff700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcb015b400 con 0x7efc9c003a60 2023-10-06T05:08:12.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.904+0000 7efca67fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efcb0155390 0x7efcb00c7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:12.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.904+0000 7efca67fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efcb0155390 0x7efcb00c7b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49796/0 (socket says 172.21.15.98:49796) 2023-10-06T05:08:12.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.904+0000 7efca67fc700 1 -- 172.21.15.98:0/3854836018 learned_addr learned my addr 172.21.15.98:0/3854836018 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:08:12.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.904+0000 7efcaeffd700 1 --2- 172.21.15.98:0/3854836018 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efc9c003a60 0x7efcb015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:12.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.904+0000 7efcaf7fe700 1 --2- 172.21.15.98:0/3854836018 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efcb00c80c0 0x7efcb00ce950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:12.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.904+0000 7efca67fc700 1 -- 172.21.15.98:0/3854836018 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efc9c003a60 msgr2=0x7efcb015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:12.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.904+0000 7efca67fc700 1 --2- 172.21.15.98:0/3854836018 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efc9c003a60 0x7efcb015bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:12.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.904+0000 7efca67fc700 1 -- 172.21.15.98:0/3854836018 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efcb00c80c0 msgr2=0x7efcb00ce950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:12.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.904+0000 7efca67fc700 1 --2- 172.21.15.98:0/3854836018 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efcb00c80c0 0x7efcb00ce950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:12.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.904+0000 7efca67fc700 1 -- 172.21.15.98:0/3854836018 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcb015b6e0 con 0x7efcb0155390 2023-10-06T05:08:12.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.904+0000 7efca67fc700 1 --2- 172.21.15.98:0/3854836018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efcb0155390 0x7efcb00c7b80 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7efc98011b40 tx=0x7efc980091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:12.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.904+0000 7efca77fe700 1 -- 172.21.15.98:0/3854836018 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc9800fa70 con 0x7efcb0155390 2023-10-06T05:08:12.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.905+0000 7efca77fe700 1 -- 172.21.15.98:0/3854836018 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc9800fbd0 con 0x7efcb0155390 2023-10-06T05:08:12.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.905+0000 7efcaffff700 1 -- 172.21.15.98:0/3854836018 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcb015b9c0 con 0x7efcb0155390 2023-10-06T05:08:12.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.905+0000 7efca77fe700 1 -- 172.21.15.98:0/3854836018 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc98014440 con 0x7efcb0155390 2023-10-06T05:08:12.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.905+0000 7efcaffff700 1 -- 172.21.15.98:0/3854836018 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcb0155800 con 0x7efcb0155390 2023-10-06T05:08:12.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.906+0000 7efcaffff700 1 -- 172.21.15.98:0/3854836018 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcb015bd30 con 0x7efcb0155390 2023-10-06T05:08:12.912 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.906+0000 7efca77fe700 1 -- 172.21.15.98:0/3854836018 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7efc98014870 con 0x7efcb0155390 2023-10-06T05:08:12.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.906+0000 7efca77fe700 1 --2- 172.21.15.98:0/3854836018 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7efc90066f30 0x7efc900693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:12.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.906+0000 7efcaeffd700 1 --2- 172.21.15.98:0/3854836018 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7efc90066f30 0x7efc900693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:12.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.906+0000 7efca77fe700 1 -- 172.21.15.98:0/3854836018 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5612+0+0 (secure 0 0 0) 0x7efc98018030 con 0x7efcb0155390 2023-10-06T05:08:12.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.907+0000 7efcaeffd700 1 --2- 172.21.15.98:0/3854836018 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7efc90066f30 0x7efc900693e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7efca000c1a0 tx=0x7efca0015000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:12.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:12.911+0000 7efca77fe700 1 -- 172.21.15.98:0/3854836018 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7efc98052210 con 0x7efcb0155390 2023-10-06T05:08:13.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.096+0000 7efcaffff700 1 -- 172.21.15.98:0/3854836018 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7efcb005f230 con 0x7efcb0155390 2023-10-06T05:08:13.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.100+0000 7efca77fe700 1 -- 172.21.15.98:0/3854836018 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v11) v1 ==== 104+0+0 (secure 0 0 0) 0x7efc98059e60 con 0x7efcb0155390 2023-10-06T05:08:13.104 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.102+0000 7efcaffff700 1 -- 172.21.15.98:0/3854836018 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7efc90066f30 msgr2=0x7efc900693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:13.104 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.102+0000 7efcaffff700 1 --2- 172.21.15.98:0/3854836018 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7efc90066f30 0x7efc900693e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7efca000c1a0 tx=0x7efca0015000 comp rx=0 tx=0).stop 2023-10-06T05:08:13.104 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.102+0000 7efcaffff700 1 -- 172.21.15.98:0/3854836018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efcb0155390 msgr2=0x7efcb00c7b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:13.104 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.102+0000 7efcaffff700 1 --2- 172.21.15.98:0/3854836018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efcb0155390 0x7efcb00c7b80 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7efc98011b40 tx=0x7efc980091a0 comp rx=0 tx=0).stop 2023-10-06T05:08:13.104 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.102+0000 7efcaffff700 1 -- 172.21.15.98:0/3854836018 shutdown_connections 2023-10-06T05:08:13.104 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.102+0000 7efcaffff700 1 --2- 172.21.15.98:0/3854836018 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efcb00c80c0 0x7efcb00ce950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:13.105 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.102+0000 7efcaffff700 1 --2- 172.21.15.98:0/3854836018 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efc9c003a60 0x7efcb015bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:13.105 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.102+0000 7efcaffff700 1 --2- 172.21.15.98:0/3854836018 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7efc90066f30 0x7efc900693e0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:13.105 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.102+0000 7efcaffff700 1 --2- 172.21.15.98:0/3854836018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efcb0155390 0x7efcb00c7b80 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:13.105 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.102+0000 7efcaffff700 1 -- 172.21.15.98:0/3854836018 >> 172.21.15.98:0/3854836018 conn(0x7efcb0099520 msgr2=0x7efcb015d120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:13.105 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.102+0000 7efcaffff700 1 -- 172.21.15.98:0/3854836018 shutdown_connections 2023-10-06T05:08:13.105 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.102+0000 7efcaffff700 1 -- 172.21.15.98:0/3854836018 wait complete. 2023-10-06T05:08:13.116 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_readahead.TestReadahead.test_flush' 2023-10-06T05:08:13.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.227+0000 7f590ff85700 1 Processor -- start 2023-10-06T05:08:13.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.227+0000 7f590ff85700 1 -- start start 2023-10-06T05:08:13.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.228+0000 7f590ff85700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f5908156350 0x7f59080cc1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:13.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.228+0000 7f590ff85700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f590806c740 con 0x7f59080cc710 2023-10-06T05:08:13.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.228+0000 7f590ff85700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f590806c9a0 con 0x7f590806c330 2023-10-06T05:08:13.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.228+0000 7f590ff85700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59080ccb90 con 0x7f5908156350 2023-10-06T05:08:13.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.228+0000 7f590d520700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f59080cc710 0x7f590806bc20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:47072/0 (socket says 172.21.15.98:47072) 2023-10-06T05:08:13.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.228+0000 7f590d520700 1 -- 172.21.15.98:0/3555900470 learned_addr learned my addr 172.21.15.98:0/3555900470 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:08:13.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.229+0000 7f590dd21700 1 --2- 172.21.15.98:0/3555900470 >> v2:172.21.15.174:3300/0 conn(0x7f5908156350 0x7f59080cc1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:13.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.229+0000 7f590cd1f700 1 -- 172.21.15.98:0/3555900470 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 903066678 0 0) 0x7f590806c740 con 0x7f59080cc710 2023-10-06T05:08:13.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.229+0000 7f590cd1f700 1 -- 172.21.15.98:0/3555900470 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58f0003570 con 0x7f59080cc710 2023-10-06T05:08:13.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.229+0000 7f590cd1f700 1 -- 172.21.15.98:0/3555900470 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1001709319 0 0) 0x7f590806c9a0 con 0x7f590806c330 2023-10-06T05:08:13.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.229+0000 7f590cd1f700 1 -- 172.21.15.98:0/3555900470 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f590806c740 con 0x7f590806c330 2023-10-06T05:08:13.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.229+0000 7f590cd1f700 1 -- 172.21.15.98:0/3555900470 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3395090567 0 0) 0x7f58f0003570 con 0x7f59080cc710 2023-10-06T05:08:13.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.229+0000 7f590cd1f700 1 -- 172.21.15.98:0/3555900470 >> v2:172.21.15.174:3300/0 conn(0x7f5908156350 msgr2=0x7f59080cc1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:13.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.229+0000 7f590cd1f700 1 --2- 172.21.15.98:0/3555900470 >> v2:172.21.15.174:3300/0 conn(0x7f5908156350 0x7f59080cc1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:13.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.229+0000 7f590cd1f700 1 -- 172.21.15.98:0/3555900470 >> v1:172.21.15.174:6789/0 conn(0x7f590806c330 legacy=0x7f59080c8880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:13.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.229+0000 7f590cd1f700 1 -- 172.21.15.98:0/3555900470 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5908156b20 con 0x7f59080cc710 2023-10-06T05:08:13.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.229+0000 7f590cd1f700 1 -- 172.21.15.98:0/3555900470 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1543991876 0 0) 0x7f58fc0017f0 con 0x7f59080cc710 2023-10-06T05:08:13.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.229+0000 7f590cd1f700 1 -- 172.21.15.98:0/3555900470 >> v1:172.21.15.98:6789/0 conn(0x7f59080cc710 legacy=0x7f590806bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:13.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.229+0000 7f590cd1f700 1 --2- 172.21.15.98:0/3555900470 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f58f0003bf0 0x7f58f0003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:13.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.229+0000 7f590cd1f700 1 --2- 172.21.15.98:0/3555900470 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f58f0004810 0x7f58f0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:13.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.229+0000 7f590cd1f700 1 --2- 172.21.15.98:0/3555900470 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58f00051c0 0x7f58f000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:13.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.229+0000 7f590cd1f700 1 -- 172.21.15.98:0/3555900470 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59080ccb90 con 0x7f58f00051c0 2023-10-06T05:08:13.236 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.229+0000 7f590cd1f700 1 -- 172.21.15.98:0/3555900470 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58f000c0b0 con 0x7f58f0003bf0 2023-10-06T05:08:13.236 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.229+0000 7f590cd1f700 1 -- 172.21.15.98:0/3555900470 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58f000c270 con 0x7f58f0004810 2023-10-06T05:08:13.236 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.229+0000 7f590e522700 1 --2- 172.21.15.98:0/3555900470 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58f00051c0 0x7f58f000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:13.236 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.229+0000 7f590dd21700 1 --2- 172.21.15.98:0/3555900470 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f58f0003bf0 0x7f58f0003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:13.236 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.229+0000 7f590d520700 1 --2- 172.21.15.98:0/3555900470 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f58f0004810 0x7f58f0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:13.236 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.229+0000 7f590e522700 1 -- 172.21.15.98:0/3555900470 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f58f0004810 msgr2=0x7f58f0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:13.236 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.229+0000 7f590e522700 1 --2- 172.21.15.98:0/3555900470 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f58f0004810 0x7f58f0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:13.236 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.229+0000 7f590e522700 1 -- 172.21.15.98:0/3555900470 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f58f0003bf0 msgr2=0x7f58f0003fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:13.236 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.229+0000 7f590e522700 1 --2- 172.21.15.98:0/3555900470 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f58f0003bf0 0x7f58f0003fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:13.236 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.229+0000 7f590e522700 1 -- 172.21.15.98:0/3555900470 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58f0003570 con 0x7f58f00051c0 2023-10-06T05:08:13.236 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.229+0000 7f590d520700 1 --2- 172.21.15.98:0/3555900470 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f58f0004810 0x7f58f0004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:08:13.236 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.230+0000 7f590e522700 1 -- 172.21.15.98:0/3555900470 reap_dead start 2023-10-06T05:08:13.237 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.230+0000 7f590e522700 1 --2- 172.21.15.98:0/3555900470 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58f00051c0 0x7f58f000ba60 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f58f8001a10 tx=0x7f58f8007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d355244da833902a server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:13.237 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.230+0000 7f590cd1f700 1 -- 172.21.15.98:0/3555900470 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f800c5d0 con 0x7f58f00051c0 2023-10-06T05:08:13.237 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.230+0000 7f590cd1f700 1 -- 172.21.15.98:0/3555900470 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58f800c730 con 0x7f58f00051c0 2023-10-06T05:08:13.237 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.230+0000 7f590cd1f700 1 -- 172.21.15.98:0/3555900470 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f800cad0 con 0x7f58f00051c0 2023-10-06T05:08:13.237 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.230+0000 7f590ff85700 1 -- 172.21.15.98:0/3555900470 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58f00051c0 msgr2=0x7f58f000ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:13.237 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.230+0000 7f590ff85700 1 --2- 172.21.15.98:0/3555900470 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58f00051c0 0x7f58f000ba60 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f58f8001a10 tx=0x7f58f8007d30 comp rx=0 tx=0).stop 2023-10-06T05:08:13.237 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.230+0000 7f590ff85700 1 -- 172.21.15.98:0/3555900470 shutdown_connections 2023-10-06T05:08:13.237 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.230+0000 7f590ff85700 1 --2- 172.21.15.98:0/3555900470 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f58f00051c0 0x7f58f000ba60 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:13.237 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.230+0000 7f590ff85700 1 -- 172.21.15.98:0/3555900470 >> 172.21.15.98:0/3555900470 conn(0x7f59080cdeb0 msgr2=0x7f59080c9e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:13.237 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.230+0000 7f590ff85700 1 -- 172.21.15.98:0/3555900470 shutdown_connections 2023-10-06T05:08:13.237 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.230+0000 7f590ff85700 1 -- 172.21.15.98:0/3555900470 wait complete. 2023-10-06T05:08:13.237 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.231+0000 7f590ff85700 1 Processor -- start 2023-10-06T05:08:13.238 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.231+0000 7f590ff85700 1 -- start start 2023-10-06T05:08:13.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.232+0000 7f590ff85700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f58f00051c0 0x7f590813fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:13.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.232+0000 7f590ff85700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59080b02e0 0x7f59080b2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:13.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.232+0000 7f590ff85700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59080c66a0 0x7f59080c6b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:13.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.232+0000 7f590ff85700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59080b0880 con 0x7f59080c66a0 2023-10-06T05:08:13.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.232+0000 7f590ff85700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59081403e0 con 0x7f59080b02e0 2023-10-06T05:08:13.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.232+0000 7f590ff85700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f590814c9f0 con 0x7f58f00051c0 2023-10-06T05:08:13.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.232+0000 7f590e522700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59080c66a0 0x7f59080c6b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:13.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.232+0000 7f590e522700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59080c66a0 0x7f59080c6b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49828/0 (socket says 172.21.15.98:49828) 2023-10-06T05:08:13.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.232+0000 7f590e522700 1 -- 172.21.15.98:0/655838925 learned_addr learned my addr 172.21.15.98:0/655838925 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:08:13.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.232+0000 7f590d520700 1 --2- 172.21.15.98:0/655838925 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59080b02e0 0x7f59080b2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:13.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.232+0000 7f590dd21700 1 --2- 172.21.15.98:0/655838925 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f58f00051c0 0x7f590813fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:13.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.232+0000 7f590e522700 1 -- 172.21.15.98:0/655838925 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f58f00051c0 msgr2=0x7f590813fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:13.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.232+0000 7f590e522700 1 --2- 172.21.15.98:0/655838925 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f58f00051c0 0x7f590813fea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:13.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.232+0000 7f590e522700 1 -- 172.21.15.98:0/655838925 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59080b02e0 msgr2=0x7f59080b2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:13.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.232+0000 7f590e522700 1 --2- 172.21.15.98:0/655838925 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59080b02e0 0x7f59080b2b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:13.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.233+0000 7f590e522700 1 -- 172.21.15.98:0/655838925 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59080b30b0 con 0x7f59080c66a0 2023-10-06T05:08:13.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.233+0000 7f590d520700 1 --2- 172.21.15.98:0/655838925 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59080b02e0 0x7f59080b2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:08:13.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.233+0000 7f590dd21700 1 --2- 172.21.15.98:0/655838925 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f58f00051c0 0x7f590813fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:08:13.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.233+0000 7f590e522700 1 --2- 172.21.15.98:0/655838925 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59080c66a0 0x7f59080c6b20 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f58f800c3c0 tx=0x7f58f8014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:13.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.233+0000 7f5905ffb700 1 -- 172.21.15.98:0/655838925 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f800ce80 con 0x7f59080c66a0 2023-10-06T05:08:13.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.233+0000 7f5905ffb700 1 -- 172.21.15.98:0/655838925 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58f8007250 con 0x7f59080c66a0 2023-10-06T05:08:13.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.233+0000 7f590ff85700 1 -- 172.21.15.98:0/655838925 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f590814e020 con 0x7f59080c66a0 2023-10-06T05:08:13.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.233+0000 7f5905ffb700 1 -- 172.21.15.98:0/655838925 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f8007610 con 0x7f59080c66a0 2023-10-06T05:08:13.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.233+0000 7f590ff85700 1 -- 172.21.15.98:0/655838925 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59080b1970 con 0x7f59080c66a0 2023-10-06T05:08:13.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.234+0000 7f590ff85700 1 -- 172.21.15.98:0/655838925 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5908064a90 con 0x7f59080c66a0 2023-10-06T05:08:13.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.235+0000 7f5905ffb700 1 -- 172.21.15.98:0/655838925 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f58f8013430 con 0x7f59080c66a0 2023-10-06T05:08:13.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.235+0000 7f5905ffb700 1 --2- 172.21.15.98:0/655838925 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f58f4066f30 0x7f58f40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:13.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.235+0000 7f5905ffb700 1 -- 172.21.15.98:0/655838925 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5612+0+0 (secure 0 0 0) 0x7f58f8084e80 con 0x7f59080c66a0 2023-10-06T05:08:13.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.235+0000 7f590dd21700 1 --2- 172.21.15.98:0/655838925 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f58f4066f30 0x7f58f40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:13.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.235+0000 7f590dd21700 1 --2- 172.21.15.98:0/655838925 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f58f4066f30 0x7f58f40693e0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f5900006f00 tx=0x7f5900008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:13.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.239+0000 7f5905ffb700 1 -- 172.21.15.98:0/655838925 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f58f80521d0 con 0x7f59080c66a0 2023-10-06T05:08:13.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.500+0000 7f590ff85700 1 -- 172.21.15.98:0/655838925 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_readahead.TestReadahead.test_flush"]} v 0) v1 -- 0x7f59080b30b0 con 0x7f59080c66a0 2023-10-06T05:08:13.760 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.758+0000 7f5905ffb700 1 -- 172.21.15.98:0/655838925 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_readahead.TestReadahead.test_flush"]}]=0 v97) v1 ==== 131+0+0 (secure 0 0 0) 0x7f59080b30b0 con 0x7f59080c66a0 2023-10-06T05:08:13.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.760+0000 7f590ff85700 1 -- 172.21.15.98:0/655838925 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f58f4066f30 msgr2=0x7f58f40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:13.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.760+0000 7f590ff85700 1 --2- 172.21.15.98:0/655838925 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f58f4066f30 0x7f58f40693e0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f5900006f00 tx=0x7f5900008040 comp rx=0 tx=0).stop 2023-10-06T05:08:13.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.760+0000 7f590ff85700 1 -- 172.21.15.98:0/655838925 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59080c66a0 msgr2=0x7f59080c6b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:13.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.760+0000 7f590ff85700 1 --2- 172.21.15.98:0/655838925 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59080c66a0 0x7f59080c6b20 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f58f800c3c0 tx=0x7f58f8014000 comp rx=0 tx=0).stop 2023-10-06T05:08:13.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.760+0000 7f590ff85700 1 -- 172.21.15.98:0/655838925 shutdown_connections 2023-10-06T05:08:13.763 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.760+0000 7f590ff85700 1 --2- 172.21.15.98:0/655838925 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f58f4066f30 0x7f58f40693e0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:13.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.760+0000 7f590ff85700 1 --2- 172.21.15.98:0/655838925 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59080c66a0 0x7f59080c6b20 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:13.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.760+0000 7f590ff85700 1 --2- 172.21.15.98:0/655838925 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f58f00051c0 0x7f590813fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:13.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.760+0000 7f590ff85700 1 --2- 172.21.15.98:0/655838925 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59080b02e0 0x7f59080b2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:13.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.760+0000 7f590ff85700 1 -- 172.21.15.98:0/655838925 >> 172.21.15.98:0/655838925 conn(0x7f59080cdeb0 msgr2=0x7f59080cbed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:13.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.760+0000 7f590ff85700 1 -- 172.21.15.98:0/655838925 shutdown_connections 2023-10-06T05:08:13.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:13.760+0000 7f590ff85700 1 -- 172.21.15.98:0/655838925 wait complete. 2023-10-06T05:08:13.775 INFO:tasks.cephfs_test_runner:test_flush (tasks.cephfs.test_readahead.TestReadahead) ... ok 2023-10-06T05:08:13.775 INFO:tasks.cephfs_test_runner: 2023-10-06T05:08:13.776 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2023-10-06T05:08:13.776 INFO:tasks.cephfs_test_runner:Ran 1 test in 36.126s 2023-10-06T05:08:13.776 INFO:tasks.cephfs_test_runner: 2023-10-06T05:08:13.776 INFO:tasks.cephfs_test_runner:OK 2023-10-06T05:08:13.776 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2023-10-06T05:08:13.786 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2023-10-06T05:08:13.805 INFO:tasks.kclient:Unmounting kernel clients... 2023-10-06T05:08:13.805 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:08:13.805 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list 2023-10-06T05:08:13.821 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-06T05:08:13.821 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-06T05:08:13.821 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:08:13.821 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-06T05:08:13.905 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:08:13.905 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-06T05:08:13.958 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:08:13.958 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link delete ceph-brx 2023-10-06T05:08:14.031 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:08:14.031 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list 2023-10-06T05:08:14.045 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:08:14.046 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link delete ceph-brx 2023-10-06T05:08:14.114 INFO:teuthology.orchestra.run.smithi098.stderr:Cannot find device "ceph-brx" 2023-10-06T05:08:14.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-06T05:08:14.116 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:08:14.116 DEBUG:teuthology.orchestra.run.smithi174:> ip netns list 2023-10-06T05:08:14.134 INFO:teuthology.orchestra.run.smithi174.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2023-10-06T05:08:14.134 INFO:teuthology.orchestra.run.smithi174.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-10-06T05:08:14.135 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:08:14.135 DEBUG:teuthology.orchestra.run.smithi174:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2023-10-06T05:08:14.212 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:08:14.212 DEBUG:teuthology.orchestra.run.smithi174:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2023-10-06T05:08:14.296 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:08:14.296 DEBUG:teuthology.orchestra.run.smithi174:> sudo ip link delete ceph-brx 2023-10-06T05:08:14.391 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:08:14.391 DEBUG:teuthology.orchestra.run.smithi174:> ip netns list 2023-10-06T05:08:14.406 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T05:08:14.406 DEBUG:teuthology.orchestra.run.smithi174:> sudo ip link delete ceph-brx 2023-10-06T05:08:14.474 INFO:teuthology.orchestra.run.smithi174.stderr:Cannot find device "ceph-brx" 2023-10-06T05:08:14.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-06T05:08:14.476 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-10-06T05:08:14.487 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2023-10-06T05:08:14.488 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T05:08:14.604 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.602+0000 7f918b59e700 1 Processor -- start 2023-10-06T05:08:14.604 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.602+0000 7f918b59e700 1 -- start start 2023-10-06T05:08:14.606 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.603+0000 7f918b59e700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f918c140b40 0x7f918c0aec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:14.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.603+0000 7f918b59e700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f918c0af1c0 0x7f918c15adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:14.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.603+0000 7f918b59e700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f918c06d920 con 0x7f918c140770 2023-10-06T05:08:14.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.603+0000 7f918b59e700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f918c0c6500 con 0x7f918c140b40 2023-10-06T05:08:14.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.603+0000 7f918b59e700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f918c0c67e0 con 0x7f918c0af1c0 2023-10-06T05:08:14.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.603+0000 7f9189d9b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f918c140b40 0x7f918c0aec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:14.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.603+0000 7f9189d9b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f918c140b40 0x7f918c0aec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49830/0 (socket says 172.21.15.98:49830) 2023-10-06T05:08:14.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.603+0000 7f918a59c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f918c140770 0x7f918c06d210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:47084/0 (socket says 172.21.15.98:47084) 2023-10-06T05:08:14.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.603+0000 7f9189d9b700 1 -- 172.21.15.98:0/1394047945 learned_addr learned my addr 172.21.15.98:0/1394047945 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:08:14.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.603+0000 7f918ad9d700 1 --2- 172.21.15.98:0/1394047945 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f918c0af1c0 0x7f918c15adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:14.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.603+0000 7f9189d9b700 1 -- 172.21.15.98:0/1394047945 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f918c0af1c0 msgr2=0x7f918c15adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:14.608 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.603+0000 7f9189d9b700 1 --2- 172.21.15.98:0/1394047945 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f918c0af1c0 0x7f918c15adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:14.608 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.603+0000 7f9189d9b700 1 -- 172.21.15.98:0/1394047945 >> v1:172.21.15.98:6789/0 conn(0x7f918c140770 legacy=0x7f918c06d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:14.608 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.603+0000 7f9189d9b700 1 -- 172.21.15.98:0/1394047945 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f918c0c6ac0 con 0x7f918c140b40 2023-10-06T05:08:14.608 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.603+0000 7f918959a700 1 -- 172.21.15.98:0/1394047945 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 391472279 0 0) 0x7f917c001110 con 0x7f918c140770 2023-10-06T05:08:14.608 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.604+0000 7f9189d9b700 1 --2- 172.21.15.98:0/1394047945 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f918c140b40 0x7f918c0aec80 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f91800099b0 tx=0x7f9180009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ae333f6c94630b2 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:14.608 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.604+0000 7f918959a700 1 -- 172.21.15.98:0/1394047945 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f918000b860 con 0x7f918c140b40 2023-10-06T05:08:14.608 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.604+0000 7f918959a700 1 -- 172.21.15.98:0/1394047945 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f918000b9c0 con 0x7f918c140b40 2023-10-06T05:08:14.608 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.604+0000 7f918959a700 1 -- 172.21.15.98:0/1394047945 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f918000bd60 con 0x7f918c140b40 2023-10-06T05:08:14.608 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.604+0000 7f918b59e700 1 -- 172.21.15.98:0/1394047945 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f918c140b40 msgr2=0x7f918c0aec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:14.608 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.604+0000 7f918b59e700 1 --2- 172.21.15.98:0/1394047945 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f918c140b40 0x7f918c0aec80 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f91800099b0 tx=0x7f9180009250 comp rx=0 tx=0).stop 2023-10-06T05:08:14.609 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.604+0000 7f918b59e700 1 -- 172.21.15.98:0/1394047945 shutdown_connections 2023-10-06T05:08:14.609 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.604+0000 7f918b59e700 1 --2- 172.21.15.98:0/1394047945 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f918c0af1c0 0x7f918c15adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:14.609 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.604+0000 7f918b59e700 1 --2- 172.21.15.98:0/1394047945 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f918c140b40 0x7f918c0aec80 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:14.609 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.604+0000 7f918b59e700 1 -- 172.21.15.98:0/1394047945 >> 172.21.15.98:0/1394047945 conn(0x7f918c068080 msgr2=0x7f918c0cbec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:14.609 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.604+0000 7f918b59e700 1 -- 172.21.15.98:0/1394047945 shutdown_connections 2023-10-06T05:08:14.609 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.604+0000 7f918b59e700 1 -- 172.21.15.98:0/1394047945 wait complete. 2023-10-06T05:08:14.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.605+0000 7f918b59e700 1 Processor -- start 2023-10-06T05:08:14.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.605+0000 7f918b59e700 1 -- start start 2023-10-06T05:08:14.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.606+0000 7f918b59e700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f918c0af1c0 0x7f918c13e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:14.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.606+0000 7f918b59e700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f918c140770 0x7f918c0c0c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:14.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.606+0000 7f918b59e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f918c169ae0 0x7f918c0ca3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:14.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.606+0000 7f918b59e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f918c0c21e0 con 0x7f918c169ae0 2023-10-06T05:08:14.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.606+0000 7f918b59e700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f918c0c15d0 con 0x7f918c0af1c0 2023-10-06T05:08:14.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.606+0000 7f918b59e700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f918c0c18b0 con 0x7f918c140770 2023-10-06T05:08:14.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.606+0000 7f918ad9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f918c169ae0 0x7f918c0ca3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:14.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.606+0000 7f918ad9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f918c169ae0 0x7f918c0ca3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49844/0 (socket says 172.21.15.98:49844) 2023-10-06T05:08:14.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.606+0000 7f918ad9d700 1 -- 172.21.15.98:0/489209131 learned_addr learned my addr 172.21.15.98:0/489209131 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:08:14.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.606+0000 7f9189d9b700 1 --2- 172.21.15.98:0/489209131 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f918c140770 0x7f918c0c0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:14.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.606+0000 7f918a59c700 1 --2- 172.21.15.98:0/489209131 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f918c0af1c0 0x7f918c13e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:14.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.607+0000 7f918ad9d700 1 -- 172.21.15.98:0/489209131 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f918c140770 msgr2=0x7f918c0c0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:14.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.607+0000 7f918ad9d700 1 --2- 172.21.15.98:0/489209131 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f918c140770 0x7f918c0c0c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:14.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.607+0000 7f918ad9d700 1 -- 172.21.15.98:0/489209131 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f918c0af1c0 msgr2=0x7f918c13e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:14.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.607+0000 7f918ad9d700 1 --2- 172.21.15.98:0/489209131 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f918c0af1c0 0x7f918c13e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:14.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.607+0000 7f918ad9d700 1 -- 172.21.15.98:0/489209131 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91800094f0 con 0x7f918c169ae0 2023-10-06T05:08:14.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.607+0000 7f918a59c700 1 --2- 172.21.15.98:0/489209131 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f918c0af1c0 0x7f918c13e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:08:14.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.607+0000 7f918ad9d700 1 --2- 172.21.15.98:0/489209131 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f918c169ae0 0x7f918c0ca3d0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f9184009fa0 tx=0x7f9184009900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:14.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.607+0000 7f917a7fc700 1 -- 172.21.15.98:0/489209131 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f918400ec70 con 0x7f918c169ae0 2023-10-06T05:08:14.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.607+0000 7f917a7fc700 1 -- 172.21.15.98:0/489209131 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f918400edd0 con 0x7f918c169ae0 2023-10-06T05:08:14.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.607+0000 7f917a7fc700 1 -- 172.21.15.98:0/489209131 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9184014670 con 0x7f918c169ae0 2023-10-06T05:08:14.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.607+0000 7f918b59e700 1 -- 172.21.15.98:0/489209131 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f918c0c1b90 con 0x7f918c169ae0 2023-10-06T05:08:14.615 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.607+0000 7f918b59e700 1 -- 172.21.15.98:0/489209131 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f918c13e570 con 0x7f918c169ae0 2023-10-06T05:08:14.615 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.608+0000 7f918b59e700 1 -- 172.21.15.98:0/489209131 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f918c05f230 con 0x7f918c169ae0 2023-10-06T05:08:14.615 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.608+0000 7f917a7fc700 1 -- 172.21.15.98:0/489209131 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f91840149e0 con 0x7f918c169ae0 2023-10-06T05:08:14.615 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.608+0000 7f917a7fc700 1 --2- 172.21.15.98:0/489209131 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f916c066fb0 0x7f916c069460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:14.615 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.609+0000 7f917a7fc700 1 -- 172.21.15.98:0/489209131 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5612+0+0 (secure 0 0 0) 0x7f9184084cb0 con 0x7f918c169ae0 2023-10-06T05:08:14.615 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.609+0000 7f918a59c700 1 --2- 172.21.15.98:0/489209131 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f916c066fb0 0x7f916c069460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:14.615 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.609+0000 7f918a59c700 1 --2- 172.21.15.98:0/489209131 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f916c066fb0 0x7f916c069460 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f917c007fd0 tx=0x7f917c007af0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:14.615 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.613+0000 7f917a7fc700 1 -- 172.21.15.98:0/489209131 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9184051ee0 con 0x7f918c169ae0 2023-10-06T05:08:14.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.798+0000 7f918b59e700 1 -- 172.21.15.98:0/489209131 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f918c0c6da0 con 0x7f918c169ae0 2023-10-06T05:08:14.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.799+0000 7f917a7fc700 1 -- 172.21.15.98:0/489209131 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+13984 (secure 0 0 0) 0x7f9184055500 con 0x7f918c169ae0 2023-10-06T05:08:14.802 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:08:14.802 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":46,"fsid":"f50c5df5-cad5-47a4-9bb1-8c791d4311ec","created":"2023-10-06T05:06:32.971175+0000","modified":"2023-10-06T05:08:12.753522+0000","last_up_change":"2023-10-06T05:06:44.016489+0000","last_in_change":"2023-10-06T05:06:35.544511+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-06T05:06:44.559671+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-06T05:06:51.432161+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d9495d95-da75-4c3b-8927-e9bae6a8b093","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":90627},{"type":"v1","addr":"172.21.15.98:6801","nonce":90627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":90627},{"type":"v1","addr":"172.21.15.98:6803","nonce":90627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":90627},{"type":"v1","addr":"172.21.15.98:6807","nonce":90627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":90627},{"type":"v1","addr":"172.21.15.98:6805","nonce":90627}]},"public_addr":"172.21.15.98:6801/90627","cluster_addr":"172.21.15.98:6803/90627","heartbeat_back_addr":"172.21.15.98:6807/90627","heartbeat_front_addr":"172.21.15.98:6805/90627","state":["exists","up"]},{"osd":1,"uuid":"f25ba5be-d65e-4707-a134-154092deb754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":90628},{"type":"v1","addr":"172.21.15.98:6809","nonce":90628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":90628},{"type":"v1","addr":"172.21.15.98:6811","nonce":90628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":90628},{"type":"v1","addr":"172.21.15.98:6815","nonce":90628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":90628},{"type":"v1","addr":"172.21.15.98:6813","nonce":90628}]},"public_addr":"172.21.15.98:6809/90628","cluster_addr":"172.21.15.98:6811/90628","heartbeat_back_addr":"172.21.15.98:6815/90628","heartbeat_front_addr":"172.21.15.98:6813/90628","state":["exists","up"]},{"osd":2,"uuid":"5ab9e028-945c-43e2-8c95-3b0b4cf38e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":90662},{"type":"v1","addr":"172.21.15.98:6818","nonce":90662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":90662},{"type":"v1","addr":"172.21.15.98:6822","nonce":90662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":90662},{"type":"v1","addr":"172.21.15.98:6830","nonce":90662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":90662},{"type":"v1","addr":"172.21.15.98:6826","nonce":90662}]},"public_addr":"172.21.15.98:6818/90662","cluster_addr":"172.21.15.98:6822/90662","heartbeat_back_addr":"172.21.15.98:6830/90662","heartbeat_front_addr":"172.21.15.98:6826/90662","state":["exists","up"]},{"osd":3,"uuid":"3bb803ff-1738-46bf-be62-2267bf378822","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6817","nonce":90663},{"type":"v1","addr":"172.21.15.98:6819","nonce":90663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6821","nonce":90663},{"type":"v1","addr":"172.21.15.98:6823","nonce":90663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6829","nonce":90663},{"type":"v1","addr":"172.21.15.98:6831","nonce":90663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6825","nonce":90663},{"type":"v1","addr":"172.21.15.98:6827","nonce":90663}]},"public_addr":"172.21.15.98:6819/90663","cluster_addr":"172.21.15.98:6823/90663","heartbeat_back_addr":"172.21.15.98:6831/90663","heartbeat_front_addr":"172.21.15.98:6827/90663","state":["exists","up"]},{"osd":4,"uuid":"7d57a4c9-6462-43f6-b64a-7f9e9b4353d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":90847},{"type":"v1","addr":"172.21.15.174:6801","nonce":90847}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":90847},{"type":"v1","addr":"172.21.15.174:6803","nonce":90847}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":90847},{"type":"v1","addr":"172.21.15.174:6807","nonce":90847}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":90847},{"type":"v1","addr":"172.21.15.174:6805","nonce":90847}]},"public_addr":"172.21.15.174:6801/90847","cluster_addr":"172.21.15.174:6803/90847","heartbeat_back_addr":"172.21.15.174:6807/90847","heartbeat_front_addr":"172.21.15.174:6805/90847","state":["exists","up"]},{"osd":5,"uuid":"83287ea4-e44d-4af9-bc77-36d47c60d4a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":90848},{"type":"v1","addr":"172.21.15.174:6809","nonce":90848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":90848},{"type":"v1","addr":"172.21.15.174:6811","nonce":90848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":90848},{"type":"v1","addr":"172.21.15.174:6815","nonce":90848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":90848},{"type":"v1","addr":"172.21.15.174:6813","nonce":90848}]},"public_addr":"172.21.15.174:6809/90848","cluster_addr":"172.21.15.174:6811/90848","heartbeat_back_addr":"172.21.15.174:6815/90848","heartbeat_front_addr":"172.21.15.174:6813/90848","state":["exists","up"]},{"osd":6,"uuid":"496a6272-892e-43c4-a145-206623c22f99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":90877},{"type":"v1","addr":"172.21.15.174:6817","nonce":90877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":90877},{"type":"v1","addr":"172.21.15.174:6819","nonce":90877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":90877},{"type":"v1","addr":"172.21.15.174:6823","nonce":90877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":90877},{"type":"v1","addr":"172.21.15.174:6821","nonce":90877}]},"public_addr":"172.21.15.174:6817/90877","cluster_addr":"172.21.15.174:6819/90877","heartbeat_back_addr":"172.21.15.174:6823/90877","heartbeat_front_addr":"172.21.15.174:6821/90877","state":["exists","up"]},{"osd":7,"uuid":"49967e34-8b07-4bab-a13e-0c16d6a0f3a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":90878},{"type":"v1","addr":"172.21.15.174:6825","nonce":90878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":90878},{"type":"v1","addr":"172.21.15.174:6827","nonce":90878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":90878},{"type":"v1","addr":"172.21.15.174:6831","nonce":90878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":90878},{"type":"v1","addr":"172.21.15.174:6829","nonce":90878}]},"public_addr":"172.21.15.174:6825/90878","cluster_addr":"172.21.15.174:6827/90878","heartbeat_back_addr":"172.21.15.174:6831/90878","heartbeat_front_addr":"172.21.15.174:6829/90878","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.446479+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.514266+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.794108+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.810894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.731919+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.717756+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.737435+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.508682+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6837/1617154626":"2023-10-07T05:08:09.607156+0000","172.21.15.98:6835/1617154626":"2023-10-07T05:08:09.607156+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T05:08:14.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.801+0000 7f918b59e700 1 -- 172.21.15.98:0/489209131 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f916c066fb0 msgr2=0x7f916c069460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:14.803 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.801+0000 7f918b59e700 1 --2- 172.21.15.98:0/489209131 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f916c066fb0 0x7f916c069460 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f917c007fd0 tx=0x7f917c007af0 comp rx=0 tx=0).stop 2023-10-06T05:08:14.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.801+0000 7f918b59e700 1 -- 172.21.15.98:0/489209131 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f918c169ae0 msgr2=0x7f918c0ca3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:14.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.801+0000 7f918b59e700 1 --2- 172.21.15.98:0/489209131 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f918c169ae0 0x7f918c0ca3d0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f9184009fa0 tx=0x7f9184009900 comp rx=0 tx=0).stop 2023-10-06T05:08:14.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.801+0000 7f918b59e700 1 -- 172.21.15.98:0/489209131 shutdown_connections 2023-10-06T05:08:14.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.801+0000 7f918b59e700 1 --2- 172.21.15.98:0/489209131 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f916c066fb0 0x7f916c069460 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:14.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.801+0000 7f918b59e700 1 --2- 172.21.15.98:0/489209131 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f918c169ae0 0x7f918c0ca3d0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:14.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.801+0000 7f918b59e700 1 --2- 172.21.15.98:0/489209131 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f918c0af1c0 0x7f918c13e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:14.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.801+0000 7f918b59e700 1 --2- 172.21.15.98:0/489209131 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f918c140770 0x7f918c0c0c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:14.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.801+0000 7f918b59e700 1 -- 172.21.15.98:0/489209131 >> 172.21.15.98:0/489209131 conn(0x7f918c068080 msgr2=0x7f918c13f5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:14.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.801+0000 7f918b59e700 1 -- 172.21.15.98:0/489209131 shutdown_connections 2023-10-06T05:08:14.804 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.801+0000 7f918b59e700 1 -- 172.21.15.98:0/489209131 wait complete. 2023-10-06T05:08:14.815 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-06T05:08:14.815 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-06T05:08:14.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.927+0000 7f781bfff700 1 Processor -- start 2023-10-06T05:08:14.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.927+0000 7f781bfff700 1 -- start start 2023-10-06T05:08:14.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.928+0000 7f781bfff700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f781c159440 0x7f781c0b3090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:14.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.928+0000 7f781bfff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f781c0b2100 0x7f781c0b2520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:14.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.928+0000 7f781bfff700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f781c0cb7a0 con 0x7f781c0b35d0 2023-10-06T05:08:14.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.928+0000 7f781bfff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f781c0b0860 con 0x7f781c0b2100 2023-10-06T05:08:14.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.928+0000 7f781bfff700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f781c0cb150 con 0x7f781c159440 2023-10-06T05:08:14.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.928+0000 7f781b7fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f781c0b2100 0x7f781c0b2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:14.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.928+0000 7f781b7fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f781c0b2100 0x7f781c0b2520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49848/0 (socket says 172.21.15.98:49848) 2023-10-06T05:08:14.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.928+0000 7f781b7fe700 1 -- 172.21.15.98:0/494002804 learned_addr learned my addr 172.21.15.98:0/494002804 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:08:14.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.928+0000 7f781affd700 1 --2- 172.21.15.98:0/494002804 >> v2:172.21.15.174:3300/0 conn(0x7f781c159440 0x7f781c0b3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:14.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.928+0000 7f781b7fe700 1 -- 172.21.15.98:0/494002804 >> v2:172.21.15.174:3300/0 conn(0x7f781c159440 msgr2=0x7f781c0b3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:14.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.928+0000 7f781b7fe700 1 --2- 172.21.15.98:0/494002804 >> v2:172.21.15.174:3300/0 conn(0x7f781c159440 0x7f781c0b3090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:14.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.928+0000 7f781b7fe700 1 -- 172.21.15.98:0/494002804 >> v1:172.21.15.174:6789/0 conn(0x7f781c0b35d0 legacy=0x7f781c0cbc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:14.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.928+0000 7f781b7fe700 1 -- 172.21.15.98:0/494002804 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f781c0cb430 con 0x7f781c0b2100 2023-10-06T05:08:14.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.929+0000 7f781b7fe700 1 --2- 172.21.15.98:0/494002804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f781c0b2100 0x7f781c0b2520 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f781c06f9f0 tx=0x7f7814009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ae725ee8f3b24b server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:14.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.929+0000 7f781a7fc700 1 -- 172.21.15.98:0/494002804 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f781400e070 con 0x7f781c0b2100 2023-10-06T05:08:14.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.929+0000 7f781a7fc700 1 -- 172.21.15.98:0/494002804 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f781400ba10 con 0x7f781c0b2100 2023-10-06T05:08:14.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.929+0000 7f781a7fc700 1 -- 172.21.15.98:0/494002804 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78140092e0 con 0x7f781c0b2100 2023-10-06T05:08:14.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.929+0000 7f781bfff700 1 -- 172.21.15.98:0/494002804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f781c0b2100 msgr2=0x7f781c0b2520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:14.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.929+0000 7f781bfff700 1 --2- 172.21.15.98:0/494002804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f781c0b2100 0x7f781c0b2520 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f781c06f9f0 tx=0x7f7814009840 comp rx=0 tx=0).stop 2023-10-06T05:08:14.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.929+0000 7f781bfff700 1 -- 172.21.15.98:0/494002804 shutdown_connections 2023-10-06T05:08:14.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.929+0000 7f781bfff700 1 --2- 172.21.15.98:0/494002804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f781c0b2100 0x7f781c0b2520 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:14.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.929+0000 7f781bfff700 1 --2- 172.21.15.98:0/494002804 >> v2:172.21.15.174:3300/0 conn(0x7f781c159440 0x7f781c0b3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:14.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.929+0000 7f781bfff700 1 -- 172.21.15.98:0/494002804 >> 172.21.15.98:0/494002804 conn(0x7f781c06ece0 msgr2=0x7f781c06f0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:14.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.929+0000 7f781bfff700 1 -- 172.21.15.98:0/494002804 shutdown_connections 2023-10-06T05:08:14.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.929+0000 7f781bfff700 1 -- 172.21.15.98:0/494002804 wait complete. 2023-10-06T05:08:14.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.930+0000 7f781bfff700 1 Processor -- start 2023-10-06T05:08:14.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.930+0000 7f781bfff700 1 -- start start 2023-10-06T05:08:14.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.931+0000 7f781bfff700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f781c0b2100 0x7f781c1475a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:14.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.931+0000 7f781bfff700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f781c0b35d0 0x7f781c1699d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:14.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.931+0000 7f781bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f781c159440 0x7f781c0c3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:14.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.931+0000 7f781bfff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f781c0c77c0 con 0x7f781c159440 2023-10-06T05:08:14.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.931+0000 7f781bfff700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f781c16a450 con 0x7f781c0b2100 2023-10-06T05:08:14.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.931+0000 7f781bfff700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f781c16a730 con 0x7f781c0b35d0 2023-10-06T05:08:14.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.931+0000 7f781b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f781c159440 0x7f781c0c3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:14.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.931+0000 7f781b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f781c159440 0x7f781c0c3ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49860/0 (socket says 172.21.15.98:49860) 2023-10-06T05:08:14.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.931+0000 7f781b7fe700 1 -- 172.21.15.98:0/3675375317 learned_addr learned my addr 172.21.15.98:0/3675375317 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:08:14.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.931+0000 7f781affd700 1 --2- 172.21.15.98:0/3675375317 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f781c0b2100 0x7f781c1475a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:14.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.931+0000 7f78127fc700 1 --2- 172.21.15.98:0/3675375317 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f781c0b35d0 0x7f781c1699d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:14.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.932+0000 7f781b7fe700 1 -- 172.21.15.98:0/3675375317 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f781c0b35d0 msgr2=0x7f781c1699d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:14.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.932+0000 7f781b7fe700 1 --2- 172.21.15.98:0/3675375317 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f781c0b35d0 0x7f781c1699d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:14.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.932+0000 7f781b7fe700 1 -- 172.21.15.98:0/3675375317 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f781c0b2100 msgr2=0x7f781c1475a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:14.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.932+0000 7f781b7fe700 1 --2- 172.21.15.98:0/3675375317 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f781c0b2100 0x7f781c1475a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:14.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.932+0000 7f781b7fe700 1 -- 172.21.15.98:0/3675375317 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7814009ae0 con 0x7f781c159440 2023-10-06T05:08:14.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.932+0000 7f781b7fe700 1 --2- 172.21.15.98:0/3675375317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f781c159440 0x7f781c0c3ac0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f781c069150 tx=0x7f7814017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:14.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.932+0000 7f78137fe700 1 -- 172.21.15.98:0/3675375317 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7814012e60 con 0x7f781c159440 2023-10-06T05:08:14.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.932+0000 7f78137fe700 1 -- 172.21.15.98:0/3675375317 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78140092e0 con 0x7f781c159440 2023-10-06T05:08:14.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.932+0000 7f78137fe700 1 -- 172.21.15.98:0/3675375317 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f781401d650 con 0x7f781c159440 2023-10-06T05:08:14.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.932+0000 7f781bfff700 1 -- 172.21.15.98:0/3675375317 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f781c0c7450 con 0x7f781c159440 2023-10-06T05:08:14.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.932+0000 7f781bfff700 1 -- 172.21.15.98:0/3675375317 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f781c154900 con 0x7f781c159440 2023-10-06T05:08:14.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.933+0000 7f781bfff700 1 -- 172.21.15.98:0/3675375317 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f781c0b5d10 con 0x7f781c159440 2023-10-06T05:08:14.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.933+0000 7f78137fe700 1 -- 172.21.15.98:0/3675375317 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f781401da00 con 0x7f781c159440 2023-10-06T05:08:14.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.933+0000 7f78137fe700 1 --2- 172.21.15.98:0/3675375317 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f77fc066f30 0x7f77fc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:14.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.934+0000 7f78137fe700 1 -- 172.21.15.98:0/3675375317 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5612+0+0 (secure 0 0 0) 0x7f7814085e20 con 0x7f781c159440 2023-10-06T05:08:14.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.934+0000 7f781affd700 1 --2- 172.21.15.98:0/3675375317 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f77fc066f30 0x7f77fc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:14.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.934+0000 7f781affd700 1 --2- 172.21.15.98:0/3675375317 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f77fc066f30 0x7f77fc0693e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f780c006fd0 tx=0x7f780c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:14.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:14.938+0000 7f78137fe700 1 -- 172.21.15.98:0/3675375317 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f78140530f0 con 0x7f781c159440 2023-10-06T05:08:15.117 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.115+0000 7f781bfff700 1 -- 172.21.15.98:0/3675375317 --> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f781c154b20 con 0x7f77fc066f30 2023-10-06T05:08:15.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.117+0000 7f78137fe700 1 -- 172.21.15.98:0/3675375317 <== mgr.4104 v2:172.21.15.98:6832/90397 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+72767 (secure 0 0 0) 0x7f780c00f660 con 0x7f77fc066f30 2023-10-06T05:08:15.120 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:08:15.122 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.119+0000 7f781bfff700 1 -- 172.21.15.98:0/3675375317 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f77fc066f30 msgr2=0x7f77fc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:15.122 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.119+0000 7f781bfff700 1 --2- 172.21.15.98:0/3675375317 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f77fc066f30 0x7f77fc0693e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f780c006fd0 tx=0x7f780c008040 comp rx=0 tx=0).stop 2023-10-06T05:08:15.122 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.119+0000 7f781bfff700 1 -- 172.21.15.98:0/3675375317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f781c159440 msgr2=0x7f781c0c3ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:15.122 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.119+0000 7f781bfff700 1 --2- 172.21.15.98:0/3675375317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f781c159440 0x7f781c0c3ac0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f781c069150 tx=0x7f7814017000 comp rx=0 tx=0).stop 2023-10-06T05:08:15.122 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.119+0000 7f781bfff700 1 -- 172.21.15.98:0/3675375317 shutdown_connections 2023-10-06T05:08:15.122 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.120+0000 7f781bfff700 1 --2- 172.21.15.98:0/3675375317 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f77fc066f30 0x7f77fc0693e0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:15.123 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.120+0000 7f781bfff700 1 --2- 172.21.15.98:0/3675375317 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f781c159440 0x7f781c0c3ac0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:15.123 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.120+0000 7f781bfff700 1 --2- 172.21.15.98:0/3675375317 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f781c0b2100 0x7f781c1475a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:15.123 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.120+0000 7f781bfff700 1 --2- 172.21.15.98:0/3675375317 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f781c0b35d0 0x7f781c1699d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:15.123 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.120+0000 7f781bfff700 1 -- 172.21.15.98:0/3675375317 >> 172.21.15.98:0/3675375317 conn(0x7f781c06ece0 msgr2=0x7f781c1586f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:15.123 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.120+0000 7f781bfff700 1 -- 172.21.15.98:0/3675375317 shutdown_connections 2023-10-06T05:08:15.123 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.120+0000 7f781bfff700 1 -- 172.21.15.98:0/3675375317 wait complete. 2023-10-06T05:08:15.123 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2023-10-06T05:08:15.133 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":85,"stamp":"2023-10-06T05:08:14.535531+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":193,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2449024,"kb_used_data":70720,"kb_used_omap":0,"kb_used_meta":2378240,"kb_avail":752525696,"statfs":{"total":773094113280,"available":770586312704,"internally_reserved":0,"allocated":72417280,"data_stored":70121580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2435317760},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-33572612,"num_objects":-30,"num_object_clones":0,"num_object_copies":-60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-30,"num_whiteouts":0,"num_read":-1,"num_read_kb":-4096,"num_write":-65,"num_write_kb":-32802,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001163"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":31,"reported_epoch":45,"state":"active+clean","last_fresh":"2023-10-06T05:08:11.918743+0000","last_change":"2023-10-06T05:06:52.627232+0000","last_active":"2023-10-06T05:08:11.918743+0000","last_peered":"2023-10-06T05:08:11.918743+0000","last_clean":"2023-10-06T05:08:11.918743+0000","last_became_active":"2023-10-06T05:06:52.626936+0000","last_became_peered":"2023-10-06T05:06:52.626936+0000","last_unstale":"2023-10-06T05:08:11.918743+0000","last_undegraded":"2023-10-06T05:08:11.918743+0000","last_fullsized":"2023-10-06T05:08:11.918743+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:20:13.925170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":32,"reported_epoch":43,"state":"active+clean","last_fresh":"2023-10-06T05:07:57.487459+0000","last_change":"2023-10-06T05:06:52.587795+0000","last_active":"2023-10-06T05:07:57.487459+0000","last_peered":"2023-10-06T05:07:57.487459+0000","last_clean":"2023-10-06T05:07:57.487459+0000","last_became_active":"2023-10-06T05:06:52.587407+0000","last_became_peered":"2023-10-06T05:06:52.587407+0000","last_unstale":"2023-10-06T05:07:57.487459+0000","last_undegraded":"2023-10-06T05:07:57.487459+0000","last_fullsized":"2023-10-06T05:07:57.487459+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:36:31.526532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":28,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.499543+0000","last_change":"2023-10-06T05:06:52.675238+0000","last_active":"2023-10-06T05:07:56.499543+0000","last_peered":"2023-10-06T05:07:56.499543+0000","last_clean":"2023-10-06T05:07:56.499543+0000","last_became_active":"2023-10-06T05:06:52.674952+0000","last_became_peered":"2023-10-06T05:06:52.674952+0000","last_unstale":"2023-10-06T05:07:56.499543+0000","last_undegraded":"2023-10-06T05:07:56.499543+0000","last_fullsized":"2023-10-06T05:07:56.499543+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:27:51.141184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":32,"reported_epoch":43,"state":"active+clean","last_fresh":"2023-10-06T05:07:57.487626+0000","last_change":"2023-10-06T05:06:52.587778+0000","last_active":"2023-10-06T05:07:57.487626+0000","last_peered":"2023-10-06T05:07:57.487626+0000","last_clean":"2023-10-06T05:07:57.487626+0000","last_became_active":"2023-10-06T05:06:52.587388+0000","last_became_peered":"2023-10-06T05:06:52.587388+0000","last_unstale":"2023-10-06T05:07:57.487626+0000","last_undegraded":"2023-10-06T05:07:57.487626+0000","last_fullsized":"2023-10-06T05:07:57.487626+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:45:52.502298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":184,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-10-06T05:08:09.611541+0000","last_change":"2023-10-06T05:06:46.307169+0000","last_active":"2023-10-06T05:08:09.611541+0000","last_peered":"2023-10-06T05:08:09.611541+0000","last_clean":"2023-10-06T05:08:09.611541+0000","last_became_active":"2023-10-06T05:06:46.306107+0000","last_became_peered":"2023-10-06T05:06:46.306107+0000","last_unstale":"2023-10-06T05:08:09.611541+0000","last_undegraded":"2023-10-06T05:08:09.611541+0000","last_fullsized":"2023-10-06T05:08:09.611541+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:45.018006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:45.018006+0000","last_clean_scrub_stamp":"2023-10-06T05:06:45.018006+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:38:17.064131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":31,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-10-06T05:08:09.611654+0000","last_change":"2023-10-06T05:06:52.588475+0000","last_active":"2023-10-06T05:08:09.611654+0000","last_peered":"2023-10-06T05:08:09.611654+0000","last_clean":"2023-10-06T05:08:09.611654+0000","last_became_active":"2023-10-06T05:06:52.588134+0000","last_became_peered":"2023-10-06T05:06:52.588134+0000","last_unstale":"2023-10-06T05:08:09.611654+0000","last_undegraded":"2023-10-06T05:08:09.611654+0000","last_fullsized":"2023-10-06T05:08:09.611654+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:35:05.269778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":31,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-10-06T05:08:09.611956+0000","last_change":"2023-10-06T05:06:52.627289+0000","last_active":"2023-10-06T05:08:09.611956+0000","last_peered":"2023-10-06T05:08:09.611956+0000","last_clean":"2023-10-06T05:08:09.611956+0000","last_became_active":"2023-10-06T05:06:52.626937+0000","last_became_peered":"2023-10-06T05:06:52.626937+0000","last_unstale":"2023-10-06T05:08:09.611956+0000","last_undegraded":"2023-10-06T05:08:09.611956+0000","last_fullsized":"2023-10-06T05:08:09.611956+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:46:59.070674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":31,"reported_epoch":45,"state":"active+clean","last_fresh":"2023-10-06T05:08:11.920128+0000","last_change":"2023-10-06T05:06:52.588853+0000","last_active":"2023-10-06T05:08:11.920128+0000","last_peered":"2023-10-06T05:08:11.920128+0000","last_clean":"2023-10-06T05:08:11.920128+0000","last_became_active":"2023-10-06T05:06:52.588561+0000","last_became_peered":"2023-10-06T05:06:52.588561+0000","last_unstale":"2023-10-06T05:08:11.920128+0000","last_undegraded":"2023-10-06T05:08:11.920128+0000","last_fullsized":"2023-10-06T05:08:11.920128+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:06:28.729493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.491222+0000","last_change":"2023-10-06T05:06:52.589519+0000","last_active":"2023-10-06T05:07:56.491222+0000","last_peered":"2023-10-06T05:07:56.491222+0000","last_clean":"2023-10-06T05:07:56.491222+0000","last_became_active":"2023-10-06T05:06:52.589246+0000","last_became_peered":"2023-10-06T05:06:52.589246+0000","last_unstale":"2023-10-06T05:07:56.491222+0000","last_undegraded":"2023-10-06T05:07:56.491222+0000","last_fullsized":"2023-10-06T05:07:56.491222+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:54:23.352764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509461,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310064,"kb_used_data":12776,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94061776,"statfs":{"total":96636764160,"available":96319258624,"internally_reserved":0,"allocated":13082624,"data_stored":12802192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.34899999999999998}]},{"osd":1,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.40300000000000002}]},{"osd":2,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.53500000000000003}]},{"osd":3,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.52200000000000002}]},{"osd":4,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.58099999999999996}]},{"osd":5,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.58999999999999997}]},{"osd":7,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.56999999999999995}]}]},{"osd":5,"up_from":15,"seq":64424509461,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297772,"kb_used_data":484,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074068,"statfs":{"total":96636764160,"available":96331845632,"internally_reserved":0,"allocated":495616,"data_stored":218706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.36699999999999999}]},{"osd":1,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.39600000000000002}]},{"osd":2,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.504}]},{"osd":3,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.44700000000000001}]},{"osd":4,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.42999999999999999}]},{"osd":6,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.54400000000000004}]},{"osd":7,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.53100000000000003}]}]},{"osd":0,"up_from":14,"seq":60129542166,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":318856,"kb_used_data":21568,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94052984,"statfs":{"total":96636764160,"available":96310255616,"internally_reserved":0,"allocated":22085632,"data_stored":21793930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":1.2809999999999999,"5min":1.2809999999999999,"15min":1.2809999999999999},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.28299999999999997}]},{"osd":2,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":4.1529999999999996,"5min":4.1529999999999996,"15min":4.1529999999999996},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.372,"5min":0.372,"15min":0.372},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":1.105,"5min":1.105,"15min":1.105},"last":0.32100000000000001}]},{"osd":3,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":1.327,"5min":1.327,"15min":1.327},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":4.1900000000000004,"5min":4.1900000000000004,"15min":4.1900000000000004},"last":0.40799999999999997}]},{"osd":4,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.46800000000000003}]},{"osd":5,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.304,"5min":1.304,"15min":1.304},"last":0.432}]},{"osd":6,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.438},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.48099999999999998}]},{"osd":7,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":1.079,"5min":1.079,"15min":1.079},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.45100000000000001}]}]},{"osd":7,"up_from":13,"seq":55834574869,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306568,"kb_used_data":9280,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94065272,"statfs":{"total":96636764160,"available":96322838528,"internally_reserved":0,"allocated":9502720,"data_stored":9211018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.5},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.42699999999999999}]},{"osd":1,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.39600000000000002}]},{"osd":2,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.59199999999999997}]},{"osd":3,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.621},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.57099999999999995}]},{"osd":4,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.65700000000000003}]},{"osd":5,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.60999999999999999}]},{"osd":6,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.63}]}]},{"osd":1,"up_from":14,"seq":60129542165,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297812,"kb_used_data":524,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074028,"statfs":{"total":96636764160,"available":96331804672,"internally_reserved":0,"allocated":536576,"data_stored":238268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":1.8879999999999999,"5min":1.8879999999999999,"15min":1.8879999999999999},"last":0.19800000000000001},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":1.595,"5min":1.595,"15min":1.595},"last":0.21299999999999999}]},{"osd":2,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.123,"5min":1.123,"15min":1.123},"last":0.31}]},{"osd":3,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.179,"5min":1.179,"15min":1.179},"last":0.22900000000000001},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1.657,"5min":1.657,"15min":1.657},"last":0.29699999999999999}]},{"osd":4,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":1.6319999999999999,"5min":1.6319999999999999,"15min":1.6319999999999999},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.837,"5min":1.837,"15min":1.837},"last":0.45000000000000001}]},{"osd":5,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.8100000000000001,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.42199999999999999}]},{"osd":6,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.6850000000000001,"5min":1.6850000000000001,"15min":1.6850000000000001},"last":0.44},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.8620000000000001,"5min":1.8620000000000001,"15min":1.8620000000000001},"last":0.52000000000000002}]},{"osd":7,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.7789999999999999,"5min":1.7789999999999999,"15min":1.7789999999999999},"last":0.502},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.46100000000000002}]}]},{"osd":2,"up_from":15,"seq":64424509460,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305992,"kb_used_data":8704,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94065848,"statfs":{"total":96636764160,"available":96323428352,"internally_reserved":0,"allocated":8912896,"data_stored":8624817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.311,"5min":0.311,"15min":0.311},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.39500000000000002}]},{"osd":1,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"last":0.307},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"last":0.33000000000000002}]},{"osd":3,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"last":0.36899999999999999}]},{"osd":4,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.73499999999999999}]},{"osd":5,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.68200000000000005}]},{"osd":6,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.67400000000000004}]},{"osd":7,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.54800000000000004}]}]},{"osd":3,"up_from":15,"seq":64424509460,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310068,"kb_used_data":12780,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94061772,"statfs":{"total":96636764160,"available":96319254528,"internally_reserved":0,"allocated":13086720,"data_stored":12802158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"last":0.248},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":3.391,"5min":3.391,"15min":3.391},"last":0.27700000000000002}]},{"osd":1,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":2.8740000000000001,"5min":2.8740000000000001,"15min":2.8740000000000001},"last":0.22700000000000001},{"interface":"front","average":{"1min":0.376,"5min":0.376,"15min":0.376},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.33400000000000002}]},{"osd":2,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":7.2210000000000001,"5min":7.2210000000000001,"15min":7.2210000000000001},"last":0.38400000000000001}]},{"osd":4,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.018,"5min":1.018,"15min":1.018},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.6930000000000001,"5min":1.6930000000000001,"15min":1.6930000000000001},"last":0.45900000000000002}]},{"osd":5,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.607,"5min":1.607,"15min":1.607},"last":0.432},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.51600000000000001}]},{"osd":6,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.657,"5min":1.657,"15min":1.657},"last":0.39800000000000002}]},{"osd":7,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.877,"5min":1.877,"15min":1.877},"last":0.502}]}]},{"osd":4,"up_from":15,"seq":64424509461,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301892,"kb_used_data":4604,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94069948,"statfs":{"total":96636764160,"available":96327626752,"internally_reserved":0,"allocated":4714496,"data_stored":4430491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.45600000000000002}]},{"osd":1,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.443},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.46600000000000003}]},{"osd":2,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.41899999999999998}]},{"osd":3,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":11.385999999999999,"5min":11.385999999999999,"15min":11.385999999999999},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.52900000000000003}]},{"osd":5,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.5}]},{"osd":6,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.374},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.55000000000000004}]},{"osd":7,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.56000000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-06T05:08:15.135 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-06T05:08:15.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.247+0000 7fedde536700 1 Processor -- start 2023-10-06T05:08:15.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.247+0000 7fedde536700 1 -- start start 2023-10-06T05:08:15.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.248+0000 7fedde536700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fedd8157600 0x7fedd806e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:15.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.248+0000 7fedde536700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedd806bcc0 con 0x7fedd8067cc0 2023-10-06T05:08:15.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.248+0000 7fedde536700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedd80ca6f0 con 0x7fedd80ca940 2023-10-06T05:08:15.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.248+0000 7fedde536700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedd806ecd0 con 0x7fedd8157600 2023-10-06T05:08:15.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.248+0000 7fedd7fff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fedd8157600 0x7fedd806e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:15.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.248+0000 7feddcad3700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fedd8067cc0 0x7fedd806c150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:47086/0 (socket says 172.21.15.98:47086) 2023-10-06T05:08:15.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.248+0000 7feddcad3700 1 -- 172.21.15.98:0/4264324948 learned_addr learned my addr 172.21.15.98:0/4264324948 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:08:15.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.249+0000 7fedd7fff700 1 -- 172.21.15.98:0/4264324948 >> v1:172.21.15.174:6789/0 conn(0x7fedd80ca940 legacy=0x7fedd80675b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:15.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.249+0000 7fedd7fff700 1 -- 172.21.15.98:0/4264324948 >> v1:172.21.15.98:6789/0 conn(0x7fedd8067cc0 legacy=0x7fedd806c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:15.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.249+0000 7fedd6ffd700 1 -- 172.21.15.98:0/4264324948 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2834742014 0 0) 0x7fedd806bcc0 con 0x7fedd8067cc0 2023-10-06T05:08:15.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.249+0000 7fedd7fff700 1 -- 172.21.15.98:0/4264324948 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedd806efb0 con 0x7fedd8157600 2023-10-06T05:08:15.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.249+0000 7fedd7fff700 1 --2- 172.21.15.98:0/4264324948 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fedd8157600 0x7fedd806e680 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fedc80098e0 tx=0x7fedc80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3863a8614bf94dd3 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:15.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.249+0000 7fedd6ffd700 1 -- 172.21.15.98:0/4264324948 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedc8009e80 con 0x7fedd8157600 2023-10-06T05:08:15.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.249+0000 7fedd6ffd700 1 -- 172.21.15.98:0/4264324948 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedc800b860 con 0x7fedd8157600 2023-10-06T05:08:15.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.249+0000 7fedd6ffd700 1 -- 172.21.15.98:0/4264324948 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedc800bbe0 con 0x7fedd8157600 2023-10-06T05:08:15.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.249+0000 7fedde536700 1 -- 172.21.15.98:0/4264324948 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fedd8157600 msgr2=0x7fedd806e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:15.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.249+0000 7fedde536700 1 --2- 172.21.15.98:0/4264324948 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fedd8157600 0x7fedd806e680 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fedc80098e0 tx=0x7fedc80091a0 comp rx=0 tx=0).stop 2023-10-06T05:08:15.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.250+0000 7fedde536700 1 -- 172.21.15.98:0/4264324948 shutdown_connections 2023-10-06T05:08:15.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.250+0000 7fedde536700 1 --2- 172.21.15.98:0/4264324948 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fedd8157600 0x7fedd806e680 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:15.254 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.250+0000 7fedde536700 1 -- 172.21.15.98:0/4264324948 >> 172.21.15.98:0/4264324948 conn(0x7fedd80cdeb0 msgr2=0x7fedd80c9e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:15.254 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.250+0000 7fedde536700 1 -- 172.21.15.98:0/4264324948 shutdown_connections 2023-10-06T05:08:15.254 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.250+0000 7fedde536700 1 -- 172.21.15.98:0/4264324948 wait complete. 2023-10-06T05:08:15.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.251+0000 7fedde536700 1 Processor -- start 2023-10-06T05:08:15.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.251+0000 7fedde536700 1 -- start start 2023-10-06T05:08:15.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.251+0000 7fedde536700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fedd8067cc0 0x7fedd80ac9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:15.256 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.252+0000 7fedde536700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fedd80ca940 0x7fedd8144c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:15.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.252+0000 7fedde536700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fedd8157600 0x7fedd815b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:15.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.252+0000 7fedde536700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedd8142950 con 0x7fedd8067cc0 2023-10-06T05:08:15.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.252+0000 7fedde536700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedd80acfa0 con 0x7fedd80ca940 2023-10-06T05:08:15.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.252+0000 7fedde536700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedd80ad280 con 0x7fedd8157600 2023-10-06T05:08:15.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.252+0000 7fedd7fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fedd8067cc0 0x7fedd80ac9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:15.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.252+0000 7fedd7fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fedd8067cc0 0x7fedd80ac9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49880/0 (socket says 172.21.15.98:49880) 2023-10-06T05:08:15.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.252+0000 7fedd7fff700 1 -- 172.21.15.98:0/2696030485 learned_addr learned my addr 172.21.15.98:0/2696030485 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:08:15.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.252+0000 7fedd77fe700 1 --2- 172.21.15.98:0/2696030485 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fedd80ca940 0x7fedd8144c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:15.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.252+0000 7feddcad3700 1 --2- 172.21.15.98:0/2696030485 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fedd8157600 0x7fedd815b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:15.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.252+0000 7fedd7fff700 1 -- 172.21.15.98:0/2696030485 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fedd8157600 msgr2=0x7fedd815b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:15.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.252+0000 7fedd7fff700 1 --2- 172.21.15.98:0/2696030485 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fedd8157600 0x7fedd815b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:15.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.252+0000 7fedd7fff700 1 -- 172.21.15.98:0/2696030485 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fedd80ca940 msgr2=0x7fedd8144c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:15.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.252+0000 7fedd7fff700 1 --2- 172.21.15.98:0/2696030485 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fedd80ca940 0x7fedd8144c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:15.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.252+0000 7fedd7fff700 1 -- 172.21.15.98:0/2696030485 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedc8009420 con 0x7fedd8067cc0 2023-10-06T05:08:15.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.252+0000 7feddcad3700 1 --2- 172.21.15.98:0/2696030485 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fedd8157600 0x7fedd815b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:08:15.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.252+0000 7fedd77fe700 1 --2- 172.21.15.98:0/2696030485 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fedd80ca940 0x7fedd8144c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:08:15.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.252+0000 7fedd7fff700 1 --2- 172.21.15.98:0/2696030485 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fedd8067cc0 0x7fedd80ac9c0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fedd806dce0 tx=0x7fedc8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:15.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.252+0000 7fedbffff700 1 -- 172.21.15.98:0/2696030485 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedc801f070 con 0x7fedd8067cc0 2023-10-06T05:08:15.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.252+0000 7fedbffff700 1 -- 172.21.15.98:0/2696030485 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedc8011420 con 0x7fedd8067cc0 2023-10-06T05:08:15.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.252+0000 7fedde536700 1 -- 172.21.15.98:0/2696030485 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedd80ad560 con 0x7fedd8067cc0 2023-10-06T05:08:15.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.252+0000 7fedbffff700 1 -- 172.21.15.98:0/2696030485 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedc800bbe0 con 0x7fedd8067cc0 2023-10-06T05:08:15.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.252+0000 7fedde536700 1 -- 172.21.15.98:0/2696030485 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedd80af9f0 con 0x7fedd8067cc0 2023-10-06T05:08:15.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.253+0000 7fedde536700 1 -- 172.21.15.98:0/2696030485 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedd8142d70 con 0x7fedd8067cc0 2023-10-06T05:08:15.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.254+0000 7fedbffff700 1 -- 172.21.15.98:0/2696030485 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fedc8011a50 con 0x7fedd8067cc0 2023-10-06T05:08:15.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.254+0000 7fedbffff700 1 --2- 172.21.15.98:0/2696030485 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fedc4066f30 0x7fedc40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:15.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.254+0000 7fedbffff700 1 -- 172.21.15.98:0/2696030485 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5612+0+0 (secure 0 0 0) 0x7fedc8085b00 con 0x7fedd8067cc0 2023-10-06T05:08:15.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.254+0000 7fedd77fe700 1 --2- 172.21.15.98:0/2696030485 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fedc4066f30 0x7fedc40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:15.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.255+0000 7fedd77fe700 1 --2- 172.21.15.98:0/2696030485 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fedc4066f30 0x7fedc40693e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fedcc007fd0 tx=0x7fedcc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:15.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.258+0000 7fedbffff700 1 -- 172.21.15.98:0/2696030485 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fedd8142d70 con 0x7fedd8067cc0 2023-10-06T05:08:15.439 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.437+0000 7fedde536700 1 -- 172.21.15.98:0/2696030485 --> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fedd8142d70 con 0x7fedc4066f30 2023-10-06T05:08:15.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.440+0000 7fedbffff700 1 -- 172.21.15.98:0/2696030485 <== mgr.4104 v2:172.21.15.98:6832/90397 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+72767 (secure 0 0 0) 0x7fedd8142d70 con 0x7fedc4066f30 2023-10-06T05:08:15.443 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:08:15.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.442+0000 7fedde536700 1 -- 172.21.15.98:0/2696030485 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fedc4066f30 msgr2=0x7fedc40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:15.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.442+0000 7fedde536700 1 --2- 172.21.15.98:0/2696030485 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fedc4066f30 0x7fedc40693e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fedcc007fd0 tx=0x7fedcc009040 comp rx=0 tx=0).stop 2023-10-06T05:08:15.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.442+0000 7fedde536700 1 -- 172.21.15.98:0/2696030485 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fedd8067cc0 msgr2=0x7fedd80ac9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:15.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.442+0000 7fedde536700 1 --2- 172.21.15.98:0/2696030485 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fedd8067cc0 0x7fedd80ac9c0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fedd806dce0 tx=0x7fedc8017000 comp rx=0 tx=0).stop 2023-10-06T05:08:15.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.442+0000 7fedde536700 1 -- 172.21.15.98:0/2696030485 shutdown_connections 2023-10-06T05:08:15.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.442+0000 7fedde536700 1 --2- 172.21.15.98:0/2696030485 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fedd8157600 0x7fedd815b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:15.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.442+0000 7fedde536700 1 --2- 172.21.15.98:0/2696030485 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fedc4066f30 0x7fedc40693e0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:15.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.442+0000 7fedde536700 1 --2- 172.21.15.98:0/2696030485 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fedd8067cc0 0x7fedd80ac9c0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:15.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.442+0000 7fedde536700 1 --2- 172.21.15.98:0/2696030485 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fedd80ca940 0x7fedd8144c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:15.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.442+0000 7fedde536700 1 -- 172.21.15.98:0/2696030485 >> 172.21.15.98:0/2696030485 conn(0x7fedd80cdeb0 msgr2=0x7fedd80675b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:15.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.442+0000 7fedde536700 1 -- 172.21.15.98:0/2696030485 shutdown_connections 2023-10-06T05:08:15.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.442+0000 7fedde536700 1 -- 172.21.15.98:0/2696030485 wait complete. 2023-10-06T05:08:15.446 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2023-10-06T05:08:15.456 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":85,"stamp":"2023-10-06T05:08:14.535531+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":193,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2449024,"kb_used_data":70720,"kb_used_omap":0,"kb_used_meta":2378240,"kb_avail":752525696,"statfs":{"total":773094113280,"available":770586312704,"internally_reserved":0,"allocated":72417280,"data_stored":70121580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2435317760},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-33572612,"num_objects":-30,"num_object_clones":0,"num_object_copies":-60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-30,"num_whiteouts":0,"num_read":-1,"num_read_kb":-4096,"num_write":-65,"num_write_kb":-32802,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001163"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":31,"reported_epoch":45,"state":"active+clean","last_fresh":"2023-10-06T05:08:11.918743+0000","last_change":"2023-10-06T05:06:52.627232+0000","last_active":"2023-10-06T05:08:11.918743+0000","last_peered":"2023-10-06T05:08:11.918743+0000","last_clean":"2023-10-06T05:08:11.918743+0000","last_became_active":"2023-10-06T05:06:52.626936+0000","last_became_peered":"2023-10-06T05:06:52.626936+0000","last_unstale":"2023-10-06T05:08:11.918743+0000","last_undegraded":"2023-10-06T05:08:11.918743+0000","last_fullsized":"2023-10-06T05:08:11.918743+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:20:13.925170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":32,"reported_epoch":43,"state":"active+clean","last_fresh":"2023-10-06T05:07:57.487459+0000","last_change":"2023-10-06T05:06:52.587795+0000","last_active":"2023-10-06T05:07:57.487459+0000","last_peered":"2023-10-06T05:07:57.487459+0000","last_clean":"2023-10-06T05:07:57.487459+0000","last_became_active":"2023-10-06T05:06:52.587407+0000","last_became_peered":"2023-10-06T05:06:52.587407+0000","last_unstale":"2023-10-06T05:07:57.487459+0000","last_undegraded":"2023-10-06T05:07:57.487459+0000","last_fullsized":"2023-10-06T05:07:57.487459+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:36:31.526532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":28,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.499543+0000","last_change":"2023-10-06T05:06:52.675238+0000","last_active":"2023-10-06T05:07:56.499543+0000","last_peered":"2023-10-06T05:07:56.499543+0000","last_clean":"2023-10-06T05:07:56.499543+0000","last_became_active":"2023-10-06T05:06:52.674952+0000","last_became_peered":"2023-10-06T05:06:52.674952+0000","last_unstale":"2023-10-06T05:07:56.499543+0000","last_undegraded":"2023-10-06T05:07:56.499543+0000","last_fullsized":"2023-10-06T05:07:56.499543+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:27:51.141184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":32,"reported_epoch":43,"state":"active+clean","last_fresh":"2023-10-06T05:07:57.487626+0000","last_change":"2023-10-06T05:06:52.587778+0000","last_active":"2023-10-06T05:07:57.487626+0000","last_peered":"2023-10-06T05:07:57.487626+0000","last_clean":"2023-10-06T05:07:57.487626+0000","last_became_active":"2023-10-06T05:06:52.587388+0000","last_became_peered":"2023-10-06T05:06:52.587388+0000","last_unstale":"2023-10-06T05:07:57.487626+0000","last_undegraded":"2023-10-06T05:07:57.487626+0000","last_fullsized":"2023-10-06T05:07:57.487626+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:45:52.502298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":184,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-10-06T05:08:09.611541+0000","last_change":"2023-10-06T05:06:46.307169+0000","last_active":"2023-10-06T05:08:09.611541+0000","last_peered":"2023-10-06T05:08:09.611541+0000","last_clean":"2023-10-06T05:08:09.611541+0000","last_became_active":"2023-10-06T05:06:46.306107+0000","last_became_peered":"2023-10-06T05:06:46.306107+0000","last_unstale":"2023-10-06T05:08:09.611541+0000","last_undegraded":"2023-10-06T05:08:09.611541+0000","last_fullsized":"2023-10-06T05:08:09.611541+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:45.018006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:45.018006+0000","last_clean_scrub_stamp":"2023-10-06T05:06:45.018006+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:38:17.064131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":31,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-10-06T05:08:09.611654+0000","last_change":"2023-10-06T05:06:52.588475+0000","last_active":"2023-10-06T05:08:09.611654+0000","last_peered":"2023-10-06T05:08:09.611654+0000","last_clean":"2023-10-06T05:08:09.611654+0000","last_became_active":"2023-10-06T05:06:52.588134+0000","last_became_peered":"2023-10-06T05:06:52.588134+0000","last_unstale":"2023-10-06T05:08:09.611654+0000","last_undegraded":"2023-10-06T05:08:09.611654+0000","last_fullsized":"2023-10-06T05:08:09.611654+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:35:05.269778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":31,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-10-06T05:08:09.611956+0000","last_change":"2023-10-06T05:06:52.627289+0000","last_active":"2023-10-06T05:08:09.611956+0000","last_peered":"2023-10-06T05:08:09.611956+0000","last_clean":"2023-10-06T05:08:09.611956+0000","last_became_active":"2023-10-06T05:06:52.626937+0000","last_became_peered":"2023-10-06T05:06:52.626937+0000","last_unstale":"2023-10-06T05:08:09.611956+0000","last_undegraded":"2023-10-06T05:08:09.611956+0000","last_fullsized":"2023-10-06T05:08:09.611956+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:46:59.070674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":31,"reported_epoch":45,"state":"active+clean","last_fresh":"2023-10-06T05:08:11.920128+0000","last_change":"2023-10-06T05:06:52.588853+0000","last_active":"2023-10-06T05:08:11.920128+0000","last_peered":"2023-10-06T05:08:11.920128+0000","last_clean":"2023-10-06T05:08:11.920128+0000","last_became_active":"2023-10-06T05:06:52.588561+0000","last_became_peered":"2023-10-06T05:06:52.588561+0000","last_unstale":"2023-10-06T05:08:11.920128+0000","last_undegraded":"2023-10-06T05:08:11.920128+0000","last_fullsized":"2023-10-06T05:08:11.920128+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:06:28.729493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.491222+0000","last_change":"2023-10-06T05:06:52.589519+0000","last_active":"2023-10-06T05:07:56.491222+0000","last_peered":"2023-10-06T05:07:56.491222+0000","last_clean":"2023-10-06T05:07:56.491222+0000","last_became_active":"2023-10-06T05:06:52.589246+0000","last_became_peered":"2023-10-06T05:06:52.589246+0000","last_unstale":"2023-10-06T05:07:56.491222+0000","last_undegraded":"2023-10-06T05:07:56.491222+0000","last_fullsized":"2023-10-06T05:07:56.491222+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:54:23.352764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509461,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310064,"kb_used_data":12776,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94061776,"statfs":{"total":96636764160,"available":96319258624,"internally_reserved":0,"allocated":13082624,"data_stored":12802192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.34899999999999998}]},{"osd":1,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.40300000000000002}]},{"osd":2,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.53500000000000003}]},{"osd":3,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.52200000000000002}]},{"osd":4,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.58099999999999996}]},{"osd":5,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.58999999999999997}]},{"osd":7,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.56999999999999995}]}]},{"osd":5,"up_from":15,"seq":64424509461,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297772,"kb_used_data":484,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074068,"statfs":{"total":96636764160,"available":96331845632,"internally_reserved":0,"allocated":495616,"data_stored":218706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.36699999999999999}]},{"osd":1,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.39600000000000002}]},{"osd":2,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.504}]},{"osd":3,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.44700000000000001}]},{"osd":4,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.42999999999999999}]},{"osd":6,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.54400000000000004}]},{"osd":7,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.53100000000000003}]}]},{"osd":0,"up_from":14,"seq":60129542166,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":318856,"kb_used_data":21568,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94052984,"statfs":{"total":96636764160,"available":96310255616,"internally_reserved":0,"allocated":22085632,"data_stored":21793930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":1.2809999999999999,"5min":1.2809999999999999,"15min":1.2809999999999999},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.28299999999999997}]},{"osd":2,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":4.1529999999999996,"5min":4.1529999999999996,"15min":4.1529999999999996},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.372,"5min":0.372,"15min":0.372},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":1.105,"5min":1.105,"15min":1.105},"last":0.32100000000000001}]},{"osd":3,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":1.327,"5min":1.327,"15min":1.327},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":4.1900000000000004,"5min":4.1900000000000004,"15min":4.1900000000000004},"last":0.40799999999999997}]},{"osd":4,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.46800000000000003}]},{"osd":5,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.304,"5min":1.304,"15min":1.304},"last":0.432}]},{"osd":6,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.438},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.48099999999999998}]},{"osd":7,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":1.079,"5min":1.079,"15min":1.079},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.45100000000000001}]}]},{"osd":7,"up_from":13,"seq":55834574869,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306568,"kb_used_data":9280,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94065272,"statfs":{"total":96636764160,"available":96322838528,"internally_reserved":0,"allocated":9502720,"data_stored":9211018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.5},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.42699999999999999}]},{"osd":1,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.39600000000000002}]},{"osd":2,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.59199999999999997}]},{"osd":3,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.621},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.57099999999999995}]},{"osd":4,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.65700000000000003}]},{"osd":5,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.60999999999999999}]},{"osd":6,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.63}]}]},{"osd":1,"up_from":14,"seq":60129542165,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297812,"kb_used_data":524,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074028,"statfs":{"total":96636764160,"available":96331804672,"internally_reserved":0,"allocated":536576,"data_stored":238268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":1.8879999999999999,"5min":1.8879999999999999,"15min":1.8879999999999999},"last":0.19800000000000001},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":1.595,"5min":1.595,"15min":1.595},"last":0.21299999999999999}]},{"osd":2,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.123,"5min":1.123,"15min":1.123},"last":0.31}]},{"osd":3,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.179,"5min":1.179,"15min":1.179},"last":0.22900000000000001},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1.657,"5min":1.657,"15min":1.657},"last":0.29699999999999999}]},{"osd":4,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":1.6319999999999999,"5min":1.6319999999999999,"15min":1.6319999999999999},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.837,"5min":1.837,"15min":1.837},"last":0.45000000000000001}]},{"osd":5,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.8100000000000001,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.42199999999999999}]},{"osd":6,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.6850000000000001,"5min":1.6850000000000001,"15min":1.6850000000000001},"last":0.44},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.8620000000000001,"5min":1.8620000000000001,"15min":1.8620000000000001},"last":0.52000000000000002}]},{"osd":7,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.7789999999999999,"5min":1.7789999999999999,"15min":1.7789999999999999},"last":0.502},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.46100000000000002}]}]},{"osd":2,"up_from":15,"seq":64424509460,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305992,"kb_used_data":8704,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94065848,"statfs":{"total":96636764160,"available":96323428352,"internally_reserved":0,"allocated":8912896,"data_stored":8624817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.311,"5min":0.311,"15min":0.311},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.39500000000000002}]},{"osd":1,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"last":0.307},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"last":0.33000000000000002}]},{"osd":3,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"last":0.36899999999999999}]},{"osd":4,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.73499999999999999}]},{"osd":5,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.68200000000000005}]},{"osd":6,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.67400000000000004}]},{"osd":7,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.54800000000000004}]}]},{"osd":3,"up_from":15,"seq":64424509460,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310068,"kb_used_data":12780,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94061772,"statfs":{"total":96636764160,"available":96319254528,"internally_reserved":0,"allocated":13086720,"data_stored":12802158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"last":0.248},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":3.391,"5min":3.391,"15min":3.391},"last":0.27700000000000002}]},{"osd":1,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":2.8740000000000001,"5min":2.8740000000000001,"15min":2.8740000000000001},"last":0.22700000000000001},{"interface":"front","average":{"1min":0.376,"5min":0.376,"15min":0.376},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.33400000000000002}]},{"osd":2,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":7.2210000000000001,"5min":7.2210000000000001,"15min":7.2210000000000001},"last":0.38400000000000001}]},{"osd":4,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.018,"5min":1.018,"15min":1.018},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.6930000000000001,"5min":1.6930000000000001,"15min":1.6930000000000001},"last":0.45900000000000002}]},{"osd":5,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.607,"5min":1.607,"15min":1.607},"last":0.432},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.51600000000000001}]},{"osd":6,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.657,"5min":1.657,"15min":1.657},"last":0.39800000000000002}]},{"osd":7,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.877,"5min":1.877,"15min":1.877},"last":0.502}]}]},{"osd":4,"up_from":15,"seq":64424509461,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301892,"kb_used_data":4604,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94069948,"statfs":{"total":96636764160,"available":96327626752,"internally_reserved":0,"allocated":4714496,"data_stored":4430491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.45600000000000002}]},{"osd":1,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.443},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.46600000000000003}]},{"osd":2,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.41899999999999998}]},{"osd":3,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":11.385999999999999,"5min":11.385999999999999,"15min":11.385999999999999},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.52900000000000003}]},{"osd":5,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.5}]},{"osd":6,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.374},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.55000000000000004}]},{"osd":7,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.56000000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-06T05:08:15.458 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-06T05:08:15.459 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-06T05:08:15.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.570+0000 7f32323f4700 1 Processor -- start 2023-10-06T05:08:15.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.570+0000 7f32323f4700 1 -- start start 2023-10-06T05:08:15.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.571+0000 7f32323f4700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f322c159470 0x7f322c0ccc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:15.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.571+0000 7f32323f4700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f322c0c6220 0x7f322c0cf090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:15.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.571+0000 7f32323f4700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f322c0cd5d0 con 0x7f322c0cd160 2023-10-06T05:08:15.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.571+0000 7f32323f4700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f322c0cc800 con 0x7f322c0c6220 2023-10-06T05:08:15.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.571+0000 7f32323f4700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f322c0cbc30 con 0x7f322c159470 2023-10-06T05:08:15.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.571+0000 7f3230bf1700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f322c0cd160 0x7f322c0ce980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:47090/0 (socket says 172.21.15.98:47090) 2023-10-06T05:08:15.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.571+0000 7f3231bf3700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f322c0c6220 0x7f322c0cf090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:15.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.571+0000 7f3230bf1700 1 -- 172.21.15.98:0/177170311 learned_addr learned my addr 172.21.15.98:0/177170311 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:08:15.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.571+0000 7f32313f2700 1 --2- 172.21.15.98:0/177170311 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f322c159470 0x7f322c0ccc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:15.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.571+0000 7f3231bf3700 1 -- 172.21.15.98:0/177170311 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f322c159470 msgr2=0x7f322c0ccc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:15.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.572+0000 7f3231bf3700 1 --2- 172.21.15.98:0/177170311 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f322c159470 0x7f322c0ccc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:15.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.572+0000 7f3231bf3700 1 -- 172.21.15.98:0/177170311 >> v1:172.21.15.98:6789/0 conn(0x7f322c0cd160 legacy=0x7f322c0ce980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:15.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.572+0000 7f3231bf3700 1 -- 172.21.15.98:0/177170311 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f322c0cbed0 con 0x7f322c0c6220 2023-10-06T05:08:15.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.572+0000 7f3223fff700 1 -- 172.21.15.98:0/177170311 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 637680595 0 0) 0x7f322c0cd5d0 con 0x7f322c0cd160 2023-10-06T05:08:15.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.572+0000 7f32313f2700 1 --2- 172.21.15.98:0/177170311 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f322c159470 0x7f322c0ccc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:08:15.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.572+0000 7f3231bf3700 1 --2- 172.21.15.98:0/177170311 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f322c0c6220 0x7f322c0cf090 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f3224009e60 tx=0x7f3224009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e94d69900d68f512 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:15.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.572+0000 7f3223fff700 1 -- 172.21.15.98:0/177170311 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f322400e070 con 0x7f322c0c6220 2023-10-06T05:08:15.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.572+0000 7f3223fff700 1 -- 172.21.15.98:0/177170311 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f322400bc30 con 0x7f322c0c6220 2023-10-06T05:08:15.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.572+0000 7f3223fff700 1 -- 172.21.15.98:0/177170311 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3224012430 con 0x7f322c0c6220 2023-10-06T05:08:15.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.572+0000 7f32323f4700 1 -- 172.21.15.98:0/177170311 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f322c0c6220 msgr2=0x7f322c0cf090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:15.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.572+0000 7f32323f4700 1 --2- 172.21.15.98:0/177170311 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f322c0c6220 0x7f322c0cf090 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f3224009e60 tx=0x7f3224009780 comp rx=0 tx=0).stop 2023-10-06T05:08:15.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.572+0000 7f32323f4700 1 -- 172.21.15.98:0/177170311 shutdown_connections 2023-10-06T05:08:15.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.572+0000 7f32323f4700 1 --2- 172.21.15.98:0/177170311 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f322c0c6220 0x7f322c0cf090 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:15.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.572+0000 7f32323f4700 1 --2- 172.21.15.98:0/177170311 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f322c159470 0x7f322c0ccc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:15.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.572+0000 7f32323f4700 1 -- 172.21.15.98:0/177170311 >> 172.21.15.98:0/177170311 conn(0x7f322c0995c0 msgr2=0x7f322c06f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:15.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.572+0000 7f32323f4700 1 -- 172.21.15.98:0/177170311 shutdown_connections 2023-10-06T05:08:15.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.572+0000 7f32323f4700 1 -- 172.21.15.98:0/177170311 wait complete. 2023-10-06T05:08:15.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.574+0000 7f32323f4700 1 Processor -- start 2023-10-06T05:08:15.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.574+0000 7f32323f4700 1 -- start start 2023-10-06T05:08:15.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.575+0000 7f32323f4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f322c0c6220 0x7f322c169020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:15.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.575+0000 7f32323f4700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f322c0cd160 0x7f322c156c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:15.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.575+0000 7f32323f4700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f322c159470 0x7f322c154c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:15.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.575+0000 7f32323f4700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f322c157d30 con 0x7f322c0c6220 2023-10-06T05:08:15.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.575+0000 7f32323f4700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f322c15d150 con 0x7f322c0cd160 2023-10-06T05:08:15.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.575+0000 7f32323f4700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f322c157490 con 0x7f322c159470 2023-10-06T05:08:15.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.575+0000 7f32313f2700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f322c0c6220 0x7f322c169020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:15.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.575+0000 7f32313f2700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f322c0c6220 0x7f322c169020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49902/0 (socket says 172.21.15.98:49902) 2023-10-06T05:08:15.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.575+0000 7f32313f2700 1 -- 172.21.15.98:0/3898225035 learned_addr learned my addr 172.21.15.98:0/3898225035 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:08:15.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.575+0000 7f3230bf1700 1 --2- 172.21.15.98:0/3898225035 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f322c0cd160 0x7f322c156c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:15.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.575+0000 7f3231bf3700 1 --2- 172.21.15.98:0/3898225035 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f322c159470 0x7f322c154c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:15.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.575+0000 7f32313f2700 1 -- 172.21.15.98:0/3898225035 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f322c159470 msgr2=0x7f322c154c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:15.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.575+0000 7f32313f2700 1 --2- 172.21.15.98:0/3898225035 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f322c159470 0x7f322c154c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:15.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.575+0000 7f32313f2700 1 -- 172.21.15.98:0/3898225035 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f322c0cd160 msgr2=0x7f322c156c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:15.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.575+0000 7f32313f2700 1 --2- 172.21.15.98:0/3898225035 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f322c0cd160 0x7f322c156c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:15.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.575+0000 7f32313f2700 1 -- 172.21.15.98:0/3898225035 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3224009a20 con 0x7f322c0c6220 2023-10-06T05:08:15.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.575+0000 7f32313f2700 1 --2- 172.21.15.98:0/3898225035 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f322c0c6220 0x7f322c169020 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f322800c8b0 tx=0x7f3228008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:15.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.575+0000 7f32217fa700 1 -- 172.21.15.98:0/3898225035 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3228005a00 con 0x7f322c0c6220 2023-10-06T05:08:15.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.575+0000 7f32217fa700 1 -- 172.21.15.98:0/3898225035 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3228005b60 con 0x7f322c0c6220 2023-10-06T05:08:15.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.575+0000 7f32323f4700 1 -- 172.21.15.98:0/3898225035 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f322c157770 con 0x7f322c0c6220 2023-10-06T05:08:15.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.575+0000 7f32217fa700 1 -- 172.21.15.98:0/3898225035 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3228011600 con 0x7f322c0c6220 2023-10-06T05:08:15.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.575+0000 7f32323f4700 1 -- 172.21.15.98:0/3898225035 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f322c157180 con 0x7f322c0c6220 2023-10-06T05:08:15.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.576+0000 7f32323f4700 1 -- 172.21.15.98:0/3898225035 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f322c05f230 con 0x7f322c0c6220 2023-10-06T05:08:15.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.577+0000 7f32217fa700 1 -- 172.21.15.98:0/3898225035 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f32280119f0 con 0x7f322c0c6220 2023-10-06T05:08:15.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.577+0000 7f32217fa700 1 --2- 172.21.15.98:0/3898225035 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f3210066f30 0x7f32100693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:15.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.577+0000 7f32217fa700 1 -- 172.21.15.98:0/3898225035 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5612+0+0 (secure 0 0 0) 0x7f3228084170 con 0x7f322c0c6220 2023-10-06T05:08:15.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.577+0000 7f3230bf1700 1 --2- 172.21.15.98:0/3898225035 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f3210066f30 0x7f32100693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:15.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.577+0000 7f3230bf1700 1 --2- 172.21.15.98:0/3898225035 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f3210066f30 0x7f32100693e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f322c15ccd0 tx=0x7f321c007eb0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:15.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.581+0000 7f32217fa700 1 -- 172.21.15.98:0/3898225035 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3228051440 con 0x7f322c0c6220 2023-10-06T05:08:15.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.759+0000 7f32323f4700 1 -- 172.21.15.98:0/3898225035 --> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f322c15d290 con 0x7f3210066f30 2023-10-06T05:08:15.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.762+0000 7f32217fa700 1 -- 172.21.15.98:0/3898225035 <== mgr.4104 v2:172.21.15.98:6832/90397 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+72767 (secure 0 0 0) 0x7f321c004d20 con 0x7f3210066f30 2023-10-06T05:08:15.765 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:08:15.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.764+0000 7f32323f4700 1 -- 172.21.15.98:0/3898225035 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f3210066f30 msgr2=0x7f32100693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:15.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.764+0000 7f32323f4700 1 --2- 172.21.15.98:0/3898225035 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f3210066f30 0x7f32100693e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f322c15ccd0 tx=0x7f321c007eb0 comp rx=0 tx=0).stop 2023-10-06T05:08:15.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.764+0000 7f32323f4700 1 -- 172.21.15.98:0/3898225035 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f322c0c6220 msgr2=0x7f322c169020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:15.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.764+0000 7f32323f4700 1 --2- 172.21.15.98:0/3898225035 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f322c0c6220 0x7f322c169020 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f322800c8b0 tx=0x7f3228008040 comp rx=0 tx=0).stop 2023-10-06T05:08:15.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.764+0000 7f32323f4700 1 -- 172.21.15.98:0/3898225035 shutdown_connections 2023-10-06T05:08:15.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.764+0000 7f32323f4700 1 --2- 172.21.15.98:0/3898225035 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f322c159470 0x7f322c154c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:15.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.764+0000 7f32323f4700 1 --2- 172.21.15.98:0/3898225035 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f3210066f30 0x7f32100693e0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:15.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.764+0000 7f32323f4700 1 --2- 172.21.15.98:0/3898225035 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f322c0c6220 0x7f322c169020 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:15.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.764+0000 7f32323f4700 1 --2- 172.21.15.98:0/3898225035 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f322c0cd160 0x7f322c156c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:15.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.764+0000 7f32323f4700 1 -- 172.21.15.98:0/3898225035 >> 172.21.15.98:0/3898225035 conn(0x7f322c0995c0 msgr2=0x7f322c0d02e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:15.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.764+0000 7f32323f4700 1 -- 172.21.15.98:0/3898225035 shutdown_connections 2023-10-06T05:08:15.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.764+0000 7f32323f4700 1 -- 172.21.15.98:0/3898225035 wait complete. 2023-10-06T05:08:15.768 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2023-10-06T05:08:15.778 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":85,"stamp":"2023-10-06T05:08:14.535531+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":193,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2449024,"kb_used_data":70720,"kb_used_omap":0,"kb_used_meta":2378240,"kb_avail":752525696,"statfs":{"total":773094113280,"available":770586312704,"internally_reserved":0,"allocated":72417280,"data_stored":70121580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2435317760},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-33572612,"num_objects":-30,"num_object_clones":0,"num_object_copies":-60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-30,"num_whiteouts":0,"num_read":-1,"num_read_kb":-4096,"num_write":-65,"num_write_kb":-32802,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001163"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":31,"reported_epoch":45,"state":"active+clean","last_fresh":"2023-10-06T05:08:11.918743+0000","last_change":"2023-10-06T05:06:52.627232+0000","last_active":"2023-10-06T05:08:11.918743+0000","last_peered":"2023-10-06T05:08:11.918743+0000","last_clean":"2023-10-06T05:08:11.918743+0000","last_became_active":"2023-10-06T05:06:52.626936+0000","last_became_peered":"2023-10-06T05:06:52.626936+0000","last_unstale":"2023-10-06T05:08:11.918743+0000","last_undegraded":"2023-10-06T05:08:11.918743+0000","last_fullsized":"2023-10-06T05:08:11.918743+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:20:13.925170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":32,"reported_epoch":43,"state":"active+clean","last_fresh":"2023-10-06T05:07:57.487459+0000","last_change":"2023-10-06T05:06:52.587795+0000","last_active":"2023-10-06T05:07:57.487459+0000","last_peered":"2023-10-06T05:07:57.487459+0000","last_clean":"2023-10-06T05:07:57.487459+0000","last_became_active":"2023-10-06T05:06:52.587407+0000","last_became_peered":"2023-10-06T05:06:52.587407+0000","last_unstale":"2023-10-06T05:07:57.487459+0000","last_undegraded":"2023-10-06T05:07:57.487459+0000","last_fullsized":"2023-10-06T05:07:57.487459+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:36:31.526532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":28,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.499543+0000","last_change":"2023-10-06T05:06:52.675238+0000","last_active":"2023-10-06T05:07:56.499543+0000","last_peered":"2023-10-06T05:07:56.499543+0000","last_clean":"2023-10-06T05:07:56.499543+0000","last_became_active":"2023-10-06T05:06:52.674952+0000","last_became_peered":"2023-10-06T05:06:52.674952+0000","last_unstale":"2023-10-06T05:07:56.499543+0000","last_undegraded":"2023-10-06T05:07:56.499543+0000","last_fullsized":"2023-10-06T05:07:56.499543+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:27:51.141184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":32,"reported_epoch":43,"state":"active+clean","last_fresh":"2023-10-06T05:07:57.487626+0000","last_change":"2023-10-06T05:06:52.587778+0000","last_active":"2023-10-06T05:07:57.487626+0000","last_peered":"2023-10-06T05:07:57.487626+0000","last_clean":"2023-10-06T05:07:57.487626+0000","last_became_active":"2023-10-06T05:06:52.587388+0000","last_became_peered":"2023-10-06T05:06:52.587388+0000","last_unstale":"2023-10-06T05:07:57.487626+0000","last_undegraded":"2023-10-06T05:07:57.487626+0000","last_fullsized":"2023-10-06T05:07:57.487626+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:45:52.502298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":184,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-10-06T05:08:09.611541+0000","last_change":"2023-10-06T05:06:46.307169+0000","last_active":"2023-10-06T05:08:09.611541+0000","last_peered":"2023-10-06T05:08:09.611541+0000","last_clean":"2023-10-06T05:08:09.611541+0000","last_became_active":"2023-10-06T05:06:46.306107+0000","last_became_peered":"2023-10-06T05:06:46.306107+0000","last_unstale":"2023-10-06T05:08:09.611541+0000","last_undegraded":"2023-10-06T05:08:09.611541+0000","last_fullsized":"2023-10-06T05:08:09.611541+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:45.018006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:45.018006+0000","last_clean_scrub_stamp":"2023-10-06T05:06:45.018006+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:38:17.064131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":31,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-10-06T05:08:09.611654+0000","last_change":"2023-10-06T05:06:52.588475+0000","last_active":"2023-10-06T05:08:09.611654+0000","last_peered":"2023-10-06T05:08:09.611654+0000","last_clean":"2023-10-06T05:08:09.611654+0000","last_became_active":"2023-10-06T05:06:52.588134+0000","last_became_peered":"2023-10-06T05:06:52.588134+0000","last_unstale":"2023-10-06T05:08:09.611654+0000","last_undegraded":"2023-10-06T05:08:09.611654+0000","last_fullsized":"2023-10-06T05:08:09.611654+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:35:05.269778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":31,"reported_epoch":44,"state":"active+clean","last_fresh":"2023-10-06T05:08:09.611956+0000","last_change":"2023-10-06T05:06:52.627289+0000","last_active":"2023-10-06T05:08:09.611956+0000","last_peered":"2023-10-06T05:08:09.611956+0000","last_clean":"2023-10-06T05:08:09.611956+0000","last_became_active":"2023-10-06T05:06:52.626937+0000","last_became_peered":"2023-10-06T05:06:52.626937+0000","last_unstale":"2023-10-06T05:08:09.611956+0000","last_undegraded":"2023-10-06T05:08:09.611956+0000","last_fullsized":"2023-10-06T05:08:09.611956+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:46:59.070674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":31,"reported_epoch":45,"state":"active+clean","last_fresh":"2023-10-06T05:08:11.920128+0000","last_change":"2023-10-06T05:06:52.588853+0000","last_active":"2023-10-06T05:08:11.920128+0000","last_peered":"2023-10-06T05:08:11.920128+0000","last_clean":"2023-10-06T05:08:11.920128+0000","last_became_active":"2023-10-06T05:06:52.588561+0000","last_became_peered":"2023-10-06T05:06:52.588561+0000","last_unstale":"2023-10-06T05:08:11.920128+0000","last_undegraded":"2023-10-06T05:08:11.920128+0000","last_fullsized":"2023-10-06T05:08:11.920128+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:06:28.729493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":42,"state":"active+clean","last_fresh":"2023-10-06T05:07:56.491222+0000","last_change":"2023-10-06T05:06:52.589519+0000","last_active":"2023-10-06T05:07:56.491222+0000","last_peered":"2023-10-06T05:07:56.491222+0000","last_clean":"2023-10-06T05:07:56.491222+0000","last_became_active":"2023-10-06T05:06:52.589246+0000","last_became_peered":"2023-10-06T05:06:52.589246+0000","last_unstale":"2023-10-06T05:07:56.491222+0000","last_undegraded":"2023-10-06T05:07:56.491222+0000","last_fullsized":"2023-10-06T05:07:56.491222+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:54:23.352764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509461,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310064,"kb_used_data":12776,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94061776,"statfs":{"total":96636764160,"available":96319258624,"internally_reserved":0,"allocated":13082624,"data_stored":12802192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.34899999999999998}]},{"osd":1,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.40300000000000002}]},{"osd":2,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.53500000000000003}]},{"osd":3,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.52200000000000002}]},{"osd":4,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.58099999999999996}]},{"osd":5,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.58999999999999997}]},{"osd":7,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.56999999999999995}]}]},{"osd":5,"up_from":15,"seq":64424509461,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297772,"kb_used_data":484,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074068,"statfs":{"total":96636764160,"available":96331845632,"internally_reserved":0,"allocated":495616,"data_stored":218706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.36699999999999999}]},{"osd":1,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.39600000000000002}]},{"osd":2,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.504}]},{"osd":3,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.44700000000000001}]},{"osd":4,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.42999999999999999}]},{"osd":6,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.54400000000000004}]},{"osd":7,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.53100000000000003}]}]},{"osd":0,"up_from":14,"seq":60129542166,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":318856,"kb_used_data":21568,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94052984,"statfs":{"total":96636764160,"available":96310255616,"internally_reserved":0,"allocated":22085632,"data_stored":21793930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":1.2809999999999999,"5min":1.2809999999999999,"15min":1.2809999999999999},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.28299999999999997}]},{"osd":2,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":4.1529999999999996,"5min":4.1529999999999996,"15min":4.1529999999999996},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.372,"5min":0.372,"15min":0.372},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":1.105,"5min":1.105,"15min":1.105},"last":0.32100000000000001}]},{"osd":3,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":1.327,"5min":1.327,"15min":1.327},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":4.1900000000000004,"5min":4.1900000000000004,"15min":4.1900000000000004},"last":0.40799999999999997}]},{"osd":4,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.46800000000000003}]},{"osd":5,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.304,"5min":1.304,"15min":1.304},"last":0.432}]},{"osd":6,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.438},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.48099999999999998}]},{"osd":7,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":1.079,"5min":1.079,"15min":1.079},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.45100000000000001}]}]},{"osd":7,"up_from":13,"seq":55834574869,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306568,"kb_used_data":9280,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94065272,"statfs":{"total":96636764160,"available":96322838528,"internally_reserved":0,"allocated":9502720,"data_stored":9211018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.5},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.42699999999999999}]},{"osd":1,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.39600000000000002}]},{"osd":2,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.59199999999999997}]},{"osd":3,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.621},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.57099999999999995}]},{"osd":4,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.65700000000000003}]},{"osd":5,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.60999999999999999}]},{"osd":6,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.63}]}]},{"osd":1,"up_from":14,"seq":60129542165,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297812,"kb_used_data":524,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074028,"statfs":{"total":96636764160,"available":96331804672,"internally_reserved":0,"allocated":536576,"data_stored":238268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":1.8879999999999999,"5min":1.8879999999999999,"15min":1.8879999999999999},"last":0.19800000000000001},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":1.595,"5min":1.595,"15min":1.595},"last":0.21299999999999999}]},{"osd":2,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.123,"5min":1.123,"15min":1.123},"last":0.31}]},{"osd":3,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.179,"5min":1.179,"15min":1.179},"last":0.22900000000000001},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1.657,"5min":1.657,"15min":1.657},"last":0.29699999999999999}]},{"osd":4,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":1.6319999999999999,"5min":1.6319999999999999,"15min":1.6319999999999999},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.837,"5min":1.837,"15min":1.837},"last":0.45000000000000001}]},{"osd":5,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.8100000000000001,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.42199999999999999}]},{"osd":6,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.6850000000000001,"5min":1.6850000000000001,"15min":1.6850000000000001},"last":0.44},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.8620000000000001,"5min":1.8620000000000001,"15min":1.8620000000000001},"last":0.52000000000000002}]},{"osd":7,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.7789999999999999,"5min":1.7789999999999999,"15min":1.7789999999999999},"last":0.502},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.46100000000000002}]}]},{"osd":2,"up_from":15,"seq":64424509460,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305992,"kb_used_data":8704,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94065848,"statfs":{"total":96636764160,"available":96323428352,"internally_reserved":0,"allocated":8912896,"data_stored":8624817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.311,"5min":0.311,"15min":0.311},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.39500000000000002}]},{"osd":1,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"last":0.307},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"last":0.33000000000000002}]},{"osd":3,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"last":0.36899999999999999}]},{"osd":4,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.73499999999999999}]},{"osd":5,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.68200000000000005}]},{"osd":6,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.67400000000000004}]},{"osd":7,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.54800000000000004}]}]},{"osd":3,"up_from":15,"seq":64424509460,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310068,"kb_used_data":12780,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94061772,"statfs":{"total":96636764160,"available":96319254528,"internally_reserved":0,"allocated":13086720,"data_stored":12802158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"last":0.248},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":3.391,"5min":3.391,"15min":3.391},"last":0.27700000000000002}]},{"osd":1,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":2.8740000000000001,"5min":2.8740000000000001,"15min":2.8740000000000001},"last":0.22700000000000001},{"interface":"front","average":{"1min":0.376,"5min":0.376,"15min":0.376},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.33400000000000002}]},{"osd":2,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":7.2210000000000001,"5min":7.2210000000000001,"15min":7.2210000000000001},"last":0.38400000000000001}]},{"osd":4,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.018,"5min":1.018,"15min":1.018},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.6930000000000001,"5min":1.6930000000000001,"15min":1.6930000000000001},"last":0.45900000000000002}]},{"osd":5,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.607,"5min":1.607,"15min":1.607},"last":0.432},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.51600000000000001}]},{"osd":6,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.657,"5min":1.657,"15min":1.657},"last":0.39800000000000002}]},{"osd":7,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.877,"5min":1.877,"15min":1.877},"last":0.502}]}]},{"osd":4,"up_from":15,"seq":64424509461,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301892,"kb_used_data":4604,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94069948,"statfs":{"total":96636764160,"available":96327626752,"internally_reserved":0,"allocated":4714496,"data_stored":4430491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.45600000000000002}]},{"osd":1,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.443},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.46600000000000003}]},{"osd":2,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.41899999999999998}]},{"osd":3,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":11.385999999999999,"5min":11.385999999999999,"15min":11.385999999999999},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.52900000000000003}]},{"osd":5,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.5}]},{"osd":6,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.374},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.55000000000000004}]},{"osd":7,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.56000000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-06T05:08:15.781 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T05:08:15.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.893+0000 7f582cad9700 1 Processor -- start 2023-10-06T05:08:15.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.893+0000 7f582cad9700 1 -- start start 2023-10-06T05:08:15.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.893+0000 7f582cad9700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5828159560 0x7f58280aa530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:15.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.893+0000 7f582cad9700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f58280b49a0 0x7f58280aaa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:15.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.893+0000 7f582cad9700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f58280b0760 0x7f58280b3890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:15.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.893+0000 7f582cad9700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58280b0c60 con 0x7f5828159560 2023-10-06T05:08:15.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.893+0000 7f582cad9700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58280b0f40 con 0x7f58280b0760 2023-10-06T05:08:15.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.893+0000 7f582cad9700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58280b1220 con 0x7f58280b49a0 2023-10-06T05:08:15.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.893+0000 7f582659c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5828159560 0x7f58280aa530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:15.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.894+0000 7f582659c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5828159560 0x7f58280aa530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49914/0 (socket says 172.21.15.98:49914) 2023-10-06T05:08:15.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.894+0000 7f582659c700 1 -- 172.21.15.98:0/1472862768 learned_addr learned my addr 172.21.15.98:0/1472862768 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:08:15.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.894+0000 7f5826d9d700 1 --2- 172.21.15.98:0/1472862768 >> v2:172.21.15.174:3300/0 conn(0x7f58280b0760 0x7f58280b3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:15.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.894+0000 7f581dd9b700 1 --2- 172.21.15.98:0/1472862768 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f58280b49a0 0x7f58280aaa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:15.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.894+0000 7f582659c700 1 -- 172.21.15.98:0/1472862768 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f58280b49a0 msgr2=0x7f58280aaa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:15.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.894+0000 7f582659c700 1 --2- 172.21.15.98:0/1472862768 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f58280b49a0 0x7f58280aaa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:15.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.894+0000 7f582659c700 1 -- 172.21.15.98:0/1472862768 >> v2:172.21.15.174:3300/0 conn(0x7f58280b0760 msgr2=0x7f58280b3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:15.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.894+0000 7f582659c700 1 --2- 172.21.15.98:0/1472862768 >> v2:172.21.15.174:3300/0 conn(0x7f58280b0760 0x7f58280b3890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:15.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.894+0000 7f582659c700 1 -- 172.21.15.98:0/1472862768 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58280b1500 con 0x7f5828159560 2023-10-06T05:08:15.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.894+0000 7f5826d9d700 1 --2- 172.21.15.98:0/1472862768 >> v2:172.21.15.174:3300/0 conn(0x7f58280b0760 0x7f58280b3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:08:15.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.894+0000 7f582659c700 1 --2- 172.21.15.98:0/1472862768 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5828159560 0x7f58280aa530 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f58280d0460 tx=0x7f5818009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2e7f15e47919376 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:15.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.894+0000 7f5825d9b700 1 -- 172.21.15.98:0/1472862768 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5818005c10 con 0x7f5828159560 2023-10-06T05:08:15.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.894+0000 7f5825d9b700 1 -- 172.21.15.98:0/1472862768 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5818005d70 con 0x7f5828159560 2023-10-06T05:08:15.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.894+0000 7f5825d9b700 1 -- 172.21.15.98:0/1472862768 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f581800b880 con 0x7f5828159560 2023-10-06T05:08:15.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.895+0000 7f582cad9700 1 -- 172.21.15.98:0/1472862768 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5828159560 msgr2=0x7f58280aa530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:15.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.895+0000 7f582cad9700 1 --2- 172.21.15.98:0/1472862768 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5828159560 0x7f58280aa530 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f58280d0460 tx=0x7f5818009220 comp rx=0 tx=0).stop 2023-10-06T05:08:15.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.895+0000 7f582cad9700 1 -- 172.21.15.98:0/1472862768 shutdown_connections 2023-10-06T05:08:15.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.895+0000 7f582cad9700 1 --2- 172.21.15.98:0/1472862768 >> v2:172.21.15.174:3300/0 conn(0x7f58280b0760 0x7f58280b3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:15.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.895+0000 7f582cad9700 1 --2- 172.21.15.98:0/1472862768 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5828159560 0x7f58280aa530 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:15.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.895+0000 7f582cad9700 1 --2- 172.21.15.98:0/1472862768 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f58280b49a0 0x7f58280aaa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:15.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.895+0000 7f582cad9700 1 -- 172.21.15.98:0/1472862768 >> 172.21.15.98:0/1472862768 conn(0x7f58280997f0 msgr2=0x7f582806f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:15.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.895+0000 7f582cad9700 1 -- 172.21.15.98:0/1472862768 shutdown_connections 2023-10-06T05:08:15.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.895+0000 7f582cad9700 1 -- 172.21.15.98:0/1472862768 wait complete. 2023-10-06T05:08:15.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.896+0000 7f582cad9700 1 Processor -- start 2023-10-06T05:08:15.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.896+0000 7f582cad9700 1 -- start start 2023-10-06T05:08:15.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.897+0000 7f582cad9700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f58280b0760 0x7f5828156180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:15.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.897+0000 7f582cad9700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f58280b49a0 0x7f582815b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:15.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.897+0000 7f582cad9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5828159560 0x7f58280b1b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:15.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.897+0000 7f582cad9700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58281566c0 con 0x7f5828159560 2023-10-06T05:08:15.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.897+0000 7f582cad9700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f582809b540 con 0x7f58280b0760 2023-10-06T05:08:15.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.897+0000 7f582cad9700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f582809b820 con 0x7f58280b49a0 2023-10-06T05:08:15.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.897+0000 7f5826d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5828159560 0x7f58280b1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:15.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.897+0000 7f5826d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5828159560 0x7f58280b1b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49920/0 (socket says 172.21.15.98:49920) 2023-10-06T05:08:15.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.897+0000 7f5826d9d700 1 -- 172.21.15.98:0/288108789 learned_addr learned my addr 172.21.15.98:0/288108789 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:08:15.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.897+0000 7f582659c700 1 --2- 172.21.15.98:0/288108789 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f58280b0760 0x7f5828156180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:15.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.897+0000 7f581dd9b700 1 --2- 172.21.15.98:0/288108789 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f58280b49a0 0x7f582815b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:15.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.897+0000 7f5826d9d700 1 -- 172.21.15.98:0/288108789 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f58280b49a0 msgr2=0x7f582815b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:15.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.897+0000 7f5826d9d700 1 --2- 172.21.15.98:0/288108789 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f58280b49a0 0x7f582815b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:15.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.897+0000 7f5826d9d700 1 -- 172.21.15.98:0/288108789 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f58280b0760 msgr2=0x7f5828156180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:15.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.897+0000 7f5826d9d700 1 --2- 172.21.15.98:0/288108789 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f58280b0760 0x7f5828156180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:15.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.897+0000 7f5826d9d700 1 -- 172.21.15.98:0/288108789 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5818009480 con 0x7f5828159560 2023-10-06T05:08:15.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.897+0000 7f5826d9d700 1 --2- 172.21.15.98:0/288108789 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5828159560 0x7f58280b1b50 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f582000a380 tx=0x7f5820007900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:15.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.897+0000 7f581effd700 1 -- 172.21.15.98:0/288108789 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5820015440 con 0x7f5828159560 2023-10-06T05:08:15.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.897+0000 7f581effd700 1 -- 172.21.15.98:0/288108789 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58200155a0 con 0x7f5828159560 2023-10-06T05:08:15.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.897+0000 7f581effd700 1 -- 172.21.15.98:0/288108789 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5820015960 con 0x7f5828159560 2023-10-06T05:08:15.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.897+0000 7f582cad9700 1 -- 172.21.15.98:0/288108789 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f582809bb00 con 0x7f5828159560 2023-10-06T05:08:15.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.897+0000 7f582cad9700 1 -- 172.21.15.98:0/288108789 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f582815bcb0 con 0x7f5828159560 2023-10-06T05:08:15.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.898+0000 7f582cad9700 1 -- 172.21.15.98:0/288108789 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5828064a90 con 0x7f5828159560 2023-10-06T05:08:15.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.899+0000 7f581effd700 1 -- 172.21.15.98:0/288108789 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f582000d440 con 0x7f5828159560 2023-10-06T05:08:15.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.899+0000 7f581effd700 1 --2- 172.21.15.98:0/288108789 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f5808066f30 0x7f58080693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:15.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.899+0000 7f581effd700 1 -- 172.21.15.98:0/288108789 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5612+0+0 (secure 0 0 0) 0x7f5820084fd0 con 0x7f5828159560 2023-10-06T05:08:15.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.899+0000 7f582659c700 1 --2- 172.21.15.98:0/288108789 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f5808066f30 0x7f58080693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:15.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.899+0000 7f582659c700 1 --2- 172.21.15.98:0/288108789 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f5808066f30 0x7f58080693e0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f582815c4d0 tx=0x7f5818016000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:15.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:15.903+0000 7f581effd700 1 -- 172.21.15.98:0/288108789 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f58200522a0 con 0x7f5828159560 2023-10-06T05:08:16.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:16.088+0000 7f582cad9700 1 -- 172.21.15.98:0/288108789 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f58280b17e0 con 0x7f5828159560 2023-10-06T05:08:16.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:16.088+0000 7f581effd700 1 -- 172.21.15.98:0/288108789 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+13984 (secure 0 0 0) 0x7f58200558c0 con 0x7f5828159560 2023-10-06T05:08:16.091 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:08:16.091 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":46,"fsid":"f50c5df5-cad5-47a4-9bb1-8c791d4311ec","created":"2023-10-06T05:06:32.971175+0000","modified":"2023-10-06T05:08:12.753522+0000","last_up_change":"2023-10-06T05:06:44.016489+0000","last_in_change":"2023-10-06T05:06:35.544511+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-06T05:06:44.559671+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-06T05:06:51.432161+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d9495d95-da75-4c3b-8927-e9bae6a8b093","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":90627},{"type":"v1","addr":"172.21.15.98:6801","nonce":90627}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":90627},{"type":"v1","addr":"172.21.15.98:6803","nonce":90627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":90627},{"type":"v1","addr":"172.21.15.98:6807","nonce":90627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":90627},{"type":"v1","addr":"172.21.15.98:6805","nonce":90627}]},"public_addr":"172.21.15.98:6801/90627","cluster_addr":"172.21.15.98:6803/90627","heartbeat_back_addr":"172.21.15.98:6807/90627","heartbeat_front_addr":"172.21.15.98:6805/90627","state":["exists","up"]},{"osd":1,"uuid":"f25ba5be-d65e-4707-a134-154092deb754","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":90628},{"type":"v1","addr":"172.21.15.98:6809","nonce":90628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":90628},{"type":"v1","addr":"172.21.15.98:6811","nonce":90628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":90628},{"type":"v1","addr":"172.21.15.98:6815","nonce":90628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":90628},{"type":"v1","addr":"172.21.15.98:6813","nonce":90628}]},"public_addr":"172.21.15.98:6809/90628","cluster_addr":"172.21.15.98:6811/90628","heartbeat_back_addr":"172.21.15.98:6815/90628","heartbeat_front_addr":"172.21.15.98:6813/90628","state":["exists","up"]},{"osd":2,"uuid":"5ab9e028-945c-43e2-8c95-3b0b4cf38e39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":90662},{"type":"v1","addr":"172.21.15.98:6818","nonce":90662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":90662},{"type":"v1","addr":"172.21.15.98:6822","nonce":90662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":90662},{"type":"v1","addr":"172.21.15.98:6830","nonce":90662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":90662},{"type":"v1","addr":"172.21.15.98:6826","nonce":90662}]},"public_addr":"172.21.15.98:6818/90662","cluster_addr":"172.21.15.98:6822/90662","heartbeat_back_addr":"172.21.15.98:6830/90662","heartbeat_front_addr":"172.21.15.98:6826/90662","state":["exists","up"]},{"osd":3,"uuid":"3bb803ff-1738-46bf-be62-2267bf378822","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6817","nonce":90663},{"type":"v1","addr":"172.21.15.98:6819","nonce":90663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6821","nonce":90663},{"type":"v1","addr":"172.21.15.98:6823","nonce":90663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6829","nonce":90663},{"type":"v1","addr":"172.21.15.98:6831","nonce":90663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6825","nonce":90663},{"type":"v1","addr":"172.21.15.98:6827","nonce":90663}]},"public_addr":"172.21.15.98:6819/90663","cluster_addr":"172.21.15.98:6823/90663","heartbeat_back_addr":"172.21.15.98:6831/90663","heartbeat_front_addr":"172.21.15.98:6827/90663","state":["exists","up"]},{"osd":4,"uuid":"7d57a4c9-6462-43f6-b64a-7f9e9b4353d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":90847},{"type":"v1","addr":"172.21.15.174:6801","nonce":90847}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":90847},{"type":"v1","addr":"172.21.15.174:6803","nonce":90847}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":90847},{"type":"v1","addr":"172.21.15.174:6807","nonce":90847}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":90847},{"type":"v1","addr":"172.21.15.174:6805","nonce":90847}]},"public_addr":"172.21.15.174:6801/90847","cluster_addr":"172.21.15.174:6803/90847","heartbeat_back_addr":"172.21.15.174:6807/90847","heartbeat_front_addr":"172.21.15.174:6805/90847","state":["exists","up"]},{"osd":5,"uuid":"83287ea4-e44d-4af9-bc77-36d47c60d4a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":90848},{"type":"v1","addr":"172.21.15.174:6809","nonce":90848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":90848},{"type":"v1","addr":"172.21.15.174:6811","nonce":90848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":90848},{"type":"v1","addr":"172.21.15.174:6815","nonce":90848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":90848},{"type":"v1","addr":"172.21.15.174:6813","nonce":90848}]},"public_addr":"172.21.15.174:6809/90848","cluster_addr":"172.21.15.174:6811/90848","heartbeat_back_addr":"172.21.15.174:6815/90848","heartbeat_front_addr":"172.21.15.174:6813/90848","state":["exists","up"]},{"osd":6,"uuid":"496a6272-892e-43c4-a145-206623c22f99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":90877},{"type":"v1","addr":"172.21.15.174:6817","nonce":90877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":90877},{"type":"v1","addr":"172.21.15.174:6819","nonce":90877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":90877},{"type":"v1","addr":"172.21.15.174:6823","nonce":90877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":90877},{"type":"v1","addr":"172.21.15.174:6821","nonce":90877}]},"public_addr":"172.21.15.174:6817/90877","cluster_addr":"172.21.15.174:6819/90877","heartbeat_back_addr":"172.21.15.174:6823/90877","heartbeat_front_addr":"172.21.15.174:6821/90877","state":["exists","up"]},{"osd":7,"uuid":"49967e34-8b07-4bab-a13e-0c16d6a0f3a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":90878},{"type":"v1","addr":"172.21.15.174:6825","nonce":90878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":90878},{"type":"v1","addr":"172.21.15.174:6827","nonce":90878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":90878},{"type":"v1","addr":"172.21.15.174:6831","nonce":90878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":90878},{"type":"v1","addr":"172.21.15.174:6829","nonce":90878}]},"public_addr":"172.21.15.174:6825/90878","cluster_addr":"172.21.15.174:6827/90878","heartbeat_back_addr":"172.21.15.174:6831/90878","heartbeat_front_addr":"172.21.15.174:6829/90878","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.446479+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.514266+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.794108+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.810894+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.731919+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.717756+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.737435+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T05:06:40.508682+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6837/1617154626":"2023-10-07T05:08:09.607156+0000","172.21.15.98:6835/1617154626":"2023-10-07T05:08:09.607156+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T05:08:16.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:16.090+0000 7f582cad9700 1 -- 172.21.15.98:0/288108789 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f5808066f30 msgr2=0x7f58080693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:16.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:16.090+0000 7f582cad9700 1 --2- 172.21.15.98:0/288108789 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f5808066f30 0x7f58080693e0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f582815c4d0 tx=0x7f5818016000 comp rx=0 tx=0).stop 2023-10-06T05:08:16.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:16.090+0000 7f582cad9700 1 -- 172.21.15.98:0/288108789 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5828159560 msgr2=0x7f58280b1b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:16.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:16.090+0000 7f582cad9700 1 --2- 172.21.15.98:0/288108789 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5828159560 0x7f58280b1b50 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f582000a380 tx=0x7f5820007900 comp rx=0 tx=0).stop 2023-10-06T05:08:16.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:16.090+0000 7f582cad9700 1 -- 172.21.15.98:0/288108789 shutdown_connections 2023-10-06T05:08:16.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:16.090+0000 7f582cad9700 1 --2- 172.21.15.98:0/288108789 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f5808066f30 0x7f58080693e0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:16.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:16.090+0000 7f582cad9700 1 --2- 172.21.15.98:0/288108789 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5828159560 0x7f58280b1b50 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:16.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:16.090+0000 7f582cad9700 1 --2- 172.21.15.98:0/288108789 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f58280b0760 0x7f5828156180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:16.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:16.090+0000 7f582cad9700 1 --2- 172.21.15.98:0/288108789 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f58280b49a0 0x7f582815b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:16.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:16.090+0000 7f582cad9700 1 -- 172.21.15.98:0/288108789 >> 172.21.15.98:0/288108789 conn(0x7f58280997f0 msgr2=0x7f58280683c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:16.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:16.090+0000 7f582cad9700 1 -- 172.21.15.98:0/288108789 shutdown_connections 2023-10-06T05:08:16.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:16.091+0000 7f582cad9700 1 -- 172.21.15.98:0/288108789 wait complete. 2023-10-06T05:08:17.106 INFO:tasks.ceph:Scrubbing osd.0 2023-10-06T05:08:17.106 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-10-06T05:08:17.237 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.235+0000 7f70fcbb6700 1 Processor -- start 2023-10-06T05:08:17.237 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.235+0000 7f70fcbb6700 1 -- start start 2023-10-06T05:08:17.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.235+0000 7f70fcbb6700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f70f8153990 0x7f70f815a370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:17.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.235+0000 7f70fcbb6700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f70f80d09c0 0x7f70f80ce860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:17.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.235+0000 7f70fcbb6700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70f80cdbf0 con 0x7f70f80cca70 2023-10-06T05:08:17.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.235+0000 7f70fcbb6700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70f80cb4c0 con 0x7f70f80d09c0 2023-10-06T05:08:17.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.235+0000 7f70fcbb6700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70f815a8b0 con 0x7f70f8153990 2023-10-06T05:08:17.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.236+0000 7f70effff700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f70f80cca70 0x7f70f80d02b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:47094/0 (socket says 172.21.15.98:47094) 2023-10-06T05:08:17.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.236+0000 7f70effff700 1 -- 172.21.15.98:0/637654914 learned_addr learned my addr 172.21.15.98:0/637654914 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:08:17.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.236+0000 7f70f7fff700 1 --2- 172.21.15.98:0/637654914 >> v2:172.21.15.174:3300/0 conn(0x7f70f80d09c0 0x7f70f80ce860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:17.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.236+0000 7f70f6ffd700 1 -- 172.21.15.98:0/637654914 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 281065378 0 0) 0x7f70f80cdbf0 con 0x7f70f80cca70 2023-10-06T05:08:17.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.236+0000 7f70f77fe700 1 --2- 172.21.15.98:0/637654914 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f70f8153990 0x7f70f815a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:17.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.236+0000 7f70f6ffd700 1 -- 172.21.15.98:0/637654914 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70e4003570 con 0x7f70f80cca70 2023-10-06T05:08:17.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.236+0000 7f70f6ffd700 1 -- 172.21.15.98:0/637654914 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3806837492 0 0) 0x7f70e4003570 con 0x7f70f80cca70 2023-10-06T05:08:17.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.236+0000 7f70f6ffd700 1 -- 172.21.15.98:0/637654914 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f70f8153990 msgr2=0x7f70f815a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:17.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.236+0000 7f70f6ffd700 1 --2- 172.21.15.98:0/637654914 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f70f8153990 0x7f70f815a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:17.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.236+0000 7f70f6ffd700 1 -- 172.21.15.98:0/637654914 >> v2:172.21.15.174:3300/0 conn(0x7f70f80d09c0 msgr2=0x7f70f80ce860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:17.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.236+0000 7f70f6ffd700 1 --2- 172.21.15.98:0/637654914 >> v2:172.21.15.174:3300/0 conn(0x7f70f80d09c0 0x7f70f80ce860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:17.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.236+0000 7f70f6ffd700 1 -- 172.21.15.98:0/637654914 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70f80ccfe0 con 0x7f70f80cca70 2023-10-06T05:08:17.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.236+0000 7f70f6ffd700 1 -- 172.21.15.98:0/637654914 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1543991876 0 0) 0x7f70e0001470 con 0x7f70f80cca70 2023-10-06T05:08:17.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.236+0000 7f70f6ffd700 1 -- 172.21.15.98:0/637654914 >> v1:172.21.15.98:6789/0 conn(0x7f70f80cca70 legacy=0x7f70f80d02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:17.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.236+0000 7f70f6ffd700 1 --2- 172.21.15.98:0/637654914 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f70e4003a60 0x7f70e4003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:17.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.236+0000 7f70f6ffd700 1 --2- 172.21.15.98:0/637654914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70e4004810 0x7f70e4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:17.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.236+0000 7f70f6ffd700 1 --2- 172.21.15.98:0/637654914 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f70f8153990 0x7f70e40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:17.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.236+0000 7f70f6ffd700 1 -- 172.21.15.98:0/637654914 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70f80cb4c0 con 0x7f70e4004810 2023-10-06T05:08:17.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.236+0000 7f70f6ffd700 1 -- 172.21.15.98:0/637654914 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70f815a8b0 con 0x7f70e4003a60 2023-10-06T05:08:17.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.236+0000 7f70f6ffd700 1 -- 172.21.15.98:0/637654914 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70e400ba60 con 0x7f70f8153990 2023-10-06T05:08:17.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.236+0000 7f70effff700 1 --2- 172.21.15.98:0/637654914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70e4004810 0x7f70e4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:17.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.236+0000 7f70f77fe700 1 --2- 172.21.15.98:0/637654914 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f70e4003a60 0x7f70e4003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:17.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.236+0000 7f70f7fff700 1 --2- 172.21.15.98:0/637654914 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f70f8153990 0x7f70e40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:17.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.237+0000 7f70effff700 1 -- 172.21.15.98:0/637654914 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f70f8153990 msgr2=0x7f70e40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:17.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.237+0000 7f70effff700 1 --2- 172.21.15.98:0/637654914 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f70f8153990 0x7f70e40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:17.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.237+0000 7f70effff700 1 -- 172.21.15.98:0/637654914 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f70e4003a60 msgr2=0x7f70e4003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:17.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.237+0000 7f70effff700 1 --2- 172.21.15.98:0/637654914 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f70e4003a60 0x7f70e4003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:17.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.237+0000 7f70effff700 1 -- 172.21.15.98:0/637654914 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70e4003570 con 0x7f70e4004810 2023-10-06T05:08:17.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.237+0000 7f70effff700 1 --2- 172.21.15.98:0/637654914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70e4004810 0x7f70e4004c80 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f70e000c3f0 tx=0x7f70e0001e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a7e11b8ab85347e server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:17.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.237+0000 7f70f6ffd700 1 -- 172.21.15.98:0/637654914 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70e000cc80 con 0x7f70e4004810 2023-10-06T05:08:17.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.237+0000 7f70f6ffd700 1 -- 172.21.15.98:0/637654914 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70e000ce00 con 0x7f70e4004810 2023-10-06T05:08:17.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.237+0000 7f70f6ffd700 1 -- 172.21.15.98:0/637654914 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70e0005b70 con 0x7f70e4004810 2023-10-06T05:08:17.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.237+0000 7f70fcbb6700 1 -- 172.21.15.98:0/637654914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70e4004810 msgr2=0x7f70e4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:17.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.237+0000 7f70fcbb6700 1 --2- 172.21.15.98:0/637654914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70e4004810 0x7f70e4004c80 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f70e000c3f0 tx=0x7f70e0001e90 comp rx=0 tx=0).stop 2023-10-06T05:08:17.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.237+0000 7f70f7fff700 1 -- 172.21.15.98:0/637654914 reap_dead start 2023-10-06T05:08:17.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.237+0000 7f70fcbb6700 1 -- 172.21.15.98:0/637654914 shutdown_connections 2023-10-06T05:08:17.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.237+0000 7f70fcbb6700 1 -- 172.21.15.98:0/637654914 >> 172.21.15.98:0/637654914 conn(0x7f70f80680c0 msgr2=0x7f70f8068490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:17.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.238+0000 7f70fcbb6700 1 -- 172.21.15.98:0/637654914 shutdown_connections 2023-10-06T05:08:17.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.238+0000 7f70fcbb6700 1 -- 172.21.15.98:0/637654914 wait complete. 2023-10-06T05:08:17.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.239+0000 7f70fcbb6700 1 Processor -- start 2023-10-06T05:08:17.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.239+0000 7f70fcbb6700 1 -- start start 2023-10-06T05:08:17.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.239+0000 7f70fcbb6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70e4004810 0x7f70f80c9450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:17.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.239+0000 7f70fcbb6700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f70f8152330 0x7f70f8169c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:17.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.239+0000 7f70fcbb6700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f70f816a1c0 0x7f70f80c6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:17.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.239+0000 7f70fcbb6700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70f80c9990 con 0x7f70e4004810 2023-10-06T05:08:17.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.239+0000 7f70fcbb6700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70f80c81c0 con 0x7f70f816a1c0 2023-10-06T05:08:17.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.239+0000 7f70fcbb6700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70f80c8fc0 con 0x7f70f8152330 2023-10-06T05:08:17.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.240+0000 7f70f77fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70e4004810 0x7f70f80c9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:17.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.240+0000 7f70f77fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70e4004810 0x7f70f80c9450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49944/0 (socket says 172.21.15.98:49944) 2023-10-06T05:08:17.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.240+0000 7f70f77fe700 1 -- 172.21.15.98:0/2547265612 learned_addr learned my addr 172.21.15.98:0/2547265612 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:08:17.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.240+0000 7f70f7fff700 1 --2- 172.21.15.98:0/2547265612 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f70f816a1c0 0x7f70f80c6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:17.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.240+0000 7f70effff700 1 --2- 172.21.15.98:0/2547265612 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f70f8152330 0x7f70f8169c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:17.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.240+0000 7f70f77fe700 1 -- 172.21.15.98:0/2547265612 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f70f8152330 msgr2=0x7f70f8169c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:17.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.240+0000 7f70f77fe700 1 --2- 172.21.15.98:0/2547265612 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f70f8152330 0x7f70f8169c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:17.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.240+0000 7f70f77fe700 1 -- 172.21.15.98:0/2547265612 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f70f816a1c0 msgr2=0x7f70f80c6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:17.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.240+0000 7f70f77fe700 1 --2- 172.21.15.98:0/2547265612 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f70f816a1c0 0x7f70f80c6240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:17.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.240+0000 7f70f77fe700 1 -- 172.21.15.98:0/2547265612 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70f80c83f0 con 0x7f70e4004810 2023-10-06T05:08:17.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.240+0000 7f70effff700 1 --2- 172.21.15.98:0/2547265612 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f70f8152330 0x7f70f8169c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:08:17.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.240+0000 7f70f77fe700 1 --2- 172.21.15.98:0/2547265612 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70e4004810 0x7f70f80c9450 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f70e8011d40 tx=0x7f70e80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:17.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.240+0000 7f70ef7fe700 1 -- 172.21.15.98:0/2547265612 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70e800fa40 con 0x7f70e4004810 2023-10-06T05:08:17.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.240+0000 7f70ef7fe700 1 -- 172.21.15.98:0/2547265612 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70e800fba0 con 0x7f70e4004810 2023-10-06T05:08:17.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.240+0000 7f70fcbb6700 1 -- 172.21.15.98:0/2547265612 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70f80c8690 con 0x7f70e4004810 2023-10-06T05:08:17.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.240+0000 7f70ef7fe700 1 -- 172.21.15.98:0/2547265612 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70e8014440 con 0x7f70e4004810 2023-10-06T05:08:17.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.240+0000 7f70fcbb6700 1 -- 172.21.15.98:0/2547265612 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70f80c8c50 con 0x7f70e4004810 2023-10-06T05:08:17.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.241+0000 7f70fcbb6700 1 -- 172.21.15.98:0/2547265612 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f70f805f230 con 0x7f70e4004810 2023-10-06T05:08:17.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.241+0000 7f70ef7fe700 1 -- 172.21.15.98:0/2547265612 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f70e8014870 con 0x7f70e4004810 2023-10-06T05:08:17.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.242+0000 7f70ef7fe700 1 --2- 172.21.15.98:0/2547265612 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f70d8066f30 0x7f70d80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:17.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.242+0000 7f70ef7fe700 1 -- 172.21.15.98:0/2547265612 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5612+0+0 (secure 0 0 0) 0x7f70e8018030 con 0x7f70e4004810 2023-10-06T05:08:17.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.242+0000 7f70effff700 1 --2- 172.21.15.98:0/2547265612 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f70d8066f30 0x7f70d80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:17.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.242+0000 7f70ef7fe700 1 --2- 172.21.15.98:0/2547265612 >> [v2:172.21.15.98:6800/90627,v1:172.21.15.98:6801/90627] conn(0x7f70d806d850 0x7f70d806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:17.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.242+0000 7f70ef7fe700 1 -- 172.21.15.98:0/2547265612 --> [v2:172.21.15.98:6800/90627,v1:172.21.15.98:6801/90627] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f70d8070370 con 0x7f70d806d850 2023-10-06T05:08:17.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.242+0000 7f70ef7fe700 1 -- 172.21.15.98:0/2547265612 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=46) v2 ==== 24+0+0 (secure 0 0 0) 0x7f70e80872b0 con 0x7f70e4004810 2023-10-06T05:08:17.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.242+0000 7f70f7fff700 1 --2- 172.21.15.98:0/2547265612 >> [v2:172.21.15.98:6800/90627,v1:172.21.15.98:6801/90627] conn(0x7f70d806d850 0x7f70d806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:17.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.242+0000 7f70effff700 1 --2- 172.21.15.98:0/2547265612 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f70d8066f30 0x7f70d80693e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f70e0001ed0 tx=0x7f70e0015000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:17.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.242+0000 7f70f7fff700 1 --2- 172.21.15.98:0/2547265612 >> [v2:172.21.15.98:6800/90627,v1:172.21.15.98:6801/90627] conn(0x7f70d806d850 0x7f70d806fca0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:17.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.243+0000 7f70ef7fe700 1 -- 172.21.15.98:0/2547265612 <== osd.0 v2:172.21.15.98:6800/90627 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f70d8070370 con 0x7f70d806d850 2023-10-06T05:08:17.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.257+0000 7f70fcbb6700 1 -- 172.21.15.98:0/2547265612 --> [v2:172.21.15.98:6800/90627,v1:172.21.15.98:6801/90627] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f70f80cd2c0 con 0x7f70d806d850 2023-10-06T05:08:17.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.257+0000 7f70ef7fe700 1 -- 172.21.15.98:0/2547265612 <== osd.0 v2:172.21.15.98:6800/90627 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f70f80cd2c0 con 0x7f70d806d850 2023-10-06T05:08:17.259 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-10-06T05:08:17.259 INFO:teuthology.orchestra.run.smithi098.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-06T05:08:17.259 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-10-06T05:08:17.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.257+0000 7f70fcbb6700 1 -- 172.21.15.98:0/2547265612 >> [v2:172.21.15.98:6800/90627,v1:172.21.15.98:6801/90627] conn(0x7f70d806d850 msgr2=0x7f70d806fca0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:17.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.257+0000 7f70fcbb6700 1 --2- 172.21.15.98:0/2547265612 >> [v2:172.21.15.98:6800/90627,v1:172.21.15.98:6801/90627] conn(0x7f70d806d850 0x7f70d806fca0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:17.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.257+0000 7f70fcbb6700 1 -- 172.21.15.98:0/2547265612 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f70d8066f30 msgr2=0x7f70d80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:17.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.257+0000 7f70fcbb6700 1 --2- 172.21.15.98:0/2547265612 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f70d8066f30 0x7f70d80693e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f70e0001ed0 tx=0x7f70e0015000 comp rx=0 tx=0).stop 2023-10-06T05:08:17.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.257+0000 7f70fcbb6700 1 -- 172.21.15.98:0/2547265612 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70e4004810 msgr2=0x7f70f80c9450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:17.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.257+0000 7f70fcbb6700 1 --2- 172.21.15.98:0/2547265612 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70e4004810 0x7f70f80c9450 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f70e8011d40 tx=0x7f70e80091a0 comp rx=0 tx=0).stop 2023-10-06T05:08:17.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.257+0000 7f70f7fff700 1 -- 172.21.15.98:0/2547265612 reap_dead start 2023-10-06T05:08:17.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.257+0000 7f70fcbb6700 1 -- 172.21.15.98:0/2547265612 shutdown_connections 2023-10-06T05:08:17.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.257+0000 7f70fcbb6700 1 -- 172.21.15.98:0/2547265612 >> 172.21.15.98:0/2547265612 conn(0x7f70f80680c0 msgr2=0x7f70f80ce700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:17.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.257+0000 7f70fcbb6700 1 -- 172.21.15.98:0/2547265612 shutdown_connections 2023-10-06T05:08:17.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.257+0000 7f70fcbb6700 1 -- 172.21.15.98:0/2547265612 wait complete. 2023-10-06T05:08:17.271 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-10-06T05:08:17.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.384+0000 7fb4c43d4700 1 Processor -- start 2023-10-06T05:08:17.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.384+0000 7fb4c43d4700 1 -- start start 2023-10-06T05:08:17.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.385+0000 7fb4c43d4700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb4bc0b49a0 0x7fb4bc0aac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:17.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.385+0000 7fb4c43d4700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fb4bc0b0760 0x7fb4bc0b1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:17.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.385+0000 7fb4c43d4700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4bc0b0c60 con 0x7fb4bc159560 2023-10-06T05:08:17.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.385+0000 7fb4c43d4700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4bc0b0f40 con 0x7fb4bc0b49a0 2023-10-06T05:08:17.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.385+0000 7fb4c43d4700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4bc0b1220 con 0x7fb4bc0b0760 2023-10-06T05:08:17.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.385+0000 7fb4c196f700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb4bc0b49a0 0x7fb4bc0aac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:17.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.385+0000 7fb4c196f700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb4bc0b49a0 0x7fb4bc0aac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49954/0 (socket says 172.21.15.98:49954) 2023-10-06T05:08:17.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.385+0000 7fb4c196f700 1 -- 172.21.15.98:0/2951458228 learned_addr learned my addr 172.21.15.98:0/2951458228 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:08:17.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.386+0000 7fb4c2971700 1 --2- 172.21.15.98:0/2951458228 >> v2:172.21.15.174:3300/0 conn(0x7fb4bc0b0760 0x7fb4bc0b1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:17.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.386+0000 7fb4c196f700 1 -- 172.21.15.98:0/2951458228 >> v2:172.21.15.174:3300/0 conn(0x7fb4bc0b0760 msgr2=0x7fb4bc0b1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:17.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.386+0000 7fb4c196f700 1 --2- 172.21.15.98:0/2951458228 >> v2:172.21.15.174:3300/0 conn(0x7fb4bc0b0760 0x7fb4bc0b1bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:17.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.386+0000 7fb4c196f700 1 -- 172.21.15.98:0/2951458228 >> v1:172.21.15.174:6789/0 conn(0x7fb4bc159560 legacy=0x7fb4bc0aa530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:17.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.386+0000 7fb4c196f700 1 -- 172.21.15.98:0/2951458228 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4bc0b1500 con 0x7fb4bc0b49a0 2023-10-06T05:08:17.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.386+0000 7fb4c196f700 1 --2- 172.21.15.98:0/2951458228 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4bc0b49a0 0x7fb4bc0aac40 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac0099b0 tx=0x7fb4ac009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1563be7c3977a19 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:17.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.386+0000 7fb4c116e700 1 -- 172.21.15.98:0/2951458228 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4ac00b7e0 con 0x7fb4bc0b49a0 2023-10-06T05:08:17.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.386+0000 7fb4c116e700 1 -- 172.21.15.98:0/2951458228 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4ac00b940 con 0x7fb4bc0b49a0 2023-10-06T05:08:17.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.386+0000 7fb4c116e700 1 -- 172.21.15.98:0/2951458228 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4ac00bce0 con 0x7fb4bc0b49a0 2023-10-06T05:08:17.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.386+0000 7fb4c43d4700 1 -- 172.21.15.98:0/2951458228 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4bc0b49a0 msgr2=0x7fb4bc0aac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:17.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.386+0000 7fb4c43d4700 1 --2- 172.21.15.98:0/2951458228 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4bc0b49a0 0x7fb4bc0aac40 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac0099b0 tx=0x7fb4ac009250 comp rx=0 tx=0).stop 2023-10-06T05:08:17.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.386+0000 7fb4c43d4700 1 -- 172.21.15.98:0/2951458228 shutdown_connections 2023-10-06T05:08:17.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.386+0000 7fb4c43d4700 1 --2- 172.21.15.98:0/2951458228 >> v2:172.21.15.174:3300/0 conn(0x7fb4bc0b0760 0x7fb4bc0b1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:17.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.386+0000 7fb4c43d4700 1 --2- 172.21.15.98:0/2951458228 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4bc0b49a0 0x7fb4bc0aac40 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:17.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.386+0000 7fb4c43d4700 1 -- 172.21.15.98:0/2951458228 >> 172.21.15.98:0/2951458228 conn(0x7fb4bc0997f0 msgr2=0x7fb4bc06f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:17.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.387+0000 7fb4c43d4700 1 -- 172.21.15.98:0/2951458228 shutdown_connections 2023-10-06T05:08:17.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.387+0000 7fb4c43d4700 1 -- 172.21.15.98:0/2951458228 wait complete. 2023-10-06T05:08:17.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.388+0000 7fb4c43d4700 1 Processor -- start 2023-10-06T05:08:17.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.388+0000 7fb4c43d4700 1 -- start start 2023-10-06T05:08:17.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.388+0000 7fb4c43d4700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb4bc0b0760 0x7fb4bc157cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:17.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.388+0000 7fb4c43d4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4bc0b49a0 0x7fb4bc0ad9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:17.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.388+0000 7fb4c43d4700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb4bc159560 0x7fb4bc0adee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:17.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.388+0000 7fb4c43d4700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4bc1581f0 con 0x7fb4bc0b49a0 2023-10-06T05:08:17.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.388+0000 7fb4c43d4700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4bc0af170 con 0x7fb4bc0b0760 2023-10-06T05:08:17.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.388+0000 7fb4c43d4700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4bc0ae560 con 0x7fb4bc159560 2023-10-06T05:08:17.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.388+0000 7fb4c196f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4bc0b49a0 0x7fb4bc0ad9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:17.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.388+0000 7fb4c196f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4bc0b49a0 0x7fb4bc0ad9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49962/0 (socket says 172.21.15.98:49962) 2023-10-06T05:08:17.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.388+0000 7fb4c196f700 1 -- 172.21.15.98:0/3454306191 learned_addr learned my addr 172.21.15.98:0/3454306191 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:08:17.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.389+0000 7fb4c2971700 1 --2- 172.21.15.98:0/3454306191 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb4bc159560 0x7fb4bc0adee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:17.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.389+0000 7fb4c2170700 1 --2- 172.21.15.98:0/3454306191 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb4bc0b0760 0x7fb4bc157cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:17.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.389+0000 7fb4c196f700 1 -- 172.21.15.98:0/3454306191 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb4bc159560 msgr2=0x7fb4bc0adee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:17.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.389+0000 7fb4c196f700 1 --2- 172.21.15.98:0/3454306191 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb4bc159560 0x7fb4bc0adee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:17.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.389+0000 7fb4c196f700 1 -- 172.21.15.98:0/3454306191 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb4bc0b0760 msgr2=0x7fb4bc157cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:17.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.389+0000 7fb4c196f700 1 --2- 172.21.15.98:0/3454306191 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb4bc0b0760 0x7fb4bc157cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:17.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.389+0000 7fb4c196f700 1 -- 172.21.15.98:0/3454306191 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4ac0094f0 con 0x7fb4bc0b49a0 2023-10-06T05:08:17.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.389+0000 7fb4c196f700 1 --2- 172.21.15.98:0/3454306191 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4bc0b49a0 0x7fb4bc0ad9a0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac0030c0 tx=0x7fb4ac016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:17.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.389+0000 7fb4b27fc700 1 -- 172.21.15.98:0/3454306191 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4ac011d90 con 0x7fb4bc0b49a0 2023-10-06T05:08:17.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.389+0000 7fb4b27fc700 1 -- 172.21.15.98:0/3454306191 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4ac020070 con 0x7fb4bc0b49a0 2023-10-06T05:08:17.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.389+0000 7fb4c43d4700 1 -- 172.21.15.98:0/3454306191 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4bc0ae840 con 0x7fb4bc0b49a0 2023-10-06T05:08:17.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.389+0000 7fb4b27fc700 1 -- 172.21.15.98:0/3454306191 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4ac01b6a0 con 0x7fb4bc0b49a0 2023-10-06T05:08:17.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.389+0000 7fb4c43d4700 1 -- 172.21.15.98:0/3454306191 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4bc0aee00 con 0x7fb4bc0b49a0 2023-10-06T05:08:17.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.390+0000 7fb4c43d4700 1 -- 172.21.15.98:0/3454306191 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4bc064a90 con 0x7fb4bc0b49a0 2023-10-06T05:08:17.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.390+0000 7fb4b27fc700 1 -- 172.21.15.98:0/3454306191 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fb4ac022030 con 0x7fb4bc0b49a0 2023-10-06T05:08:17.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.390+0000 7fb4b27fc700 1 --2- 172.21.15.98:0/3454306191 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fb4a0066f30 0x7fb4a00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:17.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.391+0000 7fb4b27fc700 1 -- 172.21.15.98:0/3454306191 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5612+0+0 (secure 0 0 0) 0x7fb4ac086470 con 0x7fb4bc0b49a0 2023-10-06T05:08:17.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.391+0000 7fb4c2170700 1 --2- 172.21.15.98:0/3454306191 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fb4a0066f30 0x7fb4a00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:17.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.391+0000 7fb4c2170700 1 --2- 172.21.15.98:0/3454306191 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fb4a0066f30 0x7fb4a00693e0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fb4b8009fd0 tx=0x7fb4b8009600 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:17.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.395+0000 7fb4b27fc700 1 -- 172.21.15.98:0/3454306191 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb4ac053740 con 0x7fb4bc0b49a0 2023-10-06T05:08:17.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.576+0000 7fb4c43d4700 1 -- 172.21.15.98:0/3454306191 --> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7fb4bc064a90 con 0x7fb4a0066f30 2023-10-06T05:08:17.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.576+0000 7fb4b27fc700 1 -- 172.21.15.98:0/3454306191 <== mgr.4104 v2:172.21.15.98:6832/90397 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fb4bc064a90 con 0x7fb4a0066f30 2023-10-06T05:08:17.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.578+0000 7fb4c43d4700 1 -- 172.21.15.98:0/3454306191 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fb4a0066f30 msgr2=0x7fb4a00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:17.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.578+0000 7fb4c43d4700 1 --2- 172.21.15.98:0/3454306191 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fb4a0066f30 0x7fb4a00693e0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fb4b8009fd0 tx=0x7fb4b8009600 comp rx=0 tx=0).stop 2023-10-06T05:08:17.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.578+0000 7fb4c43d4700 1 -- 172.21.15.98:0/3454306191 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4bc0b49a0 msgr2=0x7fb4bc0ad9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:17.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.578+0000 7fb4c43d4700 1 --2- 172.21.15.98:0/3454306191 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4bc0b49a0 0x7fb4bc0ad9a0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac0030c0 tx=0x7fb4ac016000 comp rx=0 tx=0).stop 2023-10-06T05:08:17.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.578+0000 7fb4c43d4700 1 -- 172.21.15.98:0/3454306191 shutdown_connections 2023-10-06T05:08:17.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.578+0000 7fb4c43d4700 1 --2- 172.21.15.98:0/3454306191 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb4bc159560 0x7fb4bc0adee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:17.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.578+0000 7fb4c43d4700 1 --2- 172.21.15.98:0/3454306191 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb4bc0b0760 0x7fb4bc157cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:17.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.578+0000 7fb4c43d4700 1 --2- 172.21.15.98:0/3454306191 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fb4a0066f30 0x7fb4a00693e0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:17.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.578+0000 7fb4c43d4700 1 --2- 172.21.15.98:0/3454306191 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4bc0b49a0 0x7fb4bc0ad9a0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:17.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.578+0000 7fb4c43d4700 1 -- 172.21.15.98:0/3454306191 >> 172.21.15.98:0/3454306191 conn(0x7fb4bc0997f0 msgr2=0x7fb4bc069480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:17.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.578+0000 7fb4c43d4700 1 -- 172.21.15.98:0/3454306191 shutdown_connections 2023-10-06T05:08:17.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.578+0000 7fb4c43d4700 1 -- 172.21.15.98:0/3454306191 wait complete. 2023-10-06T05:08:17.583 INFO:teuthology.orchestra.run.smithi098.stderr:instructed osd(s) 0 to deep-scrub 2023-10-06T05:08:17.592 INFO:tasks.ceph:Scrubbing osd.1 2023-10-06T05:08:17.592 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-10-06T05:08:17.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.708+0000 7fcd06f6b700 1 Processor -- start 2023-10-06T05:08:17.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.708+0000 7fcd06f6b700 1 -- start start 2023-10-06T05:08:17.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.709+0000 7fcd06f6b700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcd0015a740 0x7fcd000aede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:17.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.709+0000 7fcd06f6b700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd000b2ce0 con 0x7fcd0006ece0 2023-10-06T05:08:17.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.709+0000 7fcd06f6b700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd000b0410 con 0x7fcd000b2360 2023-10-06T05:08:17.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.709+0000 7fcd06f6b700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd0006fb40 con 0x7fcd0015a740 2023-10-06T05:08:17.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.709+0000 7fcd04d07700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fcd0006ece0 0x7fcd0006f0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:47096/0 (socket says 172.21.15.98:47096) 2023-10-06T05:08:17.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.709+0000 7fcd04d07700 1 -- 172.21.15.98:0/158611273 learned_addr learned my addr 172.21.15.98:0/158611273 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:08:17.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.709+0000 7fcd05508700 1 --2- 172.21.15.98:0/158611273 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcd0015a740 0x7fcd000aede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:17.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.709+0000 7fccf77fe700 1 -- 172.21.15.98:0/158611273 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1196892427 0 0) 0x7fcd000b2ce0 con 0x7fcd0006ece0 2023-10-06T05:08:17.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.710+0000 7fccf77fe700 1 -- 172.21.15.98:0/158611273 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcce8003570 con 0x7fcd0006ece0 2023-10-06T05:08:17.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.710+0000 7fccf77fe700 1 -- 172.21.15.98:0/158611273 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3685016999 0 0) 0x7fcce8003570 con 0x7fcd0006ece0 2023-10-06T05:08:17.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.710+0000 7fccf77fe700 1 -- 172.21.15.98:0/158611273 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcd0015a740 msgr2=0x7fcd000aede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:17.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.710+0000 7fccf77fe700 1 --2- 172.21.15.98:0/158611273 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcd0015a740 0x7fcd000aede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:17.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.710+0000 7fccf77fe700 1 -- 172.21.15.98:0/158611273 >> v1:172.21.15.174:6789/0 conn(0x7fcd000b2360 legacy=0x7fcd000b08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:17.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.710+0000 7fccf77fe700 1 -- 172.21.15.98:0/158611273 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd000b2a00 con 0x7fcd0006ece0 2023-10-06T05:08:17.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.710+0000 7fccf77fe700 1 -- 172.21.15.98:0/158611273 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1543991876 0 0) 0x7fccfc001470 con 0x7fcd0006ece0 2023-10-06T05:08:17.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.710+0000 7fccf77fe700 1 -- 172.21.15.98:0/158611273 >> v1:172.21.15.98:6789/0 conn(0x7fcd0006ece0 legacy=0x7fcd0006f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:17.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.710+0000 7fccf77fe700 1 --2- 172.21.15.98:0/158611273 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcce8003a90 0x7fcce8003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:17.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.710+0000 7fccf77fe700 1 --2- 172.21.15.98:0/158611273 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcd0015a740 0x7fcce8004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:17.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.710+0000 7fccf77fe700 1 --2- 172.21.15.98:0/158611273 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcce8004d50 0x7fcce80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:17.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.710+0000 7fccf77fe700 1 -- 172.21.15.98:0/158611273 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcce800bb10 con 0x7fcce8003a90 2023-10-06T05:08:17.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.710+0000 7fccf77fe700 1 -- 172.21.15.98:0/158611273 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcce800bc50 con 0x7fcce8004d50 2023-10-06T05:08:17.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.710+0000 7fccf77fe700 1 -- 172.21.15.98:0/158611273 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcce800be10 con 0x7fcd0015a740 2023-10-06T05:08:17.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.710+0000 7fcd04d07700 1 --2- 172.21.15.98:0/158611273 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcce8003a90 0x7fcce8003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:17.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.710+0000 7fccf7fff700 1 --2- 172.21.15.98:0/158611273 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcd0015a740 0x7fcce8004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:17.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.710+0000 7fcd05508700 1 --2- 172.21.15.98:0/158611273 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcce8004d50 0x7fcce80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:17.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.710+0000 7fcd04d07700 1 -- 172.21.15.98:0/158611273 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcd0015a740 msgr2=0x7fcce8004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:17.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.710+0000 7fcd04d07700 1 --2- 172.21.15.98:0/158611273 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcd0015a740 0x7fcce8004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:17.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.710+0000 7fcd04d07700 1 -- 172.21.15.98:0/158611273 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcce8004d50 msgr2=0x7fcce80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:17.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.710+0000 7fcd04d07700 1 --2- 172.21.15.98:0/158611273 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcce8004d50 0x7fcce80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:17.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.710+0000 7fcd04d07700 1 -- 172.21.15.98:0/158611273 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcce8003570 con 0x7fcce8003a90 2023-10-06T05:08:17.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.710+0000 7fccf7fff700 1 --2- 172.21.15.98:0/158611273 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcd0015a740 0x7fcce8004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:08:17.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.710+0000 7fcd04d07700 1 --2- 172.21.15.98:0/158611273 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcce8003a90 0x7fcce8003e60 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fccfc00c3c0 tx=0x7fccfc001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f68075f2fda1d1a server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:17.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.710+0000 7fccf77fe700 1 -- 172.21.15.98:0/158611273 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccfc00cc90 con 0x7fcce8003a90 2023-10-06T05:08:17.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.710+0000 7fccf77fe700 1 -- 172.21.15.98:0/158611273 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccfc00ce10 con 0x7fcce8003a90 2023-10-06T05:08:17.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.710+0000 7fccf77fe700 1 -- 172.21.15.98:0/158611273 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccfc005b70 con 0x7fcce8003a90 2023-10-06T05:08:17.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.711+0000 7fcd06f6b700 1 -- 172.21.15.98:0/158611273 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcce8003a90 msgr2=0x7fcce8003e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:17.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.711+0000 7fcd06f6b700 1 --2- 172.21.15.98:0/158611273 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcce8003a90 0x7fcce8003e60 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fccfc00c3c0 tx=0x7fccfc001e70 comp rx=0 tx=0).stop 2023-10-06T05:08:17.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.711+0000 7fcd05508700 1 -- 172.21.15.98:0/158611273 reap_dead start 2023-10-06T05:08:17.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.711+0000 7fcd06f6b700 1 -- 172.21.15.98:0/158611273 shutdown_connections 2023-10-06T05:08:17.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.711+0000 7fcd06f6b700 1 -- 172.21.15.98:0/158611273 >> 172.21.15.98:0/158611273 conn(0x7fcd000682f0 msgr2=0x7fcd000686f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:17.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.711+0000 7fcd06f6b700 1 -- 172.21.15.98:0/158611273 shutdown_connections 2023-10-06T05:08:17.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.711+0000 7fcd06f6b700 1 -- 172.21.15.98:0/158611273 wait complete. 2023-10-06T05:08:17.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.712+0000 7fcd06f6b700 1 Processor -- start 2023-10-06T05:08:17.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.712+0000 7fcd06f6b700 1 -- start start 2023-10-06T05:08:17.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.713+0000 7fcd06f6b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcce8003a90 0x7fcd000d0350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:17.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.713+0000 7fcd06f6b700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcd0016a680 0x7fcd000a8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:17.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.713+0000 7fcd06f6b700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcd000a8f60 0x7fcd00168fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:17.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.713+0000 7fcd06f6b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd000d0f80 con 0x7fcce8003a90 2023-10-06T05:08:17.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.713+0000 7fcd06f6b700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd000a93d0 con 0x7fcd0016a680 2023-10-06T05:08:17.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.713+0000 7fcd06f6b700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd00169be0 con 0x7fcd000a8f60 2023-10-06T05:08:17.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.713+0000 7fcd04d07700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcce8003a90 0x7fcd000d0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:17.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.713+0000 7fcd04d07700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcce8003a90 0x7fcd000d0350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49982/0 (socket says 172.21.15.98:49982) 2023-10-06T05:08:17.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.713+0000 7fcd04d07700 1 -- 172.21.15.98:0/2755917418 learned_addr learned my addr 172.21.15.98:0/2755917418 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:08:17.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.713+0000 7fccf7fff700 1 --2- 172.21.15.98:0/2755917418 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcd0016a680 0x7fcd000a8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:17.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.713+0000 7fcd05508700 1 --2- 172.21.15.98:0/2755917418 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcd000a8f60 0x7fcd00168fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:17.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.713+0000 7fcd04d07700 1 -- 172.21.15.98:0/2755917418 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcd000a8f60 msgr2=0x7fcd00168fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:17.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.713+0000 7fcd04d07700 1 --2- 172.21.15.98:0/2755917418 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcd000a8f60 0x7fcd00168fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:17.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.713+0000 7fcd04d07700 1 -- 172.21.15.98:0/2755917418 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcd0016a680 msgr2=0x7fcd000a8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:17.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.713+0000 7fcd04d07700 1 --2- 172.21.15.98:0/2755917418 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcd0016a680 0x7fcd000a8a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:17.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.713+0000 7fcd04d07700 1 -- 172.21.15.98:0/2755917418 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd000d0ca0 con 0x7fcce8003a90 2023-10-06T05:08:17.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.713+0000 7fccf7fff700 1 --2- 172.21.15.98:0/2755917418 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcd0016a680 0x7fcd000a8a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:08:17.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.713+0000 7fcd05508700 1 --2- 172.21.15.98:0/2755917418 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcd000a8f60 0x7fcd00168fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:08:17.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.713+0000 7fcd04d07700 1 --2- 172.21.15.98:0/2755917418 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcce8003a90 0x7fcd000d0350 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fccfc00c9b0 tx=0x7fccfc015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:17.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.714+0000 7fccf4ff9700 1 -- 172.21.15.98:0/2755917418 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccfc00cc90 con 0x7fcce8003a90 2023-10-06T05:08:17.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.714+0000 7fccf4ff9700 1 -- 172.21.15.98:0/2755917418 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccfc001b90 con 0x7fcce8003a90 2023-10-06T05:08:17.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.714+0000 7fcd06f6b700 1 -- 172.21.15.98:0/2755917418 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd00169e80 con 0x7fcce8003a90 2023-10-06T05:08:17.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.714+0000 7fccf4ff9700 1 -- 172.21.15.98:0/2755917418 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccfc007530 con 0x7fcce8003a90 2023-10-06T05:08:17.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.714+0000 7fcd06f6b700 1 -- 172.21.15.98:0/2755917418 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd000d0890 con 0x7fcce8003a90 2023-10-06T05:08:17.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.714+0000 7fcd06f6b700 1 -- 172.21.15.98:0/2755917418 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fcd00157c90 con 0x7fcce8003a90 2023-10-06T05:08:17.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.715+0000 7fccf4ff9700 1 -- 172.21.15.98:0/2755917418 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fccfc019030 con 0x7fcce8003a90 2023-10-06T05:08:17.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.715+0000 7fccf4ff9700 1 --2- 172.21.15.98:0/2755917418 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fccec066f30 0x7fccec0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:17.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.715+0000 7fccf4ff9700 1 -- 172.21.15.98:0/2755917418 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5612+0+0 (secure 0 0 0) 0x7fccfc085ff0 con 0x7fcce8003a90 2023-10-06T05:08:17.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.715+0000 7fccf7fff700 1 --2- 172.21.15.98:0/2755917418 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fccec066f30 0x7fccec0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:17.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.715+0000 7fccf4ff9700 1 --2- 172.21.15.98:0/2755917418 >> [v2:172.21.15.98:6808/90628,v1:172.21.15.98:6809/90628] conn(0x7fccec06d850 0x7fccec06fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:17.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.715+0000 7fccf4ff9700 1 -- 172.21.15.98:0/2755917418 --> [v2:172.21.15.98:6808/90628,v1:172.21.15.98:6809/90628] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fccec070370 con 0x7fccec06d850 2023-10-06T05:08:17.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.716+0000 7fccf4ff9700 1 -- 172.21.15.98:0/2755917418 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=46) v2 ==== 24+0+0 (secure 0 0 0) 0x7fccfc088550 con 0x7fcce8003a90 2023-10-06T05:08:17.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.716+0000 7fcd05508700 1 --2- 172.21.15.98:0/2755917418 >> [v2:172.21.15.98:6808/90628,v1:172.21.15.98:6809/90628] conn(0x7fccec06d850 0x7fccec06fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:17.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.716+0000 7fccf7fff700 1 --2- 172.21.15.98:0/2755917418 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fccec066f30 0x7fccec0693e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fccf0007a60 tx=0x7fccf0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:17.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.716+0000 7fcd05508700 1 --2- 172.21.15.98:0/2755917418 >> [v2:172.21.15.98:6808/90628,v1:172.21.15.98:6809/90628] conn(0x7fccec06d850 0x7fccec06fca0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:17.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.716+0000 7fccf4ff9700 1 -- 172.21.15.98:0/2755917418 <== osd.1 v2:172.21.15.98:6808/90628 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7fccec070370 con 0x7fccec06d850 2023-10-06T05:08:17.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.730+0000 7fcd06f6b700 1 -- 172.21.15.98:0/2755917418 --> [v2:172.21.15.98:6808/90628,v1:172.21.15.98:6809/90628] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fcd00064a90 con 0x7fccec06d850 2023-10-06T05:08:17.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.730+0000 7fccf4ff9700 1 -- 172.21.15.98:0/2755917418 <== osd.1 v2:172.21.15.98:6808/90628 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fcd00064a90 con 0x7fccec06d850 2023-10-06T05:08:17.732 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-10-06T05:08:17.732 INFO:teuthology.orchestra.run.smithi098.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-06T05:08:17.732 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-10-06T05:08:17.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.730+0000 7fcd06f6b700 1 -- 172.21.15.98:0/2755917418 >> [v2:172.21.15.98:6808/90628,v1:172.21.15.98:6809/90628] conn(0x7fccec06d850 msgr2=0x7fccec06fca0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:17.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.730+0000 7fcd06f6b700 1 --2- 172.21.15.98:0/2755917418 >> [v2:172.21.15.98:6808/90628,v1:172.21.15.98:6809/90628] conn(0x7fccec06d850 0x7fccec06fca0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:17.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.730+0000 7fcd06f6b700 1 -- 172.21.15.98:0/2755917418 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fccec066f30 msgr2=0x7fccec0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:17.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.730+0000 7fcd06f6b700 1 --2- 172.21.15.98:0/2755917418 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fccec066f30 0x7fccec0693e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fccf0007a60 tx=0x7fccf0009040 comp rx=0 tx=0).stop 2023-10-06T05:08:17.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.730+0000 7fcd06f6b700 1 -- 172.21.15.98:0/2755917418 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcce8003a90 msgr2=0x7fcd000d0350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:17.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.730+0000 7fcd06f6b700 1 --2- 172.21.15.98:0/2755917418 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcce8003a90 0x7fcd000d0350 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fccfc00c9b0 tx=0x7fccfc015000 comp rx=0 tx=0).stop 2023-10-06T05:08:17.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.730+0000 7fcd05508700 1 -- 172.21.15.98:0/2755917418 reap_dead start 2023-10-06T05:08:17.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.730+0000 7fcd06f6b700 1 -- 172.21.15.98:0/2755917418 shutdown_connections 2023-10-06T05:08:17.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.730+0000 7fcd06f6b700 1 -- 172.21.15.98:0/2755917418 >> 172.21.15.98:0/2755917418 conn(0x7fcd000682f0 msgr2=0x7fcd00157e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:17.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.731+0000 7fcd06f6b700 1 -- 172.21.15.98:0/2755917418 shutdown_connections 2023-10-06T05:08:17.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.731+0000 7fcd06f6b700 1 -- 172.21.15.98:0/2755917418 wait complete. 2023-10-06T05:08:17.744 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-10-06T05:08:17.864 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.861+0000 7f20154e9700 1 Processor -- start 2023-10-06T05:08:17.864 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.861+0000 7f20154e9700 1 -- start start 2023-10-06T05:08:17.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.862+0000 7f20154e9700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2010157040 0x7f20100cde60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:17.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.862+0000 7f20154e9700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2010157410 0x7f201006d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:17.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.862+0000 7f20154e9700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f201015b1d0 con 0x7f201015b660 2023-10-06T05:08:17.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.862+0000 7f20154e9700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f201006dae0 con 0x7f2010157040 2023-10-06T05:08:17.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.862+0000 7f20154e9700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20100ce4b0 con 0x7f2010157410 2023-10-06T05:08:17.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.862+0000 7f200ffff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2010157040 0x7f20100cde60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:17.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.862+0000 7f200ffff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2010157040 0x7f20100cde60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49992/0 (socket says 172.21.15.98:49992) 2023-10-06T05:08:17.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.862+0000 7f200ffff700 1 -- 172.21.15.98:0/3178755006 learned_addr learned my addr 172.21.15.98:0/3178755006 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:08:17.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.862+0000 7f200f7fe700 1 --2- 172.21.15.98:0/3178755006 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2010157410 0x7f201006d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:17.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.862+0000 7f200ffff700 1 -- 172.21.15.98:0/3178755006 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2010157410 msgr2=0x7f201006d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:17.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.862+0000 7f200ffff700 1 --2- 172.21.15.98:0/3178755006 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2010157410 0x7f201006d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:17.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.863+0000 7f200ffff700 1 -- 172.21.15.98:0/3178755006 >> v1:172.21.15.174:6789/0 conn(0x7f201015b660 legacy=0x7f20100c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:17.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.863+0000 7f200ffff700 1 -- 172.21.15.98:0/3178755006 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20100ce790 con 0x7f2010157040 2023-10-06T05:08:17.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.863+0000 7f200ffff700 1 --2- 172.21.15.98:0/3178755006 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2010157040 0x7f20100cde60 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f20000099b0 tx=0x7f2000009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9eb8c943f06a6802 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:17.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.863+0000 7f200effd700 1 -- 172.21.15.98:0/3178755006 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f200000b860 con 0x7f2010157040 2023-10-06T05:08:17.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.863+0000 7f200effd700 1 -- 172.21.15.98:0/3178755006 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f200000b9c0 con 0x7f2010157040 2023-10-06T05:08:17.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.863+0000 7f200effd700 1 -- 172.21.15.98:0/3178755006 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f200000bd60 con 0x7f2010157040 2023-10-06T05:08:17.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.863+0000 7f20154e9700 1 -- 172.21.15.98:0/3178755006 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2010157040 msgr2=0x7f20100cde60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:17.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.863+0000 7f20154e9700 1 --2- 172.21.15.98:0/3178755006 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2010157040 0x7f20100cde60 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f20000099b0 tx=0x7f2000009250 comp rx=0 tx=0).stop 2023-10-06T05:08:17.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.863+0000 7f20154e9700 1 -- 172.21.15.98:0/3178755006 shutdown_connections 2023-10-06T05:08:17.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.863+0000 7f20154e9700 1 --2- 172.21.15.98:0/3178755006 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2010157040 0x7f20100cde60 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:17.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.863+0000 7f20154e9700 1 --2- 172.21.15.98:0/3178755006 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2010157410 0x7f201006d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:17.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.863+0000 7f20154e9700 1 -- 172.21.15.98:0/3178755006 >> 172.21.15.98:0/3178755006 conn(0x7f201006b500 msgr2=0x7f201006b900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:17.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.863+0000 7f20154e9700 1 -- 172.21.15.98:0/3178755006 shutdown_connections 2023-10-06T05:08:17.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.863+0000 7f20154e9700 1 -- 172.21.15.98:0/3178755006 wait complete. 2023-10-06T05:08:17.870 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.864+0000 7f20154e9700 1 Processor -- start 2023-10-06T05:08:17.871 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.865+0000 7f20154e9700 1 -- start start 2023-10-06T05:08:17.871 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.865+0000 7f20154e9700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2010157040 0x7f20101599e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:17.871 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.865+0000 7f20154e9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2010157410 0x7f2010159f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:17.871 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.865+0000 7f20154e9700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f201015b660 0x7f20101679c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:17.871 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.865+0000 7f20154e9700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2010150730 con 0x7f2010157410 2023-10-06T05:08:17.871 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.865+0000 7f20154e9700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2010153460 con 0x7f2010157040 2023-10-06T05:08:17.871 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.865+0000 7f20154e9700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2010168720 con 0x7f201015b660 2023-10-06T05:08:17.871 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.865+0000 7f200f7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2010157410 0x7f2010159f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:17.871 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.865+0000 7f200f7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2010157410 0x7f2010159f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50000/0 (socket says 172.21.15.98:50000) 2023-10-06T05:08:17.871 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.866+0000 7f200f7fe700 1 -- 172.21.15.98:0/3098522999 learned_addr learned my addr 172.21.15.98:0/3098522999 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:08:17.871 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.866+0000 7f200ffff700 1 --2- 172.21.15.98:0/3098522999 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2010157040 0x7f20101599e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:17.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.866+0000 7f2014ce8700 1 --2- 172.21.15.98:0/3098522999 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f201015b660 0x7f20101679c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:17.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.866+0000 7f200f7fe700 1 -- 172.21.15.98:0/3098522999 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f201015b660 msgr2=0x7f20101679c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:17.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.866+0000 7f200f7fe700 1 --2- 172.21.15.98:0/3098522999 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f201015b660 0x7f20101679c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:17.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.866+0000 7f200f7fe700 1 -- 172.21.15.98:0/3098522999 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2010157040 msgr2=0x7f20101599e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:17.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.866+0000 7f200f7fe700 1 --2- 172.21.15.98:0/3098522999 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2010157040 0x7f20101599e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:17.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.866+0000 7f200f7fe700 1 -- 172.21.15.98:0/3098522999 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20000094f0 con 0x7f2010157410 2023-10-06T05:08:17.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.866+0000 7f200f7fe700 1 --2- 172.21.15.98:0/3098522999 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2010157410 0x7f2010159f20 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f200400d890 tx=0x7f2004008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:17.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.866+0000 7f1ff7fff700 1 -- 172.21.15.98:0/3098522999 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20040125f0 con 0x7f2010157410 2023-10-06T05:08:17.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.866+0000 7f1ff7fff700 1 -- 172.21.15.98:0/3098522999 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2004012770 con 0x7f2010157410 2023-10-06T05:08:17.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.866+0000 7f20154e9700 1 -- 172.21.15.98:0/3098522999 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2010152b30 con 0x7f2010157410 2023-10-06T05:08:17.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.866+0000 7f1ff7fff700 1 -- 172.21.15.98:0/3098522999 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2004012b30 con 0x7f2010157410 2023-10-06T05:08:17.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.866+0000 7f20154e9700 1 -- 172.21.15.98:0/3098522999 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20101530f0 con 0x7f2010157410 2023-10-06T05:08:17.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.867+0000 7f20154e9700 1 -- 172.21.15.98:0/3098522999 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f201005f230 con 0x7f2010157410 2023-10-06T05:08:17.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.867+0000 7f1ff7fff700 1 -- 172.21.15.98:0/3098522999 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f200401e460 con 0x7f2010157410 2023-10-06T05:08:17.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.868+0000 7f1ff7fff700 1 --2- 172.21.15.98:0/3098522999 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f1ffc066f30 0x7f1ffc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:17.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.868+0000 7f1ff7fff700 1 -- 172.21.15.98:0/3098522999 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5612+0+0 (secure 0 0 0) 0x7f2004084aa0 con 0x7f2010157410 2023-10-06T05:08:17.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.868+0000 7f200ffff700 1 --2- 172.21.15.98:0/3098522999 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f1ffc066f30 0x7f1ffc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:17.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.868+0000 7f200ffff700 1 --2- 172.21.15.98:0/3098522999 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f1ffc066f30 0x7f1ffc0693e0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f20000099b0 tx=0x7f2000017000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:17.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:17.872+0000 7f1ff7fff700 1 -- 172.21.15.98:0/3098522999 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2004051d70 con 0x7f2010157410 2023-10-06T05:08:18.058 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.056+0000 7f20154e9700 1 -- 172.21.15.98:0/3098522999 --> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f201015ae60 con 0x7f1ffc066f30 2023-10-06T05:08:18.058 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.056+0000 7f1ff7fff700 1 -- 172.21.15.98:0/3098522999 <== mgr.4104 v2:172.21.15.98:6832/90397 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f201015ae60 con 0x7f1ffc066f30 2023-10-06T05:08:18.061 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.058+0000 7f20154e9700 1 -- 172.21.15.98:0/3098522999 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f1ffc066f30 msgr2=0x7f1ffc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:18.061 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.058+0000 7f20154e9700 1 --2- 172.21.15.98:0/3098522999 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f1ffc066f30 0x7f1ffc0693e0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f20000099b0 tx=0x7f2000017000 comp rx=0 tx=0).stop 2023-10-06T05:08:18.061 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.058+0000 7f20154e9700 1 -- 172.21.15.98:0/3098522999 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2010157410 msgr2=0x7f2010159f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:18.061 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.058+0000 7f20154e9700 1 --2- 172.21.15.98:0/3098522999 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2010157410 0x7f2010159f20 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f200400d890 tx=0x7f2004008040 comp rx=0 tx=0).stop 2023-10-06T05:08:18.061 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.059+0000 7f20154e9700 1 -- 172.21.15.98:0/3098522999 shutdown_connections 2023-10-06T05:08:18.061 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.059+0000 7f20154e9700 1 --2- 172.21.15.98:0/3098522999 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f201015b660 0x7f20101679c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:18.061 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.059+0000 7f20154e9700 1 --2- 172.21.15.98:0/3098522999 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2010157040 0x7f20101599e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:18.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.059+0000 7f20154e9700 1 --2- 172.21.15.98:0/3098522999 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f1ffc066f30 0x7f1ffc0693e0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:18.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.059+0000 7f20154e9700 1 --2- 172.21.15.98:0/3098522999 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2010157410 0x7f2010159f20 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:18.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.059+0000 7f20154e9700 1 -- 172.21.15.98:0/3098522999 >> 172.21.15.98:0/3098522999 conn(0x7f201006b500 msgr2=0x7f20100adab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:18.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.059+0000 7f20154e9700 1 -- 172.21.15.98:0/3098522999 shutdown_connections 2023-10-06T05:08:18.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.059+0000 7f20154e9700 1 -- 172.21.15.98:0/3098522999 wait complete. 2023-10-06T05:08:18.062 INFO:teuthology.orchestra.run.smithi098.stderr:instructed osd(s) 1 to deep-scrub 2023-10-06T05:08:18.072 INFO:tasks.ceph:Scrubbing osd.2 2023-10-06T05:08:18.073 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-10-06T05:08:18.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.189+0000 7f7992b9b700 1 Processor -- start 2023-10-06T05:08:18.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.189+0000 7f7992b9b700 1 -- start start 2023-10-06T05:08:18.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.189+0000 7f7992b9b700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f798c15a740 0x7f798c0aede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:18.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.189+0000 7f7992b9b700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f798c0b2ce0 con 0x7f798c06ece0 2023-10-06T05:08:18.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.189+0000 7f7992b9b700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f798c0b0410 con 0x7f798c0b2360 2023-10-06T05:08:18.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.189+0000 7f7992b9b700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f798c06fb40 con 0x7f798c15a740 2023-10-06T05:08:18.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.189+0000 7f7990937700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f798c06ece0 0x7f798c06f0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:47112/0 (socket says 172.21.15.98:47112) 2023-10-06T05:08:18.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.189+0000 7f7990937700 1 -- 172.21.15.98:0/3169756618 learned_addr learned my addr 172.21.15.98:0/3169756618 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:08:18.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.190+0000 7f798b7fe700 1 -- 172.21.15.98:0/3169756618 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2061526542 0 0) 0x7f798c0b2ce0 con 0x7f798c06ece0 2023-10-06T05:08:18.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.190+0000 7f7991138700 1 --2- 172.21.15.98:0/3169756618 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f798c15a740 0x7f798c0aede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:18.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.190+0000 7f798b7fe700 1 -- 172.21.15.98:0/3169756618 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7974003570 con 0x7f798c06ece0 2023-10-06T05:08:18.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.190+0000 7f798b7fe700 1 -- 172.21.15.98:0/3169756618 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 454548401 0 0) 0x7f7974003570 con 0x7f798c06ece0 2023-10-06T05:08:18.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.190+0000 7f798b7fe700 1 -- 172.21.15.98:0/3169756618 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f798c15a740 msgr2=0x7f798c0aede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:18.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.190+0000 7f798b7fe700 1 --2- 172.21.15.98:0/3169756618 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f798c15a740 0x7f798c0aede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:18.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.190+0000 7f798b7fe700 1 -- 172.21.15.98:0/3169756618 >> v1:172.21.15.174:6789/0 conn(0x7f798c0b2360 legacy=0x7f798c0b08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:18.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.190+0000 7f798b7fe700 1 -- 172.21.15.98:0/3169756618 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f798c0b2a00 con 0x7f798c06ece0 2023-10-06T05:08:18.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.190+0000 7f798b7fe700 1 -- 172.21.15.98:0/3169756618 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1543991876 0 0) 0x7f797c0017f0 con 0x7f798c06ece0 2023-10-06T05:08:18.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.190+0000 7f798b7fe700 1 -- 172.21.15.98:0/3169756618 >> v1:172.21.15.98:6789/0 conn(0x7f798c06ece0 legacy=0x7f798c06f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:18.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.190+0000 7f798b7fe700 1 --2- 172.21.15.98:0/3169756618 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f798c15a740 0x7f7974003a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:18.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.190+0000 7f798b7fe700 1 --2- 172.21.15.98:0/3169756618 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7974004010 0x7f7974004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:18.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.190+0000 7f798b7fe700 1 --2- 172.21.15.98:0/3169756618 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7974004d50 0x7f79740051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:18.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.190+0000 7f798b7fe700 1 -- 172.21.15.98:0/3169756618 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f797400bbf0 con 0x7f7974004d50 2023-10-06T05:08:18.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.190+0000 7f798b7fe700 1 -- 172.21.15.98:0/3169756618 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f797400bd30 con 0x7f7974004010 2023-10-06T05:08:18.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.190+0000 7f798b7fe700 1 -- 172.21.15.98:0/3169756618 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f797400bef0 con 0x7f798c15a740 2023-10-06T05:08:18.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.190+0000 7f7991138700 1 --2- 172.21.15.98:0/3169756618 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7974004d50 0x7f79740051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:18.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.190+0000 7f7990937700 1 --2- 172.21.15.98:0/3169756618 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f798c15a740 0x7f7974003a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:18.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.190+0000 7f798bfff700 1 --2- 172.21.15.98:0/3169756618 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7974004010 0x7f7974004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:18.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.190+0000 7f7991138700 1 -- 172.21.15.98:0/3169756618 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f798c15a740 msgr2=0x7f7974003a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:18.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.190+0000 7f7991138700 1 --2- 172.21.15.98:0/3169756618 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f798c15a740 0x7f7974003a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:18.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.190+0000 7f7991138700 1 -- 172.21.15.98:0/3169756618 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7974004010 msgr2=0x7f7974004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:18.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.190+0000 7f7991138700 1 --2- 172.21.15.98:0/3169756618 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7974004010 0x7f7974004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:18.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.191+0000 7f7991138700 1 -- 172.21.15.98:0/3169756618 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7974003570 con 0x7f7974004d50 2023-10-06T05:08:18.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.191+0000 7f7991138700 1 --2- 172.21.15.98:0/3169756618 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7974004d50 0x7f79740051c0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f7984006f00 tx=0x7f79840091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8ca6d1e405d952e server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:18.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.191+0000 7f798b7fe700 1 -- 172.21.15.98:0/3169756618 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f798400e710 con 0x7f7974004d50 2023-10-06T05:08:18.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.191+0000 7f798b7fe700 1 -- 172.21.15.98:0/3169756618 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f798400e870 con 0x7f7974004d50 2023-10-06T05:08:18.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.191+0000 7f798b7fe700 1 -- 172.21.15.98:0/3169756618 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f798400ec30 con 0x7f7974004d50 2023-10-06T05:08:18.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.191+0000 7f7992b9b700 1 -- 172.21.15.98:0/3169756618 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7974004d50 msgr2=0x7f79740051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:18.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.191+0000 7f7992b9b700 1 --2- 172.21.15.98:0/3169756618 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7974004d50 0x7f79740051c0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f7984006f00 tx=0x7f79840091a0 comp rx=0 tx=0).stop 2023-10-06T05:08:18.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.191+0000 7f7991138700 1 -- 172.21.15.98:0/3169756618 reap_dead start 2023-10-06T05:08:18.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.191+0000 7f7992b9b700 1 -- 172.21.15.98:0/3169756618 shutdown_connections 2023-10-06T05:08:18.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.191+0000 7f7992b9b700 1 -- 172.21.15.98:0/3169756618 >> 172.21.15.98:0/3169756618 conn(0x7f798c0682f0 msgr2=0x7f798c0686f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:18.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.191+0000 7f7992b9b700 1 -- 172.21.15.98:0/3169756618 shutdown_connections 2023-10-06T05:08:18.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.191+0000 7f7992b9b700 1 -- 172.21.15.98:0/3169756618 wait complete. 2023-10-06T05:08:18.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.192+0000 7f7992b9b700 1 Processor -- start 2023-10-06T05:08:18.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.193+0000 7f7992b9b700 1 -- start start 2023-10-06T05:08:18.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.193+0000 7f7992b9b700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f798c16cbd0 0x7f798c169d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:18.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.193+0000 7f7992b9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f798c16dfd0 0x7f798c15d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:18.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.193+0000 7f7992b9b700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f798c16d340 0x7f798c16d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:18.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.193+0000 7f7992b9b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f798c16dd20 con 0x7f798c16dfd0 2023-10-06T05:08:18.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.193+0000 7f7992b9b700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f798c0c74a0 con 0x7f798c16cbd0 2023-10-06T05:08:18.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.193+0000 7f7992b9b700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f798c0c7d40 con 0x7f798c16d340 2023-10-06T05:08:18.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.193+0000 7f798bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f798c16dfd0 0x7f798c15d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:18.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.193+0000 7f798bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f798c16dfd0 0x7f798c15d0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50018/0 (socket says 172.21.15.98:50018) 2023-10-06T05:08:18.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.193+0000 7f798bfff700 1 -- 172.21.15.98:0/1194999482 learned_addr learned my addr 172.21.15.98:0/1194999482 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:08:18.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.194+0000 7f7990937700 1 --2- 172.21.15.98:0/1194999482 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f798c16cbd0 0x7f798c169d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:18.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.194+0000 7f7991138700 1 --2- 172.21.15.98:0/1194999482 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f798c16d340 0x7f798c16d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:18.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.194+0000 7f798bfff700 1 -- 172.21.15.98:0/1194999482 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f798c16d340 msgr2=0x7f798c16d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:18.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.194+0000 7f798bfff700 1 --2- 172.21.15.98:0/1194999482 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f798c16d340 0x7f798c16d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:18.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.194+0000 7f798bfff700 1 -- 172.21.15.98:0/1194999482 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f798c16cbd0 msgr2=0x7f798c169d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:18.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.194+0000 7f798bfff700 1 --2- 172.21.15.98:0/1194999482 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f798c16cbd0 0x7f798c169d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:18.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.194+0000 7f798bfff700 1 -- 172.21.15.98:0/1194999482 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f798c15d950 con 0x7f798c16dfd0 2023-10-06T05:08:18.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.194+0000 7f7991138700 1 --2- 172.21.15.98:0/1194999482 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f798c16d340 0x7f798c16d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:08:18.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.194+0000 7f7990937700 1 --2- 172.21.15.98:0/1194999482 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f798c16cbd0 0x7f798c169d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:08:18.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.194+0000 7f798bfff700 1 --2- 172.21.15.98:0/1194999482 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f798c16dfd0 0x7f798c15d0a0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f7980002980 tx=0x7f7980007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:18.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.194+0000 7f7988ff9700 1 -- 172.21.15.98:0/1194999482 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7980003390 con 0x7f798c16dfd0 2023-10-06T05:08:18.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.194+0000 7f7988ff9700 1 -- 172.21.15.98:0/1194999482 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f798000f8d0 con 0x7f798c16dfd0 2023-10-06T05:08:18.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.194+0000 7f7992b9b700 1 -- 172.21.15.98:0/1194999482 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f798c0c76f0 con 0x7f798c16dfd0 2023-10-06T05:08:18.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.194+0000 7f7988ff9700 1 -- 172.21.15.98:0/1194999482 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f798000fc90 con 0x7f798c16dfd0 2023-10-06T05:08:18.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.194+0000 7f7992b9b700 1 -- 172.21.15.98:0/1194999482 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f798c16a270 con 0x7f798c16dfd0 2023-10-06T05:08:18.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.195+0000 7f7992b9b700 1 -- 172.21.15.98:0/1194999482 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f798c064a90 con 0x7f798c16dfd0 2023-10-06T05:08:18.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.195+0000 7f7988ff9700 1 -- 172.21.15.98:0/1194999482 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f798001c400 con 0x7f798c16dfd0 2023-10-06T05:08:18.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.196+0000 7f7988ff9700 1 --2- 172.21.15.98:0/1194999482 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f7978066f30 0x7f79780693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:18.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.196+0000 7f7988ff9700 1 -- 172.21.15.98:0/1194999482 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5612+0+0 (secure 0 0 0) 0x7f79800848b0 con 0x7f798c16dfd0 2023-10-06T05:08:18.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.196+0000 7f7990937700 1 --2- 172.21.15.98:0/1194999482 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f7978066f30 0x7f79780693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:18.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.196+0000 7f7988ff9700 1 --2- 172.21.15.98:0/1194999482 >> [v2:172.21.15.98:6816/90662,v1:172.21.15.98:6818/90662] conn(0x7f797806d850 0x7f797806fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:18.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.196+0000 7f7988ff9700 1 -- 172.21.15.98:0/1194999482 --> [v2:172.21.15.98:6816/90662,v1:172.21.15.98:6818/90662] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7978070370 con 0x7f797806d850 2023-10-06T05:08:18.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.196+0000 7f7988ff9700 1 -- 172.21.15.98:0/1194999482 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=46) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7980089240 con 0x7f798c16dfd0 2023-10-06T05:08:18.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.196+0000 7f7991138700 1 --2- 172.21.15.98:0/1194999482 >> [v2:172.21.15.98:6816/90662,v1:172.21.15.98:6818/90662] conn(0x7f797806d850 0x7f797806fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:18.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.196+0000 7f7990937700 1 --2- 172.21.15.98:0/1194999482 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f7978066f30 0x7f79780693e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f797c001190 tx=0x7f797c006040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:18.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.196+0000 7f7991138700 1 --2- 172.21.15.98:0/1194999482 >> [v2:172.21.15.98:6816/90662,v1:172.21.15.98:6818/90662] conn(0x7f797806d850 0x7f797806fca0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:18.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.197+0000 7f7988ff9700 1 -- 172.21.15.98:0/1194999482 <== osd.2 v2:172.21.15.98:6816/90662 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f7978070370 con 0x7f797806d850 2023-10-06T05:08:18.213 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.211+0000 7f7992b9b700 1 -- 172.21.15.98:0/1194999482 --> [v2:172.21.15.98:6816/90662,v1:172.21.15.98:6818/90662] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f798c0c7b90 con 0x7f797806d850 2023-10-06T05:08:18.213 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.211+0000 7f7988ff9700 1 -- 172.21.15.98:0/1194999482 <== osd.2 v2:172.21.15.98:6816/90662 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f798c0c7b90 con 0x7f797806d850 2023-10-06T05:08:18.213 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-10-06T05:08:18.213 INFO:teuthology.orchestra.run.smithi098.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-06T05:08:18.213 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-10-06T05:08:18.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.211+0000 7f7992b9b700 1 -- 172.21.15.98:0/1194999482 >> [v2:172.21.15.98:6816/90662,v1:172.21.15.98:6818/90662] conn(0x7f797806d850 msgr2=0x7f797806fca0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:18.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.211+0000 7f7992b9b700 1 --2- 172.21.15.98:0/1194999482 >> [v2:172.21.15.98:6816/90662,v1:172.21.15.98:6818/90662] conn(0x7f797806d850 0x7f797806fca0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:18.215 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.211+0000 7f7992b9b700 1 -- 172.21.15.98:0/1194999482 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f7978066f30 msgr2=0x7f79780693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:18.215 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.211+0000 7f7992b9b700 1 --2- 172.21.15.98:0/1194999482 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f7978066f30 0x7f79780693e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f797c001190 tx=0x7f797c006040 comp rx=0 tx=0).stop 2023-10-06T05:08:18.215 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.211+0000 7f7992b9b700 1 -- 172.21.15.98:0/1194999482 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f798c16dfd0 msgr2=0x7f798c15d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:18.215 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.211+0000 7f7992b9b700 1 --2- 172.21.15.98:0/1194999482 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f798c16dfd0 0x7f798c15d0a0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f7980002980 tx=0x7f7980007d30 comp rx=0 tx=0).stop 2023-10-06T05:08:18.215 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.211+0000 7f7991138700 1 -- 172.21.15.98:0/1194999482 reap_dead start 2023-10-06T05:08:18.215 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.211+0000 7f7992b9b700 1 -- 172.21.15.98:0/1194999482 shutdown_connections 2023-10-06T05:08:18.215 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.211+0000 7f7992b9b700 1 -- 172.21.15.98:0/1194999482 >> 172.21.15.98:0/1194999482 conn(0x7f798c0682f0 msgr2=0x7f798c06f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:18.215 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.211+0000 7f7992b9b700 1 -- 172.21.15.98:0/1194999482 shutdown_connections 2023-10-06T05:08:18.215 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.211+0000 7f7992b9b700 1 -- 172.21.15.98:0/1194999482 wait complete. 2023-10-06T05:08:18.225 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-10-06T05:08:18.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.338+0000 7fc04b415700 1 Processor -- start 2023-10-06T05:08:18.340 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.338+0000 7fc04b415700 1 -- start start 2023-10-06T05:08:18.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.339+0000 7fc04b415700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc044159440 0x7fc0440b3090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:18.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.339+0000 7fc04b415700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0440cb7a0 con 0x7fc0440b2100 2023-10-06T05:08:18.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.339+0000 7fc04b415700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0440b0860 con 0x7fc0440b35d0 2023-10-06T05:08:18.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.339+0000 7fc04b415700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0440cb150 con 0x7fc044159440 2023-10-06T05:08:18.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.339+0000 7fc04ac14700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fc0440b2100 0x7fc0440b2580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:47120/0 (socket says 172.21.15.98:47120) 2023-10-06T05:08:18.345 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.339+0000 7fc04ac14700 1 -- 172.21.15.98:0/1657863456 learned_addr learned my addr 172.21.15.98:0/1657863456 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:08:18.345 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.339+0000 7fc049411700 1 -- 172.21.15.98:0/1657863456 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1598333247 0 0) 0x7fc0440cb7a0 con 0x7fc0440b2100 2023-10-06T05:08:18.345 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.339+0000 7fc049411700 1 -- 172.21.15.98:0/1657863456 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc030003570 con 0x7fc0440b2100 2023-10-06T05:08:18.345 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.339+0000 7fc04a413700 1 --2- 172.21.15.98:0/1657863456 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc044159440 0x7fc0440b3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:18.345 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.339+0000 7fc049411700 1 -- 172.21.15.98:0/1657863456 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 462167469 0 0) 0x7fc030003570 con 0x7fc0440b2100 2023-10-06T05:08:18.345 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.339+0000 7fc049411700 1 -- 172.21.15.98:0/1657863456 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc044159440 msgr2=0x7fc0440b3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:18.345 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.339+0000 7fc049411700 1 --2- 172.21.15.98:0/1657863456 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc044159440 0x7fc0440b3090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:18.345 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.339+0000 7fc049411700 1 -- 172.21.15.98:0/1657863456 >> v1:172.21.15.174:6789/0 conn(0x7fc0440b35d0 legacy=0x7fc0440cbc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:18.345 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.339+0000 7fc049411700 1 -- 172.21.15.98:0/1657863456 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0440cb430 con 0x7fc0440b2100 2023-10-06T05:08:18.345 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.339+0000 7fc049411700 1 -- 172.21.15.98:0/1657863456 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1543991876 0 0) 0x7fc03c0017f0 con 0x7fc0440b2100 2023-10-06T05:08:18.346 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.339+0000 7fc04a413700 1 --2- 172.21.15.98:0/1657863456 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc044159440 0x7fc0440b3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:08:18.346 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.339+0000 7fc049411700 1 -- 172.21.15.98:0/1657863456 >> v1:172.21.15.98:6789/0 conn(0x7fc0440b2100 legacy=0x7fc0440b2580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:18.346 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.340+0000 7fc049411700 1 --2- 172.21.15.98:0/1657863456 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc044159440 0x7fc030003a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:18.346 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.340+0000 7fc049411700 1 --2- 172.21.15.98:0/1657863456 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc030004080 0x7fc030004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:18.346 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.340+0000 7fc049411700 1 --2- 172.21.15.98:0/1657863456 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc030004d50 0x7fc0300051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:18.346 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.340+0000 7fc049411700 1 -- 172.21.15.98:0/1657863456 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0440cb150 con 0x7fc030004080 2023-10-06T05:08:18.346 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.340+0000 7fc049411700 1 -- 172.21.15.98:0/1657863456 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc03000bd40 con 0x7fc030004d50 2023-10-06T05:08:18.346 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.340+0000 7fc049411700 1 -- 172.21.15.98:0/1657863456 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc03000bf00 con 0x7fc044159440 2023-10-06T05:08:18.346 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.340+0000 7fc049c12700 1 --2- 172.21.15.98:0/1657863456 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc030004080 0x7fc030004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:18.346 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.340+0000 7fc04a413700 1 --2- 172.21.15.98:0/1657863456 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc044159440 0x7fc030003a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:18.346 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.340+0000 7fc04ac14700 1 --2- 172.21.15.98:0/1657863456 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc030004d50 0x7fc0300051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:18.346 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.340+0000 7fc049c12700 1 -- 172.21.15.98:0/1657863456 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc044159440 msgr2=0x7fc030003a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:18.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.340+0000 7fc049c12700 1 --2- 172.21.15.98:0/1657863456 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc044159440 0x7fc030003a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:18.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.340+0000 7fc049c12700 1 -- 172.21.15.98:0/1657863456 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc030004d50 msgr2=0x7fc0300051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:18.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.340+0000 7fc049c12700 1 --2- 172.21.15.98:0/1657863456 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc030004d50 0x7fc0300051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:18.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.340+0000 7fc049c12700 1 -- 172.21.15.98:0/1657863456 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc030003570 con 0x7fc030004080 2023-10-06T05:08:18.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.340+0000 7fc049c12700 1 --2- 172.21.15.98:0/1657863456 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc030004080 0x7fc030004810 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fc034001880 tx=0x7fc034007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebef0ec687ad9d06 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:18.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.340+0000 7fc049411700 1 -- 172.21.15.98:0/1657863456 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc034012420 con 0x7fc030004080 2023-10-06T05:08:18.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.340+0000 7fc049411700 1 -- 172.21.15.98:0/1657863456 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc034012580 con 0x7fc030004080 2023-10-06T05:08:18.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.340+0000 7fc049411700 1 -- 172.21.15.98:0/1657863456 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc034012920 con 0x7fc030004080 2023-10-06T05:08:18.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.341+0000 7fc04b415700 1 -- 172.21.15.98:0/1657863456 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc030004080 msgr2=0x7fc030004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:18.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.341+0000 7fc04b415700 1 --2- 172.21.15.98:0/1657863456 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc030004080 0x7fc030004810 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fc034001880 tx=0x7fc034007d30 comp rx=0 tx=0).stop 2023-10-06T05:08:18.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.341+0000 7fc04ac14700 1 -- 172.21.15.98:0/1657863456 reap_dead start 2023-10-06T05:08:18.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.341+0000 7fc04b415700 1 -- 172.21.15.98:0/1657863456 shutdown_connections 2023-10-06T05:08:18.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.341+0000 7fc04b415700 1 -- 172.21.15.98:0/1657863456 >> 172.21.15.98:0/1657863456 conn(0x7fc04406ece0 msgr2=0x7fc04406f0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:18.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.341+0000 7fc04b415700 1 -- 172.21.15.98:0/1657863456 shutdown_connections 2023-10-06T05:08:18.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.341+0000 7fc04b415700 1 -- 172.21.15.98:0/1657863456 wait complete. 2023-10-06T05:08:18.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.342+0000 7fc04b415700 1 Processor -- start 2023-10-06T05:08:18.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.342+0000 7fc04b415700 1 -- start start 2023-10-06T05:08:18.351 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.343+0000 7fc04b415700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc030004080 0x7fc04416a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:18.351 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.343+0000 7fc04b415700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc0441663d0 0x7fc044169120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:18.351 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.343+0000 7fc04b415700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc044166840 0x7fc044169660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:18.351 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.343+0000 7fc04b415700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc04416c9b0 con 0x7fc044166840 2023-10-06T05:08:18.351 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.343+0000 7fc04b415700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc044169cf0 con 0x7fc030004080 2023-10-06T05:08:18.351 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.343+0000 7fc04b415700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc044169fd0 con 0x7fc0441663d0 2023-10-06T05:08:18.351 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.343+0000 7fc04ac14700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc044166840 0x7fc044169660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:18.351 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.343+0000 7fc04ac14700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc044166840 0x7fc044169660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50040/0 (socket says 172.21.15.98:50040) 2023-10-06T05:08:18.352 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.343+0000 7fc04ac14700 1 -- 172.21.15.98:0/2775878938 learned_addr learned my addr 172.21.15.98:0/2775878938 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:08:18.352 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.343+0000 7fc049c12700 1 --2- 172.21.15.98:0/2775878938 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc0441663d0 0x7fc044169120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:18.352 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.343+0000 7fc04a413700 1 --2- 172.21.15.98:0/2775878938 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc030004080 0x7fc04416a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:18.352 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.343+0000 7fc04ac14700 1 -- 172.21.15.98:0/2775878938 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc0441663d0 msgr2=0x7fc044169120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:18.352 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.343+0000 7fc04ac14700 1 --2- 172.21.15.98:0/2775878938 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc0441663d0 0x7fc044169120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:18.352 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.343+0000 7fc04ac14700 1 -- 172.21.15.98:0/2775878938 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc030004080 msgr2=0x7fc04416a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:18.352 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.343+0000 7fc04ac14700 1 --2- 172.21.15.98:0/2775878938 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc030004080 0x7fc04416a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:18.352 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.343+0000 7fc04ac14700 1 -- 172.21.15.98:0/2775878938 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0440c5190 con 0x7fc044166840 2023-10-06T05:08:18.352 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.343+0000 7fc04ac14700 1 --2- 172.21.15.98:0/2775878938 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc044166840 0x7fc044169660 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fc03c000f80 tx=0x7fc03c0021d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:18.352 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.343+0000 7fc03a7fc700 1 -- 172.21.15.98:0/2775878938 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc03c00ac90 con 0x7fc044166840 2023-10-06T05:08:18.352 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.343+0000 7fc03a7fc700 1 -- 172.21.15.98:0/2775878938 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc03c00ae30 con 0x7fc044166840 2023-10-06T05:08:18.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.343+0000 7fc03a7fc700 1 -- 172.21.15.98:0/2775878938 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc03c013640 con 0x7fc044166840 2023-10-06T05:08:18.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.343+0000 7fc04b415700 1 -- 172.21.15.98:0/2775878938 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0440c5470 con 0x7fc044166840 2023-10-06T05:08:18.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.343+0000 7fc04b415700 1 -- 172.21.15.98:0/2775878938 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc04416a820 con 0x7fc044166840 2023-10-06T05:08:18.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.344+0000 7fc04b415700 1 -- 172.21.15.98:0/2775878938 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc04416aa10 con 0x7fc044166840 2023-10-06T05:08:18.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.345+0000 7fc03a7fc700 1 -- 172.21.15.98:0/2775878938 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fc03c013a30 con 0x7fc044166840 2023-10-06T05:08:18.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.345+0000 7fc03a7fc700 1 --2- 172.21.15.98:0/2775878938 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fc028066f30 0x7fc0280693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:18.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.345+0000 7fc03a7fc700 1 -- 172.21.15.98:0/2775878938 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5612+0+0 (secure 0 0 0) 0x7fc03c085010 con 0x7fc044166840 2023-10-06T05:08:18.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.345+0000 7fc04a413700 1 --2- 172.21.15.98:0/2775878938 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fc028066f30 0x7fc0280693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:18.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.345+0000 7fc04a413700 1 --2- 172.21.15.98:0/2775878938 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fc028066f30 0x7fc0280693e0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fc040006fd0 tx=0x7fc040008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:18.354 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.349+0000 7fc03a7fc700 1 -- 172.21.15.98:0/2775878938 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc03c0522e0 con 0x7fc044166840 2023-10-06T05:08:18.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.533+0000 7fc04b415700 1 -- 172.21.15.98:0/2775878938 --> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7fc0440c5660 con 0x7fc028066f30 2023-10-06T05:08:18.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.533+0000 7fc03a7fc700 1 -- 172.21.15.98:0/2775878938 <== mgr.4104 v2:172.21.15.98:6832/90397 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fc0440c5660 con 0x7fc028066f30 2023-10-06T05:08:18.538 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.536+0000 7fc04b415700 1 -- 172.21.15.98:0/2775878938 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fc028066f30 msgr2=0x7fc0280693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:18.538 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.536+0000 7fc04b415700 1 --2- 172.21.15.98:0/2775878938 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fc028066f30 0x7fc0280693e0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fc040006fd0 tx=0x7fc040008040 comp rx=0 tx=0).stop 2023-10-06T05:08:18.538 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.536+0000 7fc04b415700 1 -- 172.21.15.98:0/2775878938 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc044166840 msgr2=0x7fc044169660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:18.538 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.536+0000 7fc04b415700 1 --2- 172.21.15.98:0/2775878938 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc044166840 0x7fc044169660 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fc03c000f80 tx=0x7fc03c0021d0 comp rx=0 tx=0).stop 2023-10-06T05:08:18.538 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.536+0000 7fc04b415700 1 -- 172.21.15.98:0/2775878938 shutdown_connections 2023-10-06T05:08:18.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.536+0000 7fc04b415700 1 --2- 172.21.15.98:0/2775878938 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fc028066f30 0x7fc0280693e0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:18.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.536+0000 7fc04b415700 1 --2- 172.21.15.98:0/2775878938 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc044166840 0x7fc044169660 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:18.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.536+0000 7fc04b415700 1 --2- 172.21.15.98:0/2775878938 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc030004080 0x7fc04416a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:18.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.536+0000 7fc04b415700 1 --2- 172.21.15.98:0/2775878938 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc0441663d0 0x7fc044169120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:18.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.536+0000 7fc04b415700 1 -- 172.21.15.98:0/2775878938 >> 172.21.15.98:0/2775878938 conn(0x7fc04406ece0 msgr2=0x7fc044158580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:18.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.536+0000 7fc04b415700 1 -- 172.21.15.98:0/2775878938 shutdown_connections 2023-10-06T05:08:18.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.536+0000 7fc04b415700 1 -- 172.21.15.98:0/2775878938 wait complete. 2023-10-06T05:08:18.539 INFO:teuthology.orchestra.run.smithi098.stderr:instructed osd(s) 2 to deep-scrub 2023-10-06T05:08:18.549 INFO:tasks.ceph:Scrubbing osd.3 2023-10-06T05:08:18.550 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2023-10-06T05:08:18.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.665+0000 7f6cc9f2a700 1 Processor -- start 2023-10-06T05:08:18.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.665+0000 7f6cc9f2a700 1 -- start start 2023-10-06T05:08:18.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.666+0000 7f6cc9f2a700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6cc406ece0 0x7f6cc40b2360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:18.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.666+0000 7f6cc9f2a700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f6cc406f0b0 0x7f6cc40b08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:18.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.666+0000 7f6cc9f2a700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6cc40aede0 0x7f6cc40af240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:18.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.666+0000 7f6cc9f2a700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cc406fb40 con 0x7f6cc406ece0 2023-10-06T05:08:18.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.666+0000 7f6cc9f2a700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cc40b2ce0 con 0x7f6cc406f0b0 2023-10-06T05:08:18.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.666+0000 7f6cc9f2a700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cc40b0410 con 0x7f6cc40aede0 2023-10-06T05:08:18.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.666+0000 7f6cc37fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6cc406ece0 0x7f6cc40b2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:18.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.666+0000 7f6cc37fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6cc406ece0 0x7f6cc40b2360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50054/0 (socket says 172.21.15.98:50054) 2023-10-06T05:08:18.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.666+0000 7f6cc37fe700 1 -- 172.21.15.98:0/941479084 learned_addr learned my addr 172.21.15.98:0/941479084 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:08:18.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.666+0000 7f6cc37fe700 1 -- 172.21.15.98:0/941479084 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6cc40aede0 msgr2=0x7f6cc40af240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:18.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.666+0000 7f6cc37fe700 1 --2- 172.21.15.98:0/941479084 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6cc40aede0 0x7f6cc40af240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:18.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.666+0000 7f6cc37fe700 1 -- 172.21.15.98:0/941479084 >> v2:172.21.15.174:3300/0 conn(0x7f6cc406f0b0 msgr2=0x7f6cc40b08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:18.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.666+0000 7f6cc37fe700 1 --2- 172.21.15.98:0/941479084 >> v2:172.21.15.174:3300/0 conn(0x7f6cc406f0b0 0x7f6cc40b08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:18.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.666+0000 7f6cc37fe700 1 -- 172.21.15.98:0/941479084 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cc40af780 con 0x7f6cc406ece0 2023-10-06T05:08:18.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.667+0000 7f6cc37fe700 1 --2- 172.21.15.98:0/941479084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cc406ece0 0x7f6cc40b2360 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f6cc40b18e0 tx=0x7f6cb4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7481ba8a102a585 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:18.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.667+0000 7f6cc27fc700 1 -- 172.21.15.98:0/941479084 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cb400e070 con 0x7f6cc406ece0 2023-10-06T05:08:18.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.667+0000 7f6cc27fc700 1 -- 172.21.15.98:0/941479084 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cb400ba50 con 0x7f6cc406ece0 2023-10-06T05:08:18.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.667+0000 7f6cc27fc700 1 -- 172.21.15.98:0/941479084 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cb40092e0 con 0x7f6cc406ece0 2023-10-06T05:08:18.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.667+0000 7f6cc9f2a700 1 -- 172.21.15.98:0/941479084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cc406ece0 msgr2=0x7f6cc40b2360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:18.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.667+0000 7f6cc9f2a700 1 --2- 172.21.15.98:0/941479084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cc406ece0 0x7f6cc40b2360 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f6cc40b18e0 tx=0x7f6cb4009840 comp rx=0 tx=0).stop 2023-10-06T05:08:18.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.667+0000 7f6cc9f2a700 1 -- 172.21.15.98:0/941479084 shutdown_connections 2023-10-06T05:08:18.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.667+0000 7f6cc9f2a700 1 --2- 172.21.15.98:0/941479084 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6cc40aede0 0x7f6cc40af240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:18.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.667+0000 7f6cc9f2a700 1 --2- 172.21.15.98:0/941479084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cc406ece0 0x7f6cc40b2360 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:18.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.667+0000 7f6cc9f2a700 1 --2- 172.21.15.98:0/941479084 >> v2:172.21.15.174:3300/0 conn(0x7f6cc406f0b0 0x7f6cc40b08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:18.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.667+0000 7f6cc9f2a700 1 -- 172.21.15.98:0/941479084 >> 172.21.15.98:0/941479084 conn(0x7f6cc40682f0 msgr2=0x7f6cc40686f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:18.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.667+0000 7f6cc9f2a700 1 -- 172.21.15.98:0/941479084 shutdown_connections 2023-10-06T05:08:18.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.667+0000 7f6cc9f2a700 1 -- 172.21.15.98:0/941479084 wait complete. 2023-10-06T05:08:18.675 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.668+0000 7f6cc9f2a700 1 Processor -- start 2023-10-06T05:08:18.675 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.669+0000 7f6cc9f2a700 1 -- start start 2023-10-06T05:08:18.675 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.669+0000 7f6cc9f2a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cc406ece0 0x7f6cc414adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:18.675 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.669+0000 7f6cc9f2a700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6cc406f0b0 0x7f6cc40ccb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:18.676 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.669+0000 7f6cc9f2a700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6cc40aede0 0x7f6cc40cd0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:18.676 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.669+0000 7f6cc9f2a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cc40a8d20 con 0x7f6cc406ece0 2023-10-06T05:08:18.676 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.669+0000 7f6cc9f2a700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cc40a8710 con 0x7f6cc40aede0 2023-10-06T05:08:18.676 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.669+0000 7f6cc9f2a700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cc40ae880 con 0x7f6cc406f0b0 2023-10-06T05:08:18.676 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.669+0000 7f6cc37fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cc406ece0 0x7f6cc414adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:18.676 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.669+0000 7f6cc37fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cc406ece0 0x7f6cc414adc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50066/0 (socket says 172.21.15.98:50066) 2023-10-06T05:08:18.676 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.669+0000 7f6cc37fe700 1 -- 172.21.15.98:0/2350408213 learned_addr learned my addr 172.21.15.98:0/2350408213 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:08:18.676 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.670+0000 7f6cc37fe700 1 -- 172.21.15.98:0/2350408213 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6cc406f0b0 msgr2=0x7f6cc40ccb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:18.676 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.670+0000 7f6cc37fe700 1 --2- 172.21.15.98:0/2350408213 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6cc406f0b0 0x7f6cc40ccb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:18.676 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.670+0000 7f6cc37fe700 1 -- 172.21.15.98:0/2350408213 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6cc40aede0 msgr2=0x7f6cc40cd0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:18.676 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.670+0000 7f6cc3fff700 1 --2- 172.21.15.98:0/2350408213 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6cc40aede0 0x7f6cc40cd0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:18.676 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.670+0000 7f6cc37fe700 1 --2- 172.21.15.98:0/2350408213 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6cc40aede0 0x7f6cc40cd0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:18.677 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.670+0000 7f6cc37fe700 1 -- 172.21.15.98:0/2350408213 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cb4009ae0 con 0x7f6cc406ece0 2023-10-06T05:08:18.677 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.670+0000 7f6cc37fe700 1 --2- 172.21.15.98:0/2350408213 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cc406ece0 0x7f6cc414adc0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f6cc4089380 tx=0x7f6cb4018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:18.677 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.670+0000 7f6cab7fe700 1 -- 172.21.15.98:0/2350408213 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cb4008c60 con 0x7f6cc406ece0 2023-10-06T05:08:18.677 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.670+0000 7f6cab7fe700 1 -- 172.21.15.98:0/2350408213 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cb4008dc0 con 0x7f6cc406ece0 2023-10-06T05:08:18.677 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.670+0000 7f6cc9f2a700 1 -- 172.21.15.98:0/2350408213 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cc40a89b0 con 0x7f6cc406ece0 2023-10-06T05:08:18.677 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.670+0000 7f6cab7fe700 1 -- 172.21.15.98:0/2350408213 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cb40266b0 con 0x7f6cc406ece0 2023-10-06T05:08:18.677 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.670+0000 7f6cc9f2a700 1 -- 172.21.15.98:0/2350408213 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cc414b300 con 0x7f6cc406ece0 2023-10-06T05:08:18.677 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.671+0000 7f6cc9f2a700 1 -- 172.21.15.98:0/2350408213 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6cc4064a90 con 0x7f6cc406ece0 2023-10-06T05:08:18.677 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.671+0000 7f6cab7fe700 1 -- 172.21.15.98:0/2350408213 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f6cb4003040 con 0x7f6cc406ece0 2023-10-06T05:08:18.677 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.672+0000 7f6cab7fe700 1 --2- 172.21.15.98:0/2350408213 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f6cb0066f30 0x7f6cb00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:18.677 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.672+0000 7f6cab7fe700 1 -- 172.21.15.98:0/2350408213 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5612+0+0 (secure 0 0 0) 0x7f6cb4085d10 con 0x7f6cc406ece0 2023-10-06T05:08:18.677 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.672+0000 7f6cc2ffd700 1 --2- 172.21.15.98:0/2350408213 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f6cb0066f30 0x7f6cb00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:18.678 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.672+0000 7f6cab7fe700 1 --2- 172.21.15.98:0/2350408213 >> [v2:172.21.15.98:6817/90663,v1:172.21.15.98:6819/90663] conn(0x7f6cb006d850 0x7f6cb006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:18.678 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.672+0000 7f6cab7fe700 1 -- 172.21.15.98:0/2350408213 --> [v2:172.21.15.98:6817/90663,v1:172.21.15.98:6819/90663] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6cb0070370 con 0x7f6cb006d850 2023-10-06T05:08:18.678 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.672+0000 7f6cab7fe700 1 -- 172.21.15.98:0/2350408213 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=46) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6cb40882f0 con 0x7f6cc406ece0 2023-10-06T05:08:18.678 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.672+0000 7f6cc3fff700 1 --2- 172.21.15.98:0/2350408213 >> [v2:172.21.15.98:6817/90663,v1:172.21.15.98:6819/90663] conn(0x7f6cb006d850 0x7f6cb006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:18.678 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.672+0000 7f6cc2ffd700 1 --2- 172.21.15.98:0/2350408213 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f6cb0066f30 0x7f6cb00693e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f6cc416cd10 tx=0x7f6cb8006d10 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:18.678 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.672+0000 7f6cc3fff700 1 --2- 172.21.15.98:0/2350408213 >> [v2:172.21.15.98:6817/90663,v1:172.21.15.98:6819/90663] conn(0x7f6cb006d850 0x7f6cb006fca0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:18.678 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.673+0000 7f6cab7fe700 1 -- 172.21.15.98:0/2350408213 <== osd.3 v2:172.21.15.98:6817/90663 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f6cb0070370 con 0x7f6cb006d850 2023-10-06T05:08:18.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.686+0000 7f6cc9f2a700 1 -- 172.21.15.98:0/2350408213 --> [v2:172.21.15.98:6817/90663,v1:172.21.15.98:6819/90663] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f6cc40a8b40 con 0x7f6cb006d850 2023-10-06T05:08:18.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.686+0000 7f6cab7fe700 1 -- 172.21.15.98:0/2350408213 <== osd.3 v2:172.21.15.98:6817/90663 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f6cc40a8b40 con 0x7f6cb006d850 2023-10-06T05:08:18.689 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-10-06T05:08:18.689 INFO:teuthology.orchestra.run.smithi098.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-06T05:08:18.689 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-10-06T05:08:18.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.687+0000 7f6cc9f2a700 1 -- 172.21.15.98:0/2350408213 >> [v2:172.21.15.98:6817/90663,v1:172.21.15.98:6819/90663] conn(0x7f6cb006d850 msgr2=0x7f6cb006fca0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:18.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.687+0000 7f6cc9f2a700 1 --2- 172.21.15.98:0/2350408213 >> [v2:172.21.15.98:6817/90663,v1:172.21.15.98:6819/90663] conn(0x7f6cb006d850 0x7f6cb006fca0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:18.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.687+0000 7f6cc9f2a700 1 -- 172.21.15.98:0/2350408213 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f6cb0066f30 msgr2=0x7f6cb00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:18.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.687+0000 7f6cc9f2a700 1 --2- 172.21.15.98:0/2350408213 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f6cb0066f30 0x7f6cb00693e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f6cc416cd10 tx=0x7f6cb8006d10 comp rx=0 tx=0).stop 2023-10-06T05:08:18.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.687+0000 7f6cc9f2a700 1 -- 172.21.15.98:0/2350408213 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cc406ece0 msgr2=0x7f6cc414adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:18.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.687+0000 7f6cc9f2a700 1 --2- 172.21.15.98:0/2350408213 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6cc406ece0 0x7f6cc414adc0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f6cc4089380 tx=0x7f6cb4018000 comp rx=0 tx=0).stop 2023-10-06T05:08:18.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.687+0000 7f6cc3fff700 1 -- 172.21.15.98:0/2350408213 reap_dead start 2023-10-06T05:08:18.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.687+0000 7f6cc9f2a700 1 -- 172.21.15.98:0/2350408213 shutdown_connections 2023-10-06T05:08:18.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.687+0000 7f6cc9f2a700 1 -- 172.21.15.98:0/2350408213 >> 172.21.15.98:0/2350408213 conn(0x7f6cc40682f0 msgr2=0x7f6cc40ce0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:18.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.687+0000 7f6cc9f2a700 1 -- 172.21.15.98:0/2350408213 shutdown_connections 2023-10-06T05:08:18.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.687+0000 7f6cc9f2a700 1 -- 172.21.15.98:0/2350408213 wait complete. 2023-10-06T05:08:18.700 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2023-10-06T05:08:18.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.814+0000 7fc17172f700 1 Processor -- start 2023-10-06T05:08:18.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.814+0000 7fc17172f700 1 -- start start 2023-10-06T05:08:18.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.815+0000 7fc17172f700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fc16c0ad210 0x7fc16c0cd290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:18.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.815+0000 7fc17172f700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc16c0ad5e0 0x7fc16c15d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:18.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.815+0000 7fc17172f700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc16c0cecf0 con 0x7fc16c15d940 2023-10-06T05:08:18.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.815+0000 7fc17172f700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc16c15ddc0 con 0x7fc16c0ad5e0 2023-10-06T05:08:18.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.815+0000 7fc17172f700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc16c0cefd0 con 0x7fc16c0ad210 2023-10-06T05:08:18.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.815+0000 7fc1627fc700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc16c0ad5e0 0x7fc16c15d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:18.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.815+0000 7fc1627fc700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc16c0ad5e0 0x7fc16c15d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50074/0 (socket says 172.21.15.98:50074) 2023-10-06T05:08:18.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.816+0000 7fc1627fc700 1 -- 172.21.15.98:0/1421839821 learned_addr learned my addr 172.21.15.98:0/1421839821 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:08:18.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.816+0000 7fc1627fc700 1 -- 172.21.15.98:0/1421839821 >> v2:172.21.15.174:3300/0 conn(0x7fc16c0ad210 msgr2=0x7fc16c0cd290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:18.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.816+0000 7fc1627fc700 1 --2- 172.21.15.98:0/1421839821 >> v2:172.21.15.174:3300/0 conn(0x7fc16c0ad210 0x7fc16c0cd290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:18.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.816+0000 7fc1627fc700 1 -- 172.21.15.98:0/1421839821 >> v1:172.21.15.174:6789/0 conn(0x7fc16c15d940 legacy=0x7fc16c0686c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:18.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.816+0000 7fc1627fc700 1 -- 172.21.15.98:0/1421839821 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc16c0cca90 con 0x7fc16c0ad5e0 2023-10-06T05:08:18.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.816+0000 7fc1627fc700 1 --2- 172.21.15.98:0/1421839821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc16c0ad5e0 0x7fc16c15d400 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fc16c0cacb0 tx=0x7fc154009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb4826b58eacde42 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:18.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.816+0000 7fc16a7fc700 1 -- 172.21.15.98:0/1421839821 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc15400b820 con 0x7fc16c0ad5e0 2023-10-06T05:08:18.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.816+0000 7fc16a7fc700 1 -- 172.21.15.98:0/1421839821 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc15400b980 con 0x7fc16c0ad5e0 2023-10-06T05:08:18.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.816+0000 7fc16a7fc700 1 -- 172.21.15.98:0/1421839821 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc15400bd20 con 0x7fc16c0ad5e0 2023-10-06T05:08:18.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.816+0000 7fc17172f700 1 -- 172.21.15.98:0/1421839821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc16c0ad5e0 msgr2=0x7fc16c15d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:18.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.816+0000 7fc17172f700 1 --2- 172.21.15.98:0/1421839821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc16c0ad5e0 0x7fc16c15d400 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fc16c0cacb0 tx=0x7fc154009250 comp rx=0 tx=0).stop 2023-10-06T05:08:18.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.817+0000 7fc17172f700 1 -- 172.21.15.98:0/1421839821 shutdown_connections 2023-10-06T05:08:18.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.817+0000 7fc17172f700 1 --2- 172.21.15.98:0/1421839821 >> v2:172.21.15.174:3300/0 conn(0x7fc16c0ad210 0x7fc16c0cd290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:18.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.817+0000 7fc17172f700 1 --2- 172.21.15.98:0/1421839821 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc16c0ad5e0 0x7fc16c15d400 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:18.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.817+0000 7fc17172f700 1 -- 172.21.15.98:0/1421839821 >> 172.21.15.98:0/1421839821 conn(0x7fc16c067d90 msgr2=0x7fc16c0ce270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:18.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.817+0000 7fc17172f700 1 -- 172.21.15.98:0/1421839821 shutdown_connections 2023-10-06T05:08:18.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.817+0000 7fc17172f700 1 -- 172.21.15.98:0/1421839821 wait complete. 2023-10-06T05:08:18.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.818+0000 7fc17172f700 1 Processor -- start 2023-10-06T05:08:18.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.818+0000 7fc17172f700 1 -- start start 2023-10-06T05:08:18.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.819+0000 7fc17172f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc16c0ad210 0x7fc16c0c5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:18.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.819+0000 7fc17172f700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc16c0ad5e0 0x7fc16c06c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:18.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.819+0000 7fc17172f700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc16c15d940 0x7fc16c0c35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:18.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.819+0000 7fc17172f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc16c0c2e70 con 0x7fc16c0ad210 2023-10-06T05:08:18.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.819+0000 7fc17172f700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc16c0c3150 con 0x7fc16c0ad5e0 2023-10-06T05:08:18.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.819+0000 7fc17172f700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc16c0c4b10 con 0x7fc16c15d940 2023-10-06T05:08:18.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.819+0000 7fc16affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc16c0ad210 0x7fc16c0c5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:18.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.819+0000 7fc16affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc16c0ad210 0x7fc16c0c5280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50080/0 (socket says 172.21.15.98:50080) 2023-10-06T05:08:18.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.819+0000 7fc16affd700 1 -- 172.21.15.98:0/1647541114 learned_addr learned my addr 172.21.15.98:0/1647541114 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:08:18.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.819+0000 7fc1627fc700 1 --2- 172.21.15.98:0/1647541114 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc16c0ad5e0 0x7fc16c06c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:18.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.819+0000 7fc16b7fe700 1 --2- 172.21.15.98:0/1647541114 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc16c15d940 0x7fc16c0c35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:18.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.819+0000 7fc16affd700 1 -- 172.21.15.98:0/1647541114 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc16c15d940 msgr2=0x7fc16c0c35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:18.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.819+0000 7fc16affd700 1 --2- 172.21.15.98:0/1647541114 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc16c15d940 0x7fc16c0c35e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:18.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.819+0000 7fc16affd700 1 -- 172.21.15.98:0/1647541114 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc16c0ad5e0 msgr2=0x7fc16c06c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:18.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.819+0000 7fc16affd700 1 --2- 172.21.15.98:0/1647541114 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc16c0ad5e0 0x7fc16c06c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:18.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.819+0000 7fc16affd700 1 -- 172.21.15.98:0/1647541114 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1540094f0 con 0x7fc16c0ad210 2023-10-06T05:08:18.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.819+0000 7fc16b7fe700 1 --2- 172.21.15.98:0/1647541114 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc16c15d940 0x7fc16c0c35e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:08:18.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.819+0000 7fc16affd700 1 --2- 172.21.15.98:0/1647541114 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc16c0ad210 0x7fc16c0c5280 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fc15c00b870 tx=0x7fc15c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:18.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.819+0000 7fc1637fe700 1 -- 172.21.15.98:0/1647541114 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc15c00f510 con 0x7fc16c0ad210 2023-10-06T05:08:18.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.819+0000 7fc1637fe700 1 -- 172.21.15.98:0/1647541114 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc15c00f670 con 0x7fc16c0ad210 2023-10-06T05:08:18.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.819+0000 7fc17172f700 1 -- 172.21.15.98:0/1647541114 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc16c0b1530 con 0x7fc16c0ad210 2023-10-06T05:08:18.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.819+0000 7fc1637fe700 1 -- 172.21.15.98:0/1647541114 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc15c00fa30 con 0x7fc16c0ad210 2023-10-06T05:08:18.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.819+0000 7fc17172f700 1 -- 172.21.15.98:0/1647541114 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc16c0b1af0 con 0x7fc16c0ad210 2023-10-06T05:08:18.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.820+0000 7fc17172f700 1 -- 172.21.15.98:0/1647541114 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc16c064a90 con 0x7fc16c0ad210 2023-10-06T05:08:18.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.821+0000 7fc1637fe700 1 -- 172.21.15.98:0/1647541114 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fc15c01a400 con 0x7fc16c0ad210 2023-10-06T05:08:18.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.821+0000 7fc1637fe700 1 --2- 172.21.15.98:0/1647541114 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fc14c066f30 0x7fc14c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:18.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.821+0000 7fc1637fe700 1 -- 172.21.15.98:0/1647541114 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5612+0+0 (secure 0 0 0) 0x7fc15c084030 con 0x7fc16c0ad210 2023-10-06T05:08:18.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.821+0000 7fc1627fc700 1 --2- 172.21.15.98:0/1647541114 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fc14c066f30 0x7fc14c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:18.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.821+0000 7fc1627fc700 1 --2- 172.21.15.98:0/1647541114 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fc14c066f30 0x7fc14c0693e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fc16c0cd7d0 tx=0x7fc154016000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:18.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:18.825+0000 7fc1637fe700 1 -- 172.21.15.98:0/1647541114 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc15c051380 con 0x7fc16c0ad210 2023-10-06T05:08:19.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.012+0000 7fc17172f700 1 -- 172.21.15.98:0/1647541114 --> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7fc16c0c3290 con 0x7fc14c066f30 2023-10-06T05:08:19.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.012+0000 7fc1637fe700 1 -- 172.21.15.98:0/1647541114 <== mgr.4104 v2:172.21.15.98:6832/90397 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fc15400bb80 con 0x7fc14c066f30 2023-10-06T05:08:19.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.014+0000 7fc17172f700 1 -- 172.21.15.98:0/1647541114 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fc14c066f30 msgr2=0x7fc14c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:19.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.014+0000 7fc17172f700 1 --2- 172.21.15.98:0/1647541114 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fc14c066f30 0x7fc14c0693e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fc16c0cd7d0 tx=0x7fc154016000 comp rx=0 tx=0).stop 2023-10-06T05:08:19.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.014+0000 7fc17172f700 1 -- 172.21.15.98:0/1647541114 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc16c0ad210 msgr2=0x7fc16c0c5280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:19.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.014+0000 7fc17172f700 1 --2- 172.21.15.98:0/1647541114 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc16c0ad210 0x7fc16c0c5280 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fc15c00b870 tx=0x7fc15c009250 comp rx=0 tx=0).stop 2023-10-06T05:08:19.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.014+0000 7fc17172f700 1 -- 172.21.15.98:0/1647541114 shutdown_connections 2023-10-06T05:08:19.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.014+0000 7fc17172f700 1 --2- 172.21.15.98:0/1647541114 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc16c15d940 0x7fc16c0c35e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:19.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.014+0000 7fc17172f700 1 --2- 172.21.15.98:0/1647541114 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fc14c066f30 0x7fc14c0693e0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:19.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.014+0000 7fc17172f700 1 --2- 172.21.15.98:0/1647541114 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc16c0ad210 0x7fc16c0c5280 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:19.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.014+0000 7fc17172f700 1 --2- 172.21.15.98:0/1647541114 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc16c0ad5e0 0x7fc16c06c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:19.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.014+0000 7fc17172f700 1 -- 172.21.15.98:0/1647541114 >> 172.21.15.98:0/1647541114 conn(0x7fc16c067d90 msgr2=0x7fc16c06b200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:19.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.014+0000 7fc17172f700 1 -- 172.21.15.98:0/1647541114 shutdown_connections 2023-10-06T05:08:19.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.014+0000 7fc17172f700 1 -- 172.21.15.98:0/1647541114 wait complete. 2023-10-06T05:08:19.018 INFO:teuthology.orchestra.run.smithi098.stderr:instructed osd(s) 3 to deep-scrub 2023-10-06T05:08:19.028 INFO:tasks.ceph:Scrubbing osd.4 2023-10-06T05:08:19.028 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2023-10-06T05:08:19.146 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.144+0000 7faac50a5700 1 Processor -- start 2023-10-06T05:08:19.146 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.144+0000 7faac50a5700 1 -- start start 2023-10-06T05:08:19.148 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.145+0000 7faac50a5700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faac006ece0 0x7faac00b2360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:19.148 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.145+0000 7faac50a5700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7faac00aede0 0x7faac00af240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:19.149 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.145+0000 7faac50a5700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faac00b2ce0 con 0x7faac006f0b0 2023-10-06T05:08:19.149 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.145+0000 7faac50a5700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faac006fb40 con 0x7faac00aede0 2023-10-06T05:08:19.149 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.145+0000 7faac50a5700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faac00b0410 con 0x7faac006ece0 2023-10-06T05:08:19.149 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.145+0000 7faabf59e700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7faac00aede0 0x7faac00af240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:19.149 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.145+0000 7faab659c700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7faac006f0b0 0x7faac00b08a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:38794/0 (socket says 172.21.15.98:38794) 2023-10-06T05:08:19.149 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.145+0000 7faab659c700 1 -- 172.21.15.98:0/3262096070 learned_addr learned my addr 172.21.15.98:0/3262096070 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:08:19.149 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.145+0000 7faabf59e700 1 -- 172.21.15.98:0/3262096070 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faac006ece0 msgr2=0x7faac00b2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:19.149 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.145+0000 7faabed9d700 1 --2- 172.21.15.98:0/3262096070 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faac006ece0 0x7faac00b2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:19.149 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.145+0000 7faabf59e700 1 --2- 172.21.15.98:0/3262096070 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faac006ece0 0x7faac00b2360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:19.149 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.145+0000 7faabf59e700 1 -- 172.21.15.98:0/3262096070 >> v1:172.21.15.98:6789/0 conn(0x7faac006f0b0 legacy=0x7faac00b08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:19.149 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.145+0000 7faabf59e700 1 -- 172.21.15.98:0/3262096070 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faac00af780 con 0x7faac00aede0 2023-10-06T05:08:19.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.145+0000 7faabe59c700 1 -- 172.21.15.98:0/3262096070 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 859144968 0 0) 0x7faac00b2ce0 con 0x7faac006f0b0 2023-10-06T05:08:19.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.145+0000 7faabed9d700 1 --2- 172.21.15.98:0/3262096070 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faac006ece0 0x7faac00b2360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:08:19.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.146+0000 7faabf59e700 1 --2- 172.21.15.98:0/3262096070 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faac00aede0 0x7faac00af240 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7faab8009e60 tx=0x7faab8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=493711bcd7ffe7af server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:19.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.146+0000 7faabe59c700 1 -- 172.21.15.98:0/3262096070 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faab800e070 con 0x7faac00aede0 2023-10-06T05:08:19.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.146+0000 7faabe59c700 1 -- 172.21.15.98:0/3262096070 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faab800bc30 con 0x7faac00aede0 2023-10-06T05:08:19.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.146+0000 7faabe59c700 1 -- 172.21.15.98:0/3262096070 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faab8012430 con 0x7faac00aede0 2023-10-06T05:08:19.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.146+0000 7faac50a5700 1 -- 172.21.15.98:0/3262096070 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faac00aede0 msgr2=0x7faac00af240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:19.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.146+0000 7faac50a5700 1 --2- 172.21.15.98:0/3262096070 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faac00aede0 0x7faac00af240 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7faab8009e60 tx=0x7faab8009780 comp rx=0 tx=0).stop 2023-10-06T05:08:19.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.146+0000 7faac50a5700 1 -- 172.21.15.98:0/3262096070 shutdown_connections 2023-10-06T05:08:19.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.146+0000 7faac50a5700 1 --2- 172.21.15.98:0/3262096070 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faac00aede0 0x7faac00af240 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:19.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.146+0000 7faac50a5700 1 --2- 172.21.15.98:0/3262096070 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faac006ece0 0x7faac00b2360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:19.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.146+0000 7faac50a5700 1 -- 172.21.15.98:0/3262096070 >> 172.21.15.98:0/3262096070 conn(0x7faac00682f0 msgr2=0x7faac00686f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:19.151 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.146+0000 7faac50a5700 1 -- 172.21.15.98:0/3262096070 shutdown_connections 2023-10-06T05:08:19.151 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.146+0000 7faac50a5700 1 -- 172.21.15.98:0/3262096070 wait complete. 2023-10-06T05:08:19.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.148+0000 7faac50a5700 1 Processor -- start 2023-10-06T05:08:19.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.148+0000 7faac50a5700 1 -- start start 2023-10-06T05:08:19.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.148+0000 7faac50a5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faac006ece0 0x7faac00c9d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:19.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.148+0000 7faac50a5700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faac006f0b0 0x7faac01699f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:19.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.148+0000 7faac50a5700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faac00aede0 0x7faac01597b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:19.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.148+0000 7faac50a5700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faac0156480 con 0x7faac006ece0 2023-10-06T05:08:19.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.148+0000 7faac50a5700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faac015bde0 con 0x7faac006f0b0 2023-10-06T05:08:19.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.148+0000 7faac50a5700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faac015c0c0 con 0x7faac00aede0 2023-10-06T05:08:19.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.148+0000 7faabed9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faac006ece0 0x7faac00c9d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:19.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.148+0000 7faabed9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faac006ece0 0x7faac00c9d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36496/0 (socket says 172.21.15.98:36496) 2023-10-06T05:08:19.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.148+0000 7faabed9d700 1 -- 172.21.15.98:0/3060222523 learned_addr learned my addr 172.21.15.98:0/3060222523 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:08:19.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.149+0000 7faabed9d700 1 -- 172.21.15.98:0/3060222523 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faac00aede0 msgr2=0x7faac01597b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:19.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.149+0000 7faab659c700 1 --2- 172.21.15.98:0/3060222523 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faac006f0b0 0x7faac01699f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:19.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.149+0000 7faabf59e700 1 --2- 172.21.15.98:0/3060222523 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faac00aede0 0x7faac01597b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:19.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.149+0000 7faabed9d700 1 --2- 172.21.15.98:0/3060222523 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faac00aede0 0x7faac01597b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:19.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.149+0000 7faabed9d700 1 -- 172.21.15.98:0/3060222523 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faac006f0b0 msgr2=0x7faac01699f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:19.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.149+0000 7faabed9d700 1 --2- 172.21.15.98:0/3060222523 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faac006f0b0 0x7faac01699f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:19.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.149+0000 7faabed9d700 1 -- 172.21.15.98:0/3060222523 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faab8009a20 con 0x7faac006ece0 2023-10-06T05:08:19.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.149+0000 7faabf59e700 1 --2- 172.21.15.98:0/3060222523 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faac00aede0 0x7faac01597b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:08:19.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.149+0000 7faab659c700 1 --2- 172.21.15.98:0/3060222523 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faac006f0b0 0x7faac01699f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:08:19.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.149+0000 7faabed9d700 1 --2- 172.21.15.98:0/3060222523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faac006ece0 0x7faac00c9d10 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7faab000c850 tx=0x7faab0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:19.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.149+0000 7faab77fe700 1 -- 172.21.15.98:0/3060222523 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faab0011600 con 0x7faac006ece0 2023-10-06T05:08:19.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.149+0000 7faab77fe700 1 -- 172.21.15.98:0/3060222523 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faab0011760 con 0x7faac006ece0 2023-10-06T05:08:19.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.149+0000 7faac50a5700 1 -- 172.21.15.98:0/3060222523 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faac015c3a0 con 0x7faac006ece0 2023-10-06T05:08:19.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.149+0000 7faab77fe700 1 -- 172.21.15.98:0/3060222523 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faab0011b00 con 0x7faac006ece0 2023-10-06T05:08:19.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.149+0000 7faac50a5700 1 -- 172.21.15.98:0/3060222523 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faac015c960 con 0x7faac006ece0 2023-10-06T05:08:19.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.150+0000 7faac50a5700 1 -- 172.21.15.98:0/3060222523 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7faac015cee0 con 0x7faac006ece0 2023-10-06T05:08:19.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.150+0000 7faab77fe700 1 -- 172.21.15.98:0/3060222523 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7faab00073b0 con 0x7faac006ece0 2023-10-06T05:08:19.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.151+0000 7faab77fe700 1 --2- 172.21.15.98:0/3060222523 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7faaa0066f30 0x7faaa00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:19.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.151+0000 7faab77fe700 1 -- 172.21.15.98:0/3060222523 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5612+0+0 (secure 0 0 0) 0x7faab00844f0 con 0x7faac006ece0 2023-10-06T05:08:19.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.151+0000 7faab659c700 1 --2- 172.21.15.98:0/3060222523 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7faaa0066f30 0x7faaa00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:19.157 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.151+0000 7faab77fe700 1 --2- 172.21.15.98:0/3060222523 >> [v2:172.21.15.174:6800/90847,v1:172.21.15.174:6801/90847] conn(0x7faaa006d850 0x7faaa006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:19.157 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.151+0000 7faab77fe700 1 -- 172.21.15.98:0/3060222523 --> [v2:172.21.15.174:6800/90847,v1:172.21.15.174:6801/90847] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7faaa0070370 con 0x7faaa006d850 2023-10-06T05:08:19.157 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.151+0000 7faab77fe700 1 -- 172.21.15.98:0/3060222523 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=46) v2 ==== 24+0+0 (secure 0 0 0) 0x7faab0086b30 con 0x7faac006ece0 2023-10-06T05:08:19.157 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.151+0000 7faabf59e700 1 --2- 172.21.15.98:0/3060222523 >> [v2:172.21.15.174:6800/90847,v1:172.21.15.174:6801/90847] conn(0x7faaa006d850 0x7faaa006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:19.157 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.151+0000 7faab659c700 1 --2- 172.21.15.98:0/3060222523 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7faaa0066f30 0x7faaa00693e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7faac0166ca0 tx=0x7faaa8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:19.157 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.151+0000 7faabf59e700 1 --2- 172.21.15.98:0/3060222523 >> [v2:172.21.15.174:6800/90847,v1:172.21.15.174:6801/90847] conn(0x7faaa006d850 0x7faaa006fca0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:19.157 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.152+0000 7faab77fe700 1 -- 172.21.15.98:0/3060222523 <== osd.4 v2:172.21.15.174:6800/90847 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7faaa0070370 con 0x7faaa006d850 2023-10-06T05:08:19.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.166+0000 7faac50a5700 1 -- 172.21.15.98:0/3060222523 --> [v2:172.21.15.174:6800/90847,v1:172.21.15.174:6801/90847] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7faac0064a90 con 0x7faaa006d850 2023-10-06T05:08:19.168 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.166+0000 7faab77fe700 1 -- 172.21.15.98:0/3060222523 <== osd.4 v2:172.21.15.174:6800/90847 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7faac0064a90 con 0x7faaa006d850 2023-10-06T05:08:19.168 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-10-06T05:08:19.168 INFO:teuthology.orchestra.run.smithi098.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-06T05:08:19.169 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-10-06T05:08:19.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.167+0000 7faac50a5700 1 -- 172.21.15.98:0/3060222523 >> [v2:172.21.15.174:6800/90847,v1:172.21.15.174:6801/90847] conn(0x7faaa006d850 msgr2=0x7faaa006fca0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:19.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.167+0000 7faac50a5700 1 --2- 172.21.15.98:0/3060222523 >> [v2:172.21.15.174:6800/90847,v1:172.21.15.174:6801/90847] conn(0x7faaa006d850 0x7faaa006fca0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:19.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.167+0000 7faac50a5700 1 -- 172.21.15.98:0/3060222523 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7faaa0066f30 msgr2=0x7faaa00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:19.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.167+0000 7faac50a5700 1 --2- 172.21.15.98:0/3060222523 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7faaa0066f30 0x7faaa00693e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7faac0166ca0 tx=0x7faaa8009040 comp rx=0 tx=0).stop 2023-10-06T05:08:19.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.167+0000 7faac50a5700 1 -- 172.21.15.98:0/3060222523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faac006ece0 msgr2=0x7faac00c9d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:19.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.167+0000 7faac50a5700 1 --2- 172.21.15.98:0/3060222523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faac006ece0 0x7faac00c9d10 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7faab000c850 tx=0x7faab0008040 comp rx=0 tx=0).stop 2023-10-06T05:08:19.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.167+0000 7faabf59e700 1 -- 172.21.15.98:0/3060222523 reap_dead start 2023-10-06T05:08:19.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.167+0000 7faac50a5700 1 -- 172.21.15.98:0/3060222523 shutdown_connections 2023-10-06T05:08:19.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.167+0000 7faac50a5700 1 -- 172.21.15.98:0/3060222523 >> 172.21.15.98:0/3060222523 conn(0x7faac00682f0 msgr2=0x7faac00686c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:19.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.167+0000 7faac50a5700 1 -- 172.21.15.98:0/3060222523 shutdown_connections 2023-10-06T05:08:19.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.167+0000 7faac50a5700 1 -- 172.21.15.98:0/3060222523 wait complete. 2023-10-06T05:08:19.180 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2023-10-06T05:08:19.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.293+0000 7face9b3f700 1 Processor -- start 2023-10-06T05:08:19.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.293+0000 7face9b3f700 1 -- start start 2023-10-06T05:08:19.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.294+0000 7face9b3f700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7face4159560 0x7face40aa530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:19.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.294+0000 7face9b3f700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7face40b0760 0x7face40b1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:19.297 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.294+0000 7face9b3f700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7face40b0c60 con 0x7face40b49a0 2023-10-06T05:08:19.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.294+0000 7face9b3f700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7face40b0f40 con 0x7face4159560 2023-10-06T05:08:19.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.294+0000 7face9b3f700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7face40b1220 con 0x7face40b0760 2023-10-06T05:08:19.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.294+0000 7face37fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7face4159560 0x7face40aa530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:19.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.294+0000 7face37fe700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7face4159560 0x7face40aa530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36498/0 (socket says 172.21.15.98:36498) 2023-10-06T05:08:19.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.294+0000 7face37fe700 1 -- 172.21.15.98:0/3041245530 learned_addr learned my addr 172.21.15.98:0/3041245530 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:08:19.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.294+0000 7face3fff700 1 --2- 172.21.15.98:0/3041245530 >> v2:172.21.15.174:3300/0 conn(0x7face40b0760 0x7face40b1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:19.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.294+0000 7face37fe700 1 -- 172.21.15.98:0/3041245530 >> v2:172.21.15.174:3300/0 conn(0x7face40b0760 msgr2=0x7face40b1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:19.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.294+0000 7face37fe700 1 --2- 172.21.15.98:0/3041245530 >> v2:172.21.15.174:3300/0 conn(0x7face40b0760 0x7face40b1bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:19.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.294+0000 7face37fe700 1 -- 172.21.15.98:0/3041245530 >> v1:172.21.15.174:6789/0 conn(0x7face40b49a0 legacy=0x7face40aaa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:19.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.294+0000 7face37fe700 1 -- 172.21.15.98:0/3041245530 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7face40b1500 con 0x7face4159560 2023-10-06T05:08:19.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.294+0000 7face3fff700 1 --2- 172.21.15.98:0/3041245530 >> v2:172.21.15.174:3300/0 conn(0x7face40b0760 0x7face40b1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:08:19.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.294+0000 7face37fe700 1 --2- 172.21.15.98:0/3041245530 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7face4159560 0x7face40aa530 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7facd4009f70 tx=0x7facd4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31cfb2162e2b72c0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:19.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.294+0000 7face27fc700 1 -- 172.21.15.98:0/3041245530 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd400e070 con 0x7face4159560 2023-10-06T05:08:19.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.294+0000 7face27fc700 1 -- 172.21.15.98:0/3041245530 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facd400ba50 con 0x7face4159560 2023-10-06T05:08:19.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.295+0000 7face27fc700 1 -- 172.21.15.98:0/3041245530 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd40092e0 con 0x7face4159560 2023-10-06T05:08:19.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.295+0000 7face9b3f700 1 -- 172.21.15.98:0/3041245530 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7face4159560 msgr2=0x7face40aa530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:19.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.295+0000 7face9b3f700 1 --2- 172.21.15.98:0/3041245530 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7face4159560 0x7face40aa530 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7facd4009f70 tx=0x7facd4009840 comp rx=0 tx=0).stop 2023-10-06T05:08:19.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.295+0000 7face9b3f700 1 -- 172.21.15.98:0/3041245530 shutdown_connections 2023-10-06T05:08:19.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.295+0000 7face9b3f700 1 --2- 172.21.15.98:0/3041245530 >> v2:172.21.15.174:3300/0 conn(0x7face40b0760 0x7face40b1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:19.300 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.295+0000 7face9b3f700 1 --2- 172.21.15.98:0/3041245530 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7face4159560 0x7face40aa530 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:19.300 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.295+0000 7face9b3f700 1 -- 172.21.15.98:0/3041245530 >> 172.21.15.98:0/3041245530 conn(0x7face40997f0 msgr2=0x7face406f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:19.300 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.295+0000 7face9b3f700 1 -- 172.21.15.98:0/3041245530 shutdown_connections 2023-10-06T05:08:19.300 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.295+0000 7face9b3f700 1 -- 172.21.15.98:0/3041245530 wait complete. 2023-10-06T05:08:19.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.296+0000 7face9b3f700 1 Processor -- start 2023-10-06T05:08:19.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.296+0000 7face9b3f700 1 -- start start 2023-10-06T05:08:19.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.297+0000 7face9b3f700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7face40b0760 0x7face4157540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:19.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.297+0000 7face9b3f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7face40b49a0 0x7face4169c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:19.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.297+0000 7face9b3f700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7face4159560 0x7face409a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:19.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.297+0000 7face9b3f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7face416ab30 con 0x7face40b49a0 2023-10-06T05:08:19.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.297+0000 7face9b3f700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7face409a3f0 con 0x7face4159560 2023-10-06T05:08:19.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.297+0000 7face9b3f700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7face416a570 con 0x7face40b0760 2023-10-06T05:08:19.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.297+0000 7face2ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7face40b49a0 0x7face4169c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:19.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.297+0000 7face2ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7face40b49a0 0x7face4169c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36508/0 (socket says 172.21.15.98:36508) 2023-10-06T05:08:19.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.297+0000 7face2ffd700 1 -- 172.21.15.98:0/506322446 learned_addr learned my addr 172.21.15.98:0/506322446 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:08:19.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.297+0000 7face37fe700 1 --2- 172.21.15.98:0/506322446 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7face40b0760 0x7face4157540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:19.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.297+0000 7face3fff700 1 --2- 172.21.15.98:0/506322446 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7face4159560 0x7face409a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:19.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.297+0000 7face2ffd700 1 -- 172.21.15.98:0/506322446 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7face40b0760 msgr2=0x7face4157540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:19.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.297+0000 7face2ffd700 1 --2- 172.21.15.98:0/506322446 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7face40b0760 0x7face4157540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:19.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.297+0000 7face2ffd700 1 -- 172.21.15.98:0/506322446 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7face4159560 msgr2=0x7face409a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:19.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.297+0000 7face2ffd700 1 --2- 172.21.15.98:0/506322446 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7face4159560 0x7face409a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:19.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.297+0000 7face2ffd700 1 -- 172.21.15.98:0/506322446 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facd4009ae0 con 0x7face40b49a0 2023-10-06T05:08:19.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.298+0000 7face2ffd700 1 --2- 172.21.15.98:0/506322446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7face40b49a0 0x7face4169c00 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7facd800c5b0 tx=0x7facd8007f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:19.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.298+0000 7faccb7fe700 1 -- 172.21.15.98:0/506322446 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd8016070 con 0x7face40b49a0 2023-10-06T05:08:19.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.298+0000 7faccb7fe700 1 -- 172.21.15.98:0/506322446 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facd8006820 con 0x7face40b49a0 2023-10-06T05:08:19.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.298+0000 7face9b3f700 1 -- 172.21.15.98:0/506322446 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7face416a850 con 0x7face40b49a0 2023-10-06T05:08:19.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.298+0000 7faccb7fe700 1 -- 172.21.15.98:0/506322446 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd8006be0 con 0x7face40b49a0 2023-10-06T05:08:19.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.298+0000 7face9b3f700 1 -- 172.21.15.98:0/506322446 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7face4157a80 con 0x7face40b49a0 2023-10-06T05:08:19.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.299+0000 7face9b3f700 1 -- 172.21.15.98:0/506322446 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7face408aa90 con 0x7face40b49a0 2023-10-06T05:08:19.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.299+0000 7faccb7fe700 1 -- 172.21.15.98:0/506322446 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7facd8007020 con 0x7face40b49a0 2023-10-06T05:08:19.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.300+0000 7faccb7fe700 1 --2- 172.21.15.98:0/506322446 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7facd0066f30 0x7facd00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:19.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.300+0000 7faccb7fe700 1 -- 172.21.15.98:0/506322446 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5612+0+0 (secure 0 0 0) 0x7facd8083d30 con 0x7face40b49a0 2023-10-06T05:08:19.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.300+0000 7face37fe700 1 --2- 172.21.15.98:0/506322446 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7facd0066f30 0x7facd00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:19.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.300+0000 7face37fe700 1 --2- 172.21.15.98:0/506322446 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7facd0066f30 0x7facd00693e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7facd4009880 tx=0x7facd4017000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:19.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.304+0000 7faccb7fe700 1 -- 172.21.15.98:0/506322446 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7facd8051000 con 0x7face40b49a0 2023-10-06T05:08:19.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.485+0000 7face9b3f700 1 -- 172.21.15.98:0/506322446 --> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7face4064a90 con 0x7facd0066f30 2023-10-06T05:08:19.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.486+0000 7faccb7fe700 1 -- 172.21.15.98:0/506322446 <== mgr.4104 v2:172.21.15.98:6832/90397 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7face4064a90 con 0x7facd0066f30 2023-10-06T05:08:19.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.488+0000 7face9b3f700 1 -- 172.21.15.98:0/506322446 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7facd0066f30 msgr2=0x7facd00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:19.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.488+0000 7face9b3f700 1 --2- 172.21.15.98:0/506322446 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7facd0066f30 0x7facd00693e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7facd4009880 tx=0x7facd4017000 comp rx=0 tx=0).stop 2023-10-06T05:08:19.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.488+0000 7face9b3f700 1 -- 172.21.15.98:0/506322446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7face40b49a0 msgr2=0x7face4169c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:19.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.488+0000 7face9b3f700 1 --2- 172.21.15.98:0/506322446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7face40b49a0 0x7face4169c00 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7facd800c5b0 tx=0x7facd8007f20 comp rx=0 tx=0).stop 2023-10-06T05:08:19.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.488+0000 7face9b3f700 1 -- 172.21.15.98:0/506322446 shutdown_connections 2023-10-06T05:08:19.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.488+0000 7face9b3f700 1 --2- 172.21.15.98:0/506322446 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7face4159560 0x7face409a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:19.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.488+0000 7face9b3f700 1 --2- 172.21.15.98:0/506322446 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7face40b0760 0x7face4157540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:19.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.488+0000 7face9b3f700 1 --2- 172.21.15.98:0/506322446 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7facd0066f30 0x7facd00693e0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:19.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.488+0000 7face9b3f700 1 --2- 172.21.15.98:0/506322446 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7face40b49a0 0x7face4169c00 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:19.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.488+0000 7face9b3f700 1 -- 172.21.15.98:0/506322446 >> 172.21.15.98:0/506322446 conn(0x7face40997f0 msgr2=0x7face40aa660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:19.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.488+0000 7face9b3f700 1 -- 172.21.15.98:0/506322446 shutdown_connections 2023-10-06T05:08:19.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.488+0000 7face9b3f700 1 -- 172.21.15.98:0/506322446 wait complete. 2023-10-06T05:08:19.492 INFO:teuthology.orchestra.run.smithi098.stderr:instructed osd(s) 4 to deep-scrub 2023-10-06T05:08:19.502 INFO:tasks.ceph:Scrubbing osd.5 2023-10-06T05:08:19.502 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2023-10-06T05:08:19.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.617+0000 7f9409274700 1 Processor -- start 2023-10-06T05:08:19.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.617+0000 7f9409274700 1 -- start start 2023-10-06T05:08:19.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.618+0000 7f9409274700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9404153990 0x7f940415a370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:19.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.618+0000 7f9409274700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f94040cca70 0x7f94040d02b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:19.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.618+0000 7f9409274700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94040d0c50 con 0x7f94040d07f0 2023-10-06T05:08:19.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.618+0000 7f9409274700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94040cdbf0 con 0x7f94040cca70 2023-10-06T05:08:19.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.618+0000 7f9409274700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94040cb4c0 con 0x7f9404153990 2023-10-06T05:08:19.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.618+0000 7f94037fe700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f94040cca70 0x7f94040d02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:19.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.618+0000 7f9403fff700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9404153990 0x7f940415a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:19.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.618+0000 7f9408a73700 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f94040d07f0 0x7f940415cf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.98:56432/0 (socket says 172.21.15.98:56432) 2023-10-06T05:08:19.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.618+0000 7f94037fe700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f94040cca70 0x7f94040d02b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.98:56558/0 (socket says 172.21.15.98:56558) 2023-10-06T05:08:19.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.618+0000 7f9408a73700 1 -- 172.21.15.98:0/1523507655 learned_addr learned my addr 172.21.15.98:0/1523507655 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:08:19.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.619+0000 7f9403fff700 1 -- 172.21.15.98:0/1523507655 >> v2:172.21.15.174:3300/0 conn(0x7f94040cca70 msgr2=0x7f94040d02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:19.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.619+0000 7f9403fff700 1 --2- 172.21.15.98:0/1523507655 >> v2:172.21.15.174:3300/0 conn(0x7f94040cca70 0x7f94040d02b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:19.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.619+0000 7f9403fff700 1 -- 172.21.15.98:0/1523507655 >> v1:172.21.15.174:6789/0 conn(0x7f94040d07f0 legacy=0x7f940415cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:19.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.619+0000 7f9403fff700 1 -- 172.21.15.98:0/1523507655 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f940415a8b0 con 0x7f9404153990 2023-10-06T05:08:19.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.619+0000 7f94037fe700 1 --2- 172.21.15.98:0/1523507655 >> v2:172.21.15.174:3300/0 conn(0x7f94040cca70 0x7f94040d02b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T05:08:19.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.619+0000 7f9403fff700 1 --2- 172.21.15.98:0/1523507655 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9404153990 0x7f940415a370 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f93f40098e0 tx=0x7f93f40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b5344c8424487137 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:19.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.619+0000 7f9402ffd700 1 -- 172.21.15.98:0/1523507655 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93f400b640 con 0x7f9404153990 2023-10-06T05:08:19.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.619+0000 7f9402ffd700 1 -- 172.21.15.98:0/1523507655 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93f400b7a0 con 0x7f9404153990 2023-10-06T05:08:19.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.619+0000 7f9402ffd700 1 -- 172.21.15.98:0/1523507655 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93f400bb40 con 0x7f9404153990 2023-10-06T05:08:19.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.619+0000 7f9409274700 1 -- 172.21.15.98:0/1523507655 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9404153990 msgr2=0x7f940415a370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:19.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.619+0000 7f9409274700 1 --2- 172.21.15.98:0/1523507655 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9404153990 0x7f940415a370 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f93f40098e0 tx=0x7f93f40091a0 comp rx=0 tx=0).stop 2023-10-06T05:08:19.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.620+0000 7f9409274700 1 -- 172.21.15.98:0/1523507655 shutdown_connections 2023-10-06T05:08:19.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.620+0000 7f9409274700 1 --2- 172.21.15.98:0/1523507655 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9404153990 0x7f940415a370 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:19.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.620+0000 7f9409274700 1 --2- 172.21.15.98:0/1523507655 >> v2:172.21.15.174:3300/0 conn(0x7f94040cca70 0x7f94040d02b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:19.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.620+0000 7f9409274700 1 -- 172.21.15.98:0/1523507655 >> 172.21.15.98:0/1523507655 conn(0x7f94040680c0 msgr2=0x7f9404068490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:19.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.620+0000 7f9409274700 1 -- 172.21.15.98:0/1523507655 shutdown_connections 2023-10-06T05:08:19.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.620+0000 7f9409274700 1 -- 172.21.15.98:0/1523507655 wait complete. 2023-10-06T05:08:19.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.621+0000 7f9409274700 1 Processor -- start 2023-10-06T05:08:19.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.621+0000 7f9409274700 1 -- start start 2023-10-06T05:08:19.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.621+0000 7f9409274700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94040cca70 0x7f94040c85b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:19.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.621+0000 7f9409274700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f94040d07f0 0x7f94041517d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:19.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.621+0000 7f9409274700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9404153990 0x7f94040b30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:19.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.621+0000 7f9409274700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94040b5670 con 0x7f94040cca70 2023-10-06T05:08:19.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.621+0000 7f9409274700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94040c9980 con 0x7f9404153990 2023-10-06T05:08:19.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.621+0000 7f9409274700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9404151d10 con 0x7f94040d07f0 2023-10-06T05:08:19.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.622+0000 7f9403fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94040cca70 0x7f94040c85b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:19.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.622+0000 7f9403fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94040cca70 0x7f94040c85b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36518/0 (socket says 172.21.15.98:36518) 2023-10-06T05:08:19.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.622+0000 7f9403fff700 1 -- 172.21.15.98:0/1503250019 learned_addr learned my addr 172.21.15.98:0/1503250019 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:08:19.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.622+0000 7f94037fe700 1 --2- 172.21.15.98:0/1503250019 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f94040d07f0 0x7f94041517d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:19.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.622+0000 7f9408a73700 1 --2- 172.21.15.98:0/1503250019 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9404153990 0x7f94040b30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:19.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.622+0000 7f9403fff700 1 -- 172.21.15.98:0/1503250019 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f94040d07f0 msgr2=0x7f94041517d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:19.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.622+0000 7f9403fff700 1 --2- 172.21.15.98:0/1503250019 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f94040d07f0 0x7f94041517d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:19.628 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.622+0000 7f9403fff700 1 -- 172.21.15.98:0/1503250019 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9404153990 msgr2=0x7f94040b30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:19.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.622+0000 7f9403fff700 1 --2- 172.21.15.98:0/1503250019 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9404153990 0x7f94040b30c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:19.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.622+0000 7f9403fff700 1 -- 172.21.15.98:0/1503250019 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93f8009420 con 0x7f94040cca70 2023-10-06T05:08:19.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.622+0000 7f9403fff700 1 --2- 172.21.15.98:0/1503250019 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94040cca70 0x7f94040c85b0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f94040cb600 tx=0x7f93f4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:19.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.622+0000 7f93ebfff700 1 -- 172.21.15.98:0/1503250019 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93f401bcd0 con 0x7f94040cca70 2023-10-06T05:08:19.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.622+0000 7f93ebfff700 1 -- 172.21.15.98:0/1503250019 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93f401be30 con 0x7f94040cca70 2023-10-06T05:08:19.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.622+0000 7f9409274700 1 -- 172.21.15.98:0/1503250019 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93f4009420 con 0x7f94040cca70 2023-10-06T05:08:19.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.622+0000 7f93ebfff700 1 -- 172.21.15.98:0/1503250019 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93f4011670 con 0x7f94040cca70 2023-10-06T05:08:19.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.622+0000 7f9409274700 1 -- 172.21.15.98:0/1503250019 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9404169c10 con 0x7f94040cca70 2023-10-06T05:08:19.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.623+0000 7f9409274700 1 -- 172.21.15.98:0/1503250019 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f940405f230 con 0x7f94040cca70 2023-10-06T05:08:19.630 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.623+0000 7f93ebfff700 1 -- 172.21.15.98:0/1503250019 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f93f40119e0 con 0x7f94040cca70 2023-10-06T05:08:19.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.624+0000 7f93ebfff700 1 --2- 172.21.15.98:0/1503250019 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f93f0066f30 0x7f93f00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:19.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.624+0000 7f93ebfff700 1 -- 172.21.15.98:0/1503250019 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5612+0+0 (secure 0 0 0) 0x7f93f4085940 con 0x7f94040cca70 2023-10-06T05:08:19.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.624+0000 7f94037fe700 1 --2- 172.21.15.98:0/1503250019 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f93f0066f30 0x7f93f00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:19.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.624+0000 7f93ebfff700 1 --2- 172.21.15.98:0/1503250019 >> [v2:172.21.15.174:6808/90848,v1:172.21.15.174:6809/90848] conn(0x7f93f006d850 0x7f93f006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:19.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.624+0000 7f93ebfff700 1 -- 172.21.15.98:0/1503250019 --> [v2:172.21.15.174:6808/90848,v1:172.21.15.174:6809/90848] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f93f0070370 con 0x7f93f006d850 2023-10-06T05:08:19.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.624+0000 7f93ebfff700 1 -- 172.21.15.98:0/1503250019 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=46) v2 ==== 24+0+0 (secure 0 0 0) 0x7f93f408a240 con 0x7f94040cca70 2023-10-06T05:08:19.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.624+0000 7f9408a73700 1 --2- 172.21.15.98:0/1503250019 >> [v2:172.21.15.174:6808/90848,v1:172.21.15.174:6809/90848] conn(0x7f93f006d850 0x7f93f006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:19.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.624+0000 7f94037fe700 1 --2- 172.21.15.98:0/1503250019 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f93f0066f30 0x7f93f00693e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f93f80093f0 tx=0x7f93f8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:19.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.625+0000 7f9408a73700 1 --2- 172.21.15.98:0/1503250019 >> [v2:172.21.15.174:6808/90848,v1:172.21.15.174:6809/90848] conn(0x7f93f006d850 0x7f93f006fca0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:19.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.625+0000 7f93ebfff700 1 -- 172.21.15.98:0/1503250019 <== osd.5 v2:172.21.15.174:6808/90848 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f93f0070370 con 0x7f93f006d850 2023-10-06T05:08:19.641 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.639+0000 7f9409274700 1 -- 172.21.15.98:0/1503250019 --> [v2:172.21.15.174:6808/90848,v1:172.21.15.174:6809/90848] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f94040ccfe0 con 0x7f93f006d850 2023-10-06T05:08:19.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.640+0000 7f93ebfff700 1 -- 172.21.15.98:0/1503250019 <== osd.5 v2:172.21.15.174:6808/90848 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f94040ccfe0 con 0x7f93f006d850 2023-10-06T05:08:19.642 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-10-06T05:08:19.642 INFO:teuthology.orchestra.run.smithi098.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-06T05:08:19.642 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-10-06T05:08:19.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.640+0000 7f9409274700 1 -- 172.21.15.98:0/1503250019 >> [v2:172.21.15.174:6808/90848,v1:172.21.15.174:6809/90848] conn(0x7f93f006d850 msgr2=0x7f93f006fca0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:19.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.640+0000 7f9409274700 1 --2- 172.21.15.98:0/1503250019 >> [v2:172.21.15.174:6808/90848,v1:172.21.15.174:6809/90848] conn(0x7f93f006d850 0x7f93f006fca0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:19.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.640+0000 7f9409274700 1 -- 172.21.15.98:0/1503250019 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f93f0066f30 msgr2=0x7f93f00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:19.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.640+0000 7f9409274700 1 --2- 172.21.15.98:0/1503250019 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f93f0066f30 0x7f93f00693e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f93f80093f0 tx=0x7f93f8008040 comp rx=0 tx=0).stop 2023-10-06T05:08:19.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.640+0000 7f9409274700 1 -- 172.21.15.98:0/1503250019 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94040cca70 msgr2=0x7f94040c85b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:19.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.640+0000 7f9409274700 1 --2- 172.21.15.98:0/1503250019 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94040cca70 0x7f94040c85b0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f94040cb600 tx=0x7f93f4017000 comp rx=0 tx=0).stop 2023-10-06T05:08:19.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.640+0000 7f9408a73700 1 -- 172.21.15.98:0/1503250019 reap_dead start 2023-10-06T05:08:19.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.640+0000 7f9409274700 1 -- 172.21.15.98:0/1503250019 shutdown_connections 2023-10-06T05:08:19.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.640+0000 7f9409274700 1 -- 172.21.15.98:0/1503250019 >> 172.21.15.98:0/1503250019 conn(0x7f94040680c0 msgr2=0x7f94040cf180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:19.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.640+0000 7f9409274700 1 -- 172.21.15.98:0/1503250019 shutdown_connections 2023-10-06T05:08:19.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.640+0000 7f9409274700 1 -- 172.21.15.98:0/1503250019 wait complete. 2023-10-06T05:08:19.654 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2023-10-06T05:08:19.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.766+0000 7efdf4e9a700 1 Processor -- start 2023-10-06T05:08:19.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.766+0000 7efdf4e9a700 1 -- start start 2023-10-06T05:08:19.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.766+0000 7efdf4e9a700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7efdf0159560 0x7efdf00aa530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:19.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.766+0000 7efdf4e9a700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efdf00b0760 0x7efdf00b1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:19.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.766+0000 7efdf4e9a700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdf00b0c60 con 0x7efdf00b49a0 2023-10-06T05:08:19.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.766+0000 7efdf4e9a700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdf00b0f40 con 0x7efdf0159560 2023-10-06T05:08:19.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.766+0000 7efdf4e9a700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdf00b1220 con 0x7efdf00b0760 2023-10-06T05:08:19.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.766+0000 7efde5d9b700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7efdf00b49a0 0x7efdf00aaa70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:38802/0 (socket says 172.21.15.98:38802) 2023-10-06T05:08:19.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.767+0000 7efde5d9b700 1 -- 172.21.15.98:0/1375951686 learned_addr learned my addr 172.21.15.98:0/1375951686 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:08:19.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.767+0000 7efdeed9d700 1 --2- 172.21.15.98:0/1375951686 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efdf00b0760 0x7efdf00b1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:19.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.767+0000 7efdee59c700 1 --2- 172.21.15.98:0/1375951686 >> v2:172.21.15.174:3300/0 conn(0x7efdf0159560 0x7efdf00aa530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:19.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.767+0000 7efdedd9b700 1 -- 172.21.15.98:0/1375951686 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2405043081 0 0) 0x7efdf00b0c60 con 0x7efdf00b49a0 2023-10-06T05:08:19.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.767+0000 7efdedd9b700 1 -- 172.21.15.98:0/1375951686 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efddc003570 con 0x7efdf00b49a0 2023-10-06T05:08:19.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.767+0000 7efdeed9d700 1 -- 172.21.15.98:0/1375951686 >> v2:172.21.15.174:3300/0 conn(0x7efdf0159560 msgr2=0x7efdf00aa530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:19.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.767+0000 7efdeed9d700 1 --2- 172.21.15.98:0/1375951686 >> v2:172.21.15.174:3300/0 conn(0x7efdf0159560 0x7efdf00aa530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:19.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.767+0000 7efdeed9d700 1 -- 172.21.15.98:0/1375951686 >> v1:172.21.15.98:6789/0 conn(0x7efdf00b49a0 legacy=0x7efdf00aaa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:19.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.767+0000 7efdeed9d700 1 -- 172.21.15.98:0/1375951686 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdf00b1500 con 0x7efdf00b0760 2023-10-06T05:08:19.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.767+0000 7efdeed9d700 1 --2- 172.21.15.98:0/1375951686 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efdf00b0760 0x7efdf00b1bc0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7efde8009e90 tx=0x7efde8009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec30e385ca92d97f server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:19.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.767+0000 7efdedd9b700 1 -- 172.21.15.98:0/1375951686 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde800e070 con 0x7efdf00b0760 2023-10-06T05:08:19.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.767+0000 7efdedd9b700 1 -- 172.21.15.98:0/1375951686 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efde800bb40 con 0x7efdf00b0760 2023-10-06T05:08:19.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.767+0000 7efdedd9b700 1 -- 172.21.15.98:0/1375951686 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde8012430 con 0x7efdf00b0760 2023-10-06T05:08:19.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.768+0000 7efdf4e9a700 1 -- 172.21.15.98:0/1375951686 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efdf00b0760 msgr2=0x7efdf00b1bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:19.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.768+0000 7efdf4e9a700 1 --2- 172.21.15.98:0/1375951686 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efdf00b0760 0x7efdf00b1bc0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7efde8009e90 tx=0x7efde8009780 comp rx=0 tx=0).stop 2023-10-06T05:08:19.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.768+0000 7efdf4e9a700 1 -- 172.21.15.98:0/1375951686 shutdown_connections 2023-10-06T05:08:19.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.768+0000 7efdf4e9a700 1 --2- 172.21.15.98:0/1375951686 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efdf00b0760 0x7efdf00b1bc0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:19.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.768+0000 7efdf4e9a700 1 --2- 172.21.15.98:0/1375951686 >> v2:172.21.15.174:3300/0 conn(0x7efdf0159560 0x7efdf00aa530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:19.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.768+0000 7efdf4e9a700 1 -- 172.21.15.98:0/1375951686 >> 172.21.15.98:0/1375951686 conn(0x7efdf00997f0 msgr2=0x7efdf006f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:19.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.768+0000 7efdf4e9a700 1 -- 172.21.15.98:0/1375951686 shutdown_connections 2023-10-06T05:08:19.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.768+0000 7efdf4e9a700 1 -- 172.21.15.98:0/1375951686 wait complete. 2023-10-06T05:08:19.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.769+0000 7efdf4e9a700 1 Processor -- start 2023-10-06T05:08:19.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.769+0000 7efdf4e9a700 1 -- start start 2023-10-06T05:08:19.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.770+0000 7efdf4e9a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdf00b0760 0x7efdf00aaa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:19.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.770+0000 7efdf4e9a700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efdf00b49a0 0x7efdf00aceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:19.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.770+0000 7efdf4e9a700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efdf0159560 0x7efdf0158530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:19.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.770+0000 7efdf4e9a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdf015c6a0 con 0x7efdf00b0760 2023-10-06T05:08:19.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.770+0000 7efdf4e9a700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdf00ac2a0 con 0x7efdf00b49a0 2023-10-06T05:08:19.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.770+0000 7efdf4e9a700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdf00ab690 con 0x7efdf0159560 2023-10-06T05:08:19.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.770+0000 7efdee59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdf00b0760 0x7efdf00aaa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:19.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.770+0000 7efdee59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdf00b0760 0x7efdf00aaa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36530/0 (socket says 172.21.15.98:36530) 2023-10-06T05:08:19.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.770+0000 7efdee59c700 1 -- 172.21.15.98:0/530004512 learned_addr learned my addr 172.21.15.98:0/530004512 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:08:19.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.770+0000 7efdeed9d700 1 --2- 172.21.15.98:0/530004512 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efdf0159560 0x7efdf0158530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:19.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.770+0000 7efde5d9b700 1 --2- 172.21.15.98:0/530004512 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efdf00b49a0 0x7efdf00aceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:19.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.770+0000 7efdee59c700 1 -- 172.21.15.98:0/530004512 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efdf0159560 msgr2=0x7efdf0158530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:19.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.770+0000 7efdee59c700 1 --2- 172.21.15.98:0/530004512 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efdf0159560 0x7efdf0158530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:19.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.770+0000 7efdee59c700 1 -- 172.21.15.98:0/530004512 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efdf00b49a0 msgr2=0x7efdf00aceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:19.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.770+0000 7efdee59c700 1 --2- 172.21.15.98:0/530004512 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efdf00b49a0 0x7efdf00aceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:19.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.770+0000 7efdee59c700 1 -- 172.21.15.98:0/530004512 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efde8009a20 con 0x7efdf00b0760 2023-10-06T05:08:19.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.770+0000 7efde5d9b700 1 --2- 172.21.15.98:0/530004512 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efdf00b49a0 0x7efdf00aceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:08:19.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.770+0000 7efdee59c700 1 --2- 172.21.15.98:0/530004512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdf00b0760 0x7efdf00aaa00 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7efde000ea00 tx=0x7efde0009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:19.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.771+0000 7efde6ffd700 1 -- 172.21.15.98:0/530004512 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde000c850 con 0x7efdf00b0760 2023-10-06T05:08:19.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.771+0000 7efde6ffd700 1 -- 172.21.15.98:0/530004512 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efde000c9b0 con 0x7efdf00b0760 2023-10-06T05:08:19.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.771+0000 7efdf4e9a700 1 -- 172.21.15.98:0/530004512 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdf00ab970 con 0x7efdf00b0760 2023-10-06T05:08:19.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.771+0000 7efde6ffd700 1 -- 172.21.15.98:0/530004512 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde000cd30 con 0x7efdf00b0760 2023-10-06T05:08:19.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.771+0000 7efdf4e9a700 1 -- 172.21.15.98:0/530004512 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdf00abf30 con 0x7efdf00b0760 2023-10-06T05:08:19.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.772+0000 7efdf4e9a700 1 -- 172.21.15.98:0/530004512 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdf00aafa0 con 0x7efdf00b0760 2023-10-06T05:08:19.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.772+0000 7efde6ffd700 1 -- 172.21.15.98:0/530004512 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7efde0013680 con 0x7efdf00b0760 2023-10-06T05:08:19.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.772+0000 7efde6ffd700 1 --2- 172.21.15.98:0/530004512 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7efdd0066f30 0x7efdd00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:19.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.772+0000 7efde6ffd700 1 -- 172.21.15.98:0/530004512 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5612+0+0 (secure 0 0 0) 0x7efde0084b50 con 0x7efdf00b0760 2023-10-06T05:08:19.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.772+0000 7efde5d9b700 1 --2- 172.21.15.98:0/530004512 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7efdd0066f30 0x7efdd00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:19.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.773+0000 7efde5d9b700 1 --2- 172.21.15.98:0/530004512 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7efdd0066f30 0x7efdd00693e0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7efdd8007fd0 tx=0x7efdd8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:19.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.776+0000 7efde6ffd700 1 -- 172.21.15.98:0/530004512 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7efde0051e20 con 0x7efdf00b0760 2023-10-06T05:08:19.963 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.961+0000 7efdf4e9a700 1 -- 172.21.15.98:0/530004512 --> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7efdf0064a90 con 0x7efdd0066f30 2023-10-06T05:08:20.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.961+0000 7efde6ffd700 1 -- 172.21.15.98:0/530004512 <== mgr.4104 v2:172.21.15.98:6832/90397 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7efdf0064a90 con 0x7efdd0066f30 2023-10-06T05:08:20.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.963+0000 7efdf4e9a700 1 -- 172.21.15.98:0/530004512 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7efdd0066f30 msgr2=0x7efdd00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:20.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.963+0000 7efdf4e9a700 1 --2- 172.21.15.98:0/530004512 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7efdd0066f30 0x7efdd00693e0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7efdd8007fd0 tx=0x7efdd8009040 comp rx=0 tx=0).stop 2023-10-06T05:08:20.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.963+0000 7efdf4e9a700 1 -- 172.21.15.98:0/530004512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdf00b0760 msgr2=0x7efdf00aaa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:20.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.963+0000 7efdf4e9a700 1 --2- 172.21.15.98:0/530004512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdf00b0760 0x7efdf00aaa00 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7efde000ea00 tx=0x7efde0009120 comp rx=0 tx=0).stop 2023-10-06T05:08:20.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.963+0000 7efdf4e9a700 1 -- 172.21.15.98:0/530004512 shutdown_connections 2023-10-06T05:08:20.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.963+0000 7efdf4e9a700 1 --2- 172.21.15.98:0/530004512 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efdf0159560 0x7efdf0158530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:20.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.963+0000 7efdf4e9a700 1 --2- 172.21.15.98:0/530004512 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7efdd0066f30 0x7efdd00693e0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:20.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.963+0000 7efdf4e9a700 1 --2- 172.21.15.98:0/530004512 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdf00b0760 0x7efdf00aaa00 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:20.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.963+0000 7efdf4e9a700 1 --2- 172.21.15.98:0/530004512 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efdf00b49a0 0x7efdf00aceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:20.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.963+0000 7efdf4e9a700 1 -- 172.21.15.98:0/530004512 >> 172.21.15.98:0/530004512 conn(0x7efdf00997f0 msgr2=0x7efdf00b3230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:20.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.963+0000 7efdf4e9a700 1 -- 172.21.15.98:0/530004512 shutdown_connections 2023-10-06T05:08:20.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:19.963+0000 7efdf4e9a700 1 -- 172.21.15.98:0/530004512 wait complete. 2023-10-06T05:08:20.004 INFO:teuthology.orchestra.run.smithi098.stderr:instructed osd(s) 5 to deep-scrub 2023-10-06T05:08:20.005 INFO:tasks.ceph:Scrubbing osd.6 2023-10-06T05:08:20.005 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2023-10-06T05:08:20.122 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.120+0000 7f6fff59e700 1 Processor -- start 2023-10-06T05:08:20.122 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.120+0000 7f6fff59e700 1 -- start start 2023-10-06T05:08:20.124 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.121+0000 7f6fff59e700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f70000cca70 0x7f70000d02b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:20.124 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.121+0000 7f6fff59e700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f70000d07f0 0x7f70000ce860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:20.125 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.121+0000 7f6fff59e700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70000d0c50 con 0x7f7000153990 2023-10-06T05:08:20.125 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.121+0000 7f6fff59e700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70000cdbf0 con 0x7f70000cca70 2023-10-06T05:08:20.125 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.121+0000 7f6fff59e700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70000cb4c0 con 0x7f70000d07f0 2023-10-06T05:08:20.125 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.121+0000 7f6ffdd9b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f70000cca70 0x7f70000d02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:20.125 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.121+0000 7f6ffdd9b700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f70000cca70 0x7f70000d02b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36542/0 (socket says 172.21.15.98:36542) 2023-10-06T05:08:20.125 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.121+0000 7f6ffdd9b700 1 -- 172.21.15.98:0/4239883896 learned_addr learned my addr 172.21.15.98:0/4239883896 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:08:20.125 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.121+0000 7f6ffed9d700 1 --2- 172.21.15.98:0/4239883896 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f70000d07f0 0x7f70000ce860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:20.125 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.121+0000 7f6ffdd9b700 1 -- 172.21.15.98:0/4239883896 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f70000d07f0 msgr2=0x7f70000ce860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:20.125 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.121+0000 7f6ffdd9b700 1 --2- 172.21.15.98:0/4239883896 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f70000d07f0 0x7f70000ce860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:20.125 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.121+0000 7f6ffdd9b700 1 -- 172.21.15.98:0/4239883896 >> v1:172.21.15.174:6789/0 conn(0x7f7000153990 legacy=0x7f700015a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:20.126 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.121+0000 7f6ffdd9b700 1 -- 172.21.15.98:0/4239883896 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70000ccfe0 con 0x7f70000cca70 2023-10-06T05:08:20.126 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.122+0000 7f6ffdd9b700 1 --2- 172.21.15.98:0/4239883896 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70000cca70 0x7f70000d02b0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f6ff40099b0 tx=0x7f6ff4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcec2e2877e6e02f server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:20.126 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.122+0000 7f6ffd59a700 1 -- 172.21.15.98:0/4239883896 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff400b820 con 0x7f70000cca70 2023-10-06T05:08:20.126 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.122+0000 7f6ffd59a700 1 -- 172.21.15.98:0/4239883896 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ff400b980 con 0x7f70000cca70 2023-10-06T05:08:20.126 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.122+0000 7f6ffd59a700 1 -- 172.21.15.98:0/4239883896 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff400bd20 con 0x7f70000cca70 2023-10-06T05:08:20.126 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.122+0000 7f6fff59e700 1 -- 172.21.15.98:0/4239883896 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70000cca70 msgr2=0x7f70000d02b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:20.127 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.122+0000 7f6fff59e700 1 --2- 172.21.15.98:0/4239883896 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70000cca70 0x7f70000d02b0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f6ff40099b0 tx=0x7f6ff4009250 comp rx=0 tx=0).stop 2023-10-06T05:08:20.127 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.122+0000 7f6fff59e700 1 -- 172.21.15.98:0/4239883896 shutdown_connections 2023-10-06T05:08:20.127 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.122+0000 7f6fff59e700 1 --2- 172.21.15.98:0/4239883896 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f70000d07f0 0x7f70000ce860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:20.127 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.122+0000 7f6fff59e700 1 --2- 172.21.15.98:0/4239883896 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70000cca70 0x7f70000d02b0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:20.127 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.122+0000 7f6fff59e700 1 -- 172.21.15.98:0/4239883896 >> 172.21.15.98:0/4239883896 conn(0x7f70000680c0 msgr2=0x7f7000068490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:20.127 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.122+0000 7f6fff59e700 1 -- 172.21.15.98:0/4239883896 shutdown_connections 2023-10-06T05:08:20.127 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.122+0000 7f6fff59e700 1 -- 172.21.15.98:0/4239883896 wait complete. 2023-10-06T05:08:20.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.123+0000 7f6fff59e700 1 Processor -- start 2023-10-06T05:08:20.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.123+0000 7f6fff59e700 1 -- start start 2023-10-06T05:08:20.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.124+0000 7f6fff59e700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f70000cca70 0x7f70000b30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:20.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.124+0000 7f6fff59e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70000d07f0 0x7f7000150d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:20.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.124+0000 7f6fff59e700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7000153990 0x7f700016cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:20.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.124+0000 7f6fff59e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70000b0760 con 0x7f70000d07f0 2023-10-06T05:08:20.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.124+0000 7f6fff59e700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7000157240 con 0x7f7000153990 2023-10-06T05:08:20.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.124+0000 7f6fff59e700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70000b3ad0 con 0x7f70000cca70 2023-10-06T05:08:20.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.124+0000 7f6ffdd9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70000d07f0 0x7f7000150d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:20.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.124+0000 7f6ffdd9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70000d07f0 0x7f7000150d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36556/0 (socket says 172.21.15.98:36556) 2023-10-06T05:08:20.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.124+0000 7f6ffdd9b700 1 -- 172.21.15.98:0/610296519 learned_addr learned my addr 172.21.15.98:0/610296519 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:08:20.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.124+0000 7f6ffed9d700 1 --2- 172.21.15.98:0/610296519 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7000153990 0x7f700016cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:20.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.124+0000 7f6ffe59c700 1 --2- 172.21.15.98:0/610296519 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f70000cca70 0x7f70000b30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:20.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.125+0000 7f6ffdd9b700 1 -- 172.21.15.98:0/610296519 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f70000cca70 msgr2=0x7f70000b30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:20.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.125+0000 7f6ffdd9b700 1 --2- 172.21.15.98:0/610296519 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f70000cca70 0x7f70000b30c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:20.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.125+0000 7f6ffdd9b700 1 -- 172.21.15.98:0/610296519 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7000153990 msgr2=0x7f700016cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:20.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.125+0000 7f6ffdd9b700 1 --2- 172.21.15.98:0/610296519 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7000153990 0x7f700016cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:20.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.125+0000 7f6ffdd9b700 1 -- 172.21.15.98:0/610296519 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ff40094f0 con 0x7f70000d07f0 2023-10-06T05:08:20.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.125+0000 7f6ffe59c700 1 --2- 172.21.15.98:0/610296519 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f70000cca70 0x7f70000b30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:08:20.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.125+0000 7f6ffdd9b700 1 --2- 172.21.15.98:0/610296519 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70000d07f0 0x7f7000150d20 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f6ff4009290 tx=0x7f6ff4016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:20.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.125+0000 7f6fee7fc700 1 -- 172.21.15.98:0/610296519 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff4011ce0 con 0x7f70000d07f0 2023-10-06T05:08:20.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.125+0000 7f6fee7fc700 1 -- 172.21.15.98:0/610296519 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ff4020070 con 0x7f70000d07f0 2023-10-06T05:08:20.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.125+0000 7f6fff59e700 1 -- 172.21.15.98:0/610296519 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70000b3db0 con 0x7f70000d07f0 2023-10-06T05:08:20.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.125+0000 7f6fee7fc700 1 -- 172.21.15.98:0/610296519 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff401b6a0 con 0x7f70000d07f0 2023-10-06T05:08:20.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.125+0000 7f6fff59e700 1 -- 172.21.15.98:0/610296519 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7000151260 con 0x7f70000d07f0 2023-10-06T05:08:20.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.126+0000 7f6fff59e700 1 -- 172.21.15.98:0/610296519 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f70000b42d0 con 0x7f70000d07f0 2023-10-06T05:08:20.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.126+0000 7f6fee7fc700 1 -- 172.21.15.98:0/610296519 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f6ff4022030 con 0x7f70000d07f0 2023-10-06T05:08:20.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.127+0000 7f6fee7fc700 1 --2- 172.21.15.98:0/610296519 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f6fe0066f30 0x7f6fe00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:20.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.127+0000 7f6fee7fc700 1 -- 172.21.15.98:0/610296519 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5612+0+0 (secure 0 0 0) 0x7f6ff4086750 con 0x7f70000d07f0 2023-10-06T05:08:20.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.127+0000 7f6ffe59c700 1 --2- 172.21.15.98:0/610296519 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f6fe0066f30 0x7f6fe00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:20.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.127+0000 7f6fee7fc700 1 --2- 172.21.15.98:0/610296519 >> [v2:172.21.15.174:6816/90877,v1:172.21.15.174:6817/90877] conn(0x7f6fe006d850 0x7f6fe006fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:20.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.127+0000 7f6fee7fc700 1 -- 172.21.15.98:0/610296519 --> [v2:172.21.15.174:6816/90877,v1:172.21.15.174:6817/90877] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6fe0070370 con 0x7f6fe006d850 2023-10-06T05:08:20.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.127+0000 7f6fee7fc700 1 -- 172.21.15.98:0/610296519 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=46) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6ff4086b20 con 0x7f70000d07f0 2023-10-06T05:08:20.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.127+0000 7f6ffe59c700 1 --2- 172.21.15.98:0/610296519 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f6fe0066f30 0x7f6fe00693e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f6ff0001370 tx=0x7f6ff0009680 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:20.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.127+0000 7f6ffed9d700 1 --2- 172.21.15.98:0/610296519 >> [v2:172.21.15.174:6816/90877,v1:172.21.15.174:6817/90877] conn(0x7f6fe006d850 0x7f6fe006fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:20.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.127+0000 7f6ffed9d700 1 --2- 172.21.15.98:0/610296519 >> [v2:172.21.15.174:6816/90877,v1:172.21.15.174:6817/90877] conn(0x7f6fe006d850 0x7f6fe006fca0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:20.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.128+0000 7f6fee7fc700 1 -- 172.21.15.98:0/610296519 <== osd.6 v2:172.21.15.174:6816/90877 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f6fe0070370 con 0x7f6fe006d850 2023-10-06T05:08:20.144 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.142+0000 7f6fff59e700 1 -- 172.21.15.98:0/610296519 --> [v2:172.21.15.174:6816/90877,v1:172.21.15.174:6817/90877] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f70000d0c50 con 0x7f6fe006d850 2023-10-06T05:08:20.144 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.143+0000 7f6fee7fc700 1 -- 172.21.15.98:0/610296519 <== osd.6 v2:172.21.15.174:6816/90877 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f70000d0c50 con 0x7f6fe006d850 2023-10-06T05:08:20.145 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-10-06T05:08:20.145 INFO:teuthology.orchestra.run.smithi098.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-06T05:08:20.145 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-10-06T05:08:20.146 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.143+0000 7f6fff59e700 1 -- 172.21.15.98:0/610296519 >> [v2:172.21.15.174:6816/90877,v1:172.21.15.174:6817/90877] conn(0x7f6fe006d850 msgr2=0x7f6fe006fca0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:20.146 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.143+0000 7f6fff59e700 1 --2- 172.21.15.98:0/610296519 >> [v2:172.21.15.174:6816/90877,v1:172.21.15.174:6817/90877] conn(0x7f6fe006d850 0x7f6fe006fca0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:20.146 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.143+0000 7f6fff59e700 1 -- 172.21.15.98:0/610296519 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f6fe0066f30 msgr2=0x7f6fe00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:20.146 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.143+0000 7f6fff59e700 1 --2- 172.21.15.98:0/610296519 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f6fe0066f30 0x7f6fe00693e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f6ff0001370 tx=0x7f6ff0009680 comp rx=0 tx=0).stop 2023-10-06T05:08:20.147 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.143+0000 7f6fff59e700 1 -- 172.21.15.98:0/610296519 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70000d07f0 msgr2=0x7f7000150d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:20.147 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.143+0000 7f6fff59e700 1 --2- 172.21.15.98:0/610296519 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70000d07f0 0x7f7000150d20 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f6ff4009290 tx=0x7f6ff4016000 comp rx=0 tx=0).stop 2023-10-06T05:08:20.147 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.143+0000 7f6ffed9d700 1 -- 172.21.15.98:0/610296519 reap_dead start 2023-10-06T05:08:20.147 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.143+0000 7f6fff59e700 1 -- 172.21.15.98:0/610296519 shutdown_connections 2023-10-06T05:08:20.147 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.143+0000 7f6fff59e700 1 -- 172.21.15.98:0/610296519 >> 172.21.15.98:0/610296519 conn(0x7f70000680c0 msgr2=0x7f7000157570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:20.147 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.143+0000 7f6fff59e700 1 -- 172.21.15.98:0/610296519 shutdown_connections 2023-10-06T05:08:20.147 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.143+0000 7f6fff59e700 1 -- 172.21.15.98:0/610296519 wait complete. 2023-10-06T05:08:20.157 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2023-10-06T05:08:20.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.269+0000 7ff62c8d4700 1 Processor -- start 2023-10-06T05:08:20.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.269+0000 7ff62c8d4700 1 -- start start 2023-10-06T05:08:20.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.270+0000 7ff62c8d4700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ff6280b49a0 0x7ff6280aac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:20.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.270+0000 7ff62c8d4700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6280b0c60 con 0x7ff6280b0760 2023-10-06T05:08:20.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.270+0000 7ff62c8d4700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6280b0f40 con 0x7ff628159560 2023-10-06T05:08:20.282 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.270+0000 7ff62c8d4700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6280b1220 con 0x7ff6280b49a0 2023-10-06T05:08:20.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.270+0000 7ff626d9d700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7ff6280b0760 0x7ff6280b1bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:38806/0 (socket says 172.21.15.98:38806) 2023-10-06T05:08:20.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.270+0000 7ff626d9d700 1 -- 172.21.15.98:0/1477980441 learned_addr learned my addr 172.21.15.98:0/1477980441 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:08:20.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.270+0000 7ff61dd9b700 1 --2- 172.21.15.98:0/1477980441 >> v2:172.21.15.174:3300/0 conn(0x7ff6280b49a0 0x7ff6280aac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:20.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.270+0000 7ff625d9b700 1 -- 172.21.15.98:0/1477980441 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2836306527 0 0) 0x7ff6280b0c60 con 0x7ff6280b0760 2023-10-06T05:08:20.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.270+0000 7ff625d9b700 1 -- 172.21.15.98:0/1477980441 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff614003570 con 0x7ff6280b0760 2023-10-06T05:08:20.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.270+0000 7ff61dd9b700 1 -- 172.21.15.98:0/1477980441 >> v1:172.21.15.174:6789/0 conn(0x7ff628159560 legacy=0x7ff6280aa530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:20.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.270+0000 7ff61dd9b700 1 -- 172.21.15.98:0/1477980441 >> v1:172.21.15.98:6789/0 conn(0x7ff6280b0760 legacy=0x7ff6280b1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:20.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.270+0000 7ff61dd9b700 1 -- 172.21.15.98:0/1477980441 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6280b1500 con 0x7ff6280b49a0 2023-10-06T05:08:20.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.270+0000 7ff61dd9b700 1 --2- 172.21.15.98:0/1477980441 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff6280b49a0 0x7ff6280aac40 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff610009910 tx=0x7ff6100091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d1d63a5847cb8ed0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:20.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.271+0000 7ff625d9b700 1 -- 172.21.15.98:0/1477980441 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff61000b640 con 0x7ff6280b49a0 2023-10-06T05:08:20.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.271+0000 7ff625d9b700 1 -- 172.21.15.98:0/1477980441 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff61000b7a0 con 0x7ff6280b49a0 2023-10-06T05:08:20.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.271+0000 7ff625d9b700 1 -- 172.21.15.98:0/1477980441 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff61000bb20 con 0x7ff6280b49a0 2023-10-06T05:08:20.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.271+0000 7ff62c8d4700 1 -- 172.21.15.98:0/1477980441 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff6280b49a0 msgr2=0x7ff6280aac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:20.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.271+0000 7ff62c8d4700 1 --2- 172.21.15.98:0/1477980441 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff6280b49a0 0x7ff6280aac40 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff610009910 tx=0x7ff6100091a0 comp rx=0 tx=0).stop 2023-10-06T05:08:20.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.271+0000 7ff62c8d4700 1 -- 172.21.15.98:0/1477980441 shutdown_connections 2023-10-06T05:08:20.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.271+0000 7ff62c8d4700 1 --2- 172.21.15.98:0/1477980441 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff6280b49a0 0x7ff6280aac40 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:20.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.271+0000 7ff62c8d4700 1 -- 172.21.15.98:0/1477980441 >> 172.21.15.98:0/1477980441 conn(0x7ff6280997f0 msgr2=0x7ff62806f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:20.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.271+0000 7ff62c8d4700 1 -- 172.21.15.98:0/1477980441 shutdown_connections 2023-10-06T05:08:20.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.271+0000 7ff62c8d4700 1 -- 172.21.15.98:0/1477980441 wait complete. 2023-10-06T05:08:20.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.272+0000 7ff62c8d4700 1 Processor -- start 2023-10-06T05:08:20.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.272+0000 7ff62c8d4700 1 -- start start 2023-10-06T05:08:20.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.273+0000 7ff62c8d4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6280b0760 0x7ff62815c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:20.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.273+0000 7ff62c8d4700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff6280b49a0 0x7ff62809d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:20.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.273+0000 7ff62c8d4700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff628159560 0x7ff628156a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:20.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.273+0000 7ff62c8d4700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff62815acd0 con 0x7ff6280b0760 2023-10-06T05:08:20.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.273+0000 7ff62c8d4700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff62815a3e0 con 0x7ff6280b49a0 2023-10-06T05:08:20.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.273+0000 7ff62c8d4700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff62815a680 con 0x7ff628159560 2023-10-06T05:08:20.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.273+0000 7ff62659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6280b0760 0x7ff62815c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:20.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.273+0000 7ff62659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6280b0760 0x7ff62815c0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36568/0 (socket says 172.21.15.98:36568) 2023-10-06T05:08:20.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.273+0000 7ff62659c700 1 -- 172.21.15.98:0/3381890495 learned_addr learned my addr 172.21.15.98:0/3381890495 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:08:20.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.273+0000 7ff61dd9b700 1 --2- 172.21.15.98:0/3381890495 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff6280b49a0 0x7ff62809d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:20.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.273+0000 7ff62659c700 1 -- 172.21.15.98:0/3381890495 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff628159560 msgr2=0x7ff628156a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:20.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.273+0000 7ff62659c700 1 --2- 172.21.15.98:0/3381890495 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff628159560 0x7ff628156a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:20.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.273+0000 7ff62659c700 1 -- 172.21.15.98:0/3381890495 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff6280b49a0 msgr2=0x7ff62809d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:20.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.273+0000 7ff62659c700 1 --2- 172.21.15.98:0/3381890495 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff6280b49a0 0x7ff62809d600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:20.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.273+0000 7ff62659c700 1 -- 172.21.15.98:0/3381890495 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff610009420 con 0x7ff6280b0760 2023-10-06T05:08:20.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.274+0000 7ff62659c700 1 --2- 172.21.15.98:0/3381890495 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6280b0760 0x7ff62815c0a0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7ff61800c4c0 tx=0x7ff61800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:20.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.274+0000 7ff61effd700 1 -- 172.21.15.98:0/3381890495 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff618011590 con 0x7ff6280b0760 2023-10-06T05:08:20.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.274+0000 7ff61effd700 1 -- 172.21.15.98:0/3381890495 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff618011710 con 0x7ff6280b0760 2023-10-06T05:08:20.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.274+0000 7ff62c8d4700 1 -- 172.21.15.98:0/3381890495 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff62815a960 con 0x7ff6280b0760 2023-10-06T05:08:20.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.274+0000 7ff61effd700 1 -- 172.21.15.98:0/3381890495 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff618011ad0 con 0x7ff6280b0760 2023-10-06T05:08:20.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.274+0000 7ff62c8d4700 1 -- 172.21.15.98:0/3381890495 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff62809de50 con 0x7ff6280b0760 2023-10-06T05:08:20.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.275+0000 7ff62c8d4700 1 -- 172.21.15.98:0/3381890495 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff628064a90 con 0x7ff6280b0760 2023-10-06T05:08:20.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.275+0000 7ff61effd700 1 -- 172.21.15.98:0/3381890495 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7ff61801b3b0 con 0x7ff6280b0760 2023-10-06T05:08:20.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.276+0000 7ff61effd700 1 --2- 172.21.15.98:0/3381890495 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7ff608066f30 0x7ff6080693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:20.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.276+0000 7ff61dd9b700 1 --2- 172.21.15.98:0/3381890495 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7ff608066f30 0x7ff6080693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:20.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.276+0000 7ff61effd700 1 -- 172.21.15.98:0/3381890495 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5612+0+0 (secure 0 0 0) 0x7ff618083870 con 0x7ff6280b0760 2023-10-06T05:08:20.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.276+0000 7ff61dd9b700 1 --2- 172.21.15.98:0/3381890495 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7ff608066f30 0x7ff6080693e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7ff610009910 tx=0x7ff610017000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:20.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.280+0000 7ff61effd700 1 -- 172.21.15.98:0/3381890495 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff618050bf0 con 0x7ff6280b0760 2023-10-06T05:08:20.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.464+0000 7ff62c8d4700 1 -- 172.21.15.98:0/3381890495 --> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7ff62815ae10 con 0x7ff608066f30 2023-10-06T05:08:20.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.465+0000 7ff61effd700 1 -- 172.21.15.98:0/3381890495 <== mgr.4104 v2:172.21.15.98:6832/90397 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7ff610000bc0 con 0x7ff608066f30 2023-10-06T05:08:20.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.467+0000 7ff62c8d4700 1 -- 172.21.15.98:0/3381890495 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7ff608066f30 msgr2=0x7ff6080693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:20.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.467+0000 7ff62c8d4700 1 --2- 172.21.15.98:0/3381890495 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7ff608066f30 0x7ff6080693e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7ff610009910 tx=0x7ff610017000 comp rx=0 tx=0).stop 2023-10-06T05:08:20.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.467+0000 7ff62c8d4700 1 -- 172.21.15.98:0/3381890495 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6280b0760 msgr2=0x7ff62815c0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:20.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.467+0000 7ff62c8d4700 1 --2- 172.21.15.98:0/3381890495 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6280b0760 0x7ff62815c0a0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7ff61800c4c0 tx=0x7ff61800c880 comp rx=0 tx=0).stop 2023-10-06T05:08:20.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.467+0000 7ff62c8d4700 1 -- 172.21.15.98:0/3381890495 shutdown_connections 2023-10-06T05:08:20.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.467+0000 7ff62c8d4700 1 --2- 172.21.15.98:0/3381890495 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff628159560 0x7ff628156a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:20.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.467+0000 7ff62c8d4700 1 --2- 172.21.15.98:0/3381890495 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7ff608066f30 0x7ff6080693e0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:20.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.467+0000 7ff62c8d4700 1 --2- 172.21.15.98:0/3381890495 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6280b0760 0x7ff62815c0a0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:20.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.467+0000 7ff62c8d4700 1 --2- 172.21.15.98:0/3381890495 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff6280b49a0 0x7ff62809d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:20.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.467+0000 7ff62c8d4700 1 -- 172.21.15.98:0/3381890495 >> 172.21.15.98:0/3381890495 conn(0x7ff6280997f0 msgr2=0x7ff6280cf2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:20.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.467+0000 7ff62c8d4700 1 -- 172.21.15.98:0/3381890495 shutdown_connections 2023-10-06T05:08:20.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.467+0000 7ff62c8d4700 1 -- 172.21.15.98:0/3381890495 wait complete. 2023-10-06T05:08:20.471 INFO:teuthology.orchestra.run.smithi098.stderr:instructed osd(s) 6 to deep-scrub 2023-10-06T05:08:20.481 INFO:tasks.ceph:Scrubbing osd.7 2023-10-06T05:08:20.481 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2023-10-06T05:08:20.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.597+0000 7feee359e700 1 Processor -- start 2023-10-06T05:08:20.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.597+0000 7feee359e700 1 -- start start 2023-10-06T05:08:20.602 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.598+0000 7feee359e700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7feee4153990 0x7feee415a370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:20.602 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.598+0000 7feee359e700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7feee40d09c0 0x7feee40ce860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:20.602 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.598+0000 7feee359e700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feee40cdbf0 con 0x7feee40cca70 2023-10-06T05:08:20.602 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.598+0000 7feee359e700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feee40cb4c0 con 0x7feee4153990 2023-10-06T05:08:20.602 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.598+0000 7feee359e700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feee415a8b0 con 0x7feee40d09c0 2023-10-06T05:08:20.602 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.598+0000 7feee259c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7feee4153990 0x7feee415a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:20.602 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.598+0000 7feed9d9b700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7feee40cca70 0x7feee40d02b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:38822/0 (socket says 172.21.15.98:38822) 2023-10-06T05:08:20.602 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.598+0000 7feed9d9b700 1 -- 172.21.15.98:0/792439966 learned_addr learned my addr 172.21.15.98:0/792439966 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:08:20.602 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.598+0000 7feee259c700 1 -- 172.21.15.98:0/792439966 >> v2:172.21.15.174:3300/0 conn(0x7feee40d09c0 msgr2=0x7feee40ce860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:20.602 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.598+0000 7feee2d9d700 1 --2- 172.21.15.98:0/792439966 >> v2:172.21.15.174:3300/0 conn(0x7feee40d09c0 0x7feee40ce860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:20.603 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.598+0000 7feee259c700 1 --2- 172.21.15.98:0/792439966 >> v2:172.21.15.174:3300/0 conn(0x7feee40d09c0 0x7feee40ce860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:20.603 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.598+0000 7feee259c700 1 -- 172.21.15.98:0/792439966 >> v1:172.21.15.98:6789/0 conn(0x7feee40cca70 legacy=0x7feee40d02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:20.603 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.598+0000 7feee1d9b700 1 -- 172.21.15.98:0/792439966 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3836078280 0 0) 0x7feee40cdbf0 con 0x7feee40cca70 2023-10-06T05:08:20.603 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.598+0000 7feee259c700 1 -- 172.21.15.98:0/792439966 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feee40ccfe0 con 0x7feee4153990 2023-10-06T05:08:20.603 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.599+0000 7feee259c700 1 --2- 172.21.15.98:0/792439966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feee4153990 0x7feee415a370 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7feee4069060 tx=0x7feed40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=455ef3cba434e914 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:20.603 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.599+0000 7feee1d9b700 1 -- 172.21.15.98:0/792439966 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed400b6b0 con 0x7feee4153990 2023-10-06T05:08:20.603 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.599+0000 7feee1d9b700 1 -- 172.21.15.98:0/792439966 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feed400b810 con 0x7feee4153990 2023-10-06T05:08:20.603 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.599+0000 7feee1d9b700 1 -- 172.21.15.98:0/792439966 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed400bbb0 con 0x7feee4153990 2023-10-06T05:08:20.603 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.599+0000 7feee359e700 1 -- 172.21.15.98:0/792439966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feee4153990 msgr2=0x7feee415a370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:20.603 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.599+0000 7feee359e700 1 --2- 172.21.15.98:0/792439966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feee4153990 0x7feee415a370 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7feee4069060 tx=0x7feed40091a0 comp rx=0 tx=0).stop 2023-10-06T05:08:20.603 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.599+0000 7feee359e700 1 -- 172.21.15.98:0/792439966 shutdown_connections 2023-10-06T05:08:20.603 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.599+0000 7feee359e700 1 --2- 172.21.15.98:0/792439966 >> v2:172.21.15.174:3300/0 conn(0x7feee40d09c0 0x7feee40ce860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:20.603 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.599+0000 7feee359e700 1 --2- 172.21.15.98:0/792439966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feee4153990 0x7feee415a370 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:20.604 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.599+0000 7feee359e700 1 -- 172.21.15.98:0/792439966 >> 172.21.15.98:0/792439966 conn(0x7feee40680c0 msgr2=0x7feee4068490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:20.604 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.599+0000 7feee359e700 1 -- 172.21.15.98:0/792439966 shutdown_connections 2023-10-06T05:08:20.604 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.599+0000 7feee359e700 1 -- 172.21.15.98:0/792439966 wait complete. 2023-10-06T05:08:20.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.601+0000 7feee359e700 1 Processor -- start 2023-10-06T05:08:20.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.601+0000 7feee359e700 1 -- start start 2023-10-06T05:08:20.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.601+0000 7feee359e700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feee40cca70 0x7feee416deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:20.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.601+0000 7feee359e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feee40d09c0 0x7feee40b3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:20.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.601+0000 7feee359e700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7feee4153990 0x7feee40b11e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:20.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.601+0000 7feee359e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feee40c9010 con 0x7feee40d09c0 2023-10-06T05:08:20.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.601+0000 7feee359e700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feee40b41f0 con 0x7feee40cca70 2023-10-06T05:08:20.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.601+0000 7feee359e700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feee40b4490 con 0x7feee4153990 2023-10-06T05:08:20.608 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.601+0000 7feed9d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feee40d09c0 0x7feee40b3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:20.608 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.601+0000 7feed9d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feee40d09c0 0x7feee40b3cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36588/0 (socket says 172.21.15.98:36588) 2023-10-06T05:08:20.608 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.601+0000 7feed9d9b700 1 -- 172.21.15.98:0/2210197207 learned_addr learned my addr 172.21.15.98:0/2210197207 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:08:20.608 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.602+0000 7feee259c700 1 --2- 172.21.15.98:0/2210197207 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feee40cca70 0x7feee416deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:20.608 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.602+0000 7feed9d9b700 1 -- 172.21.15.98:0/2210197207 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7feee4153990 msgr2=0x7feee40b11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:20.608 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.602+0000 7feed9d9b700 1 --2- 172.21.15.98:0/2210197207 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7feee4153990 0x7feee40b11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:20.608 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.602+0000 7feed9d9b700 1 -- 172.21.15.98:0/2210197207 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feee40cca70 msgr2=0x7feee416deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:20.608 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.602+0000 7feed9d9b700 1 --2- 172.21.15.98:0/2210197207 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feee40cca70 0x7feee416deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:20.608 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.602+0000 7feed9d9b700 1 -- 172.21.15.98:0/2210197207 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feed4009420 con 0x7feee40d09c0 2023-10-06T05:08:20.608 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.602+0000 7feed9d9b700 1 --2- 172.21.15.98:0/2210197207 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feee40d09c0 0x7feee40b3cb0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7feecc00c5b0 tx=0x7feecc001070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:20.608 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.602+0000 7feedaffd700 1 -- 172.21.15.98:0/2210197207 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feecc005b30 con 0x7feee40d09c0 2023-10-06T05:08:20.608 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.602+0000 7feedaffd700 1 -- 172.21.15.98:0/2210197207 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feecc005c90 con 0x7feee40d09c0 2023-10-06T05:08:20.609 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.602+0000 7feee359e700 1 -- 172.21.15.98:0/2210197207 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feee40b4770 con 0x7feee40d09c0 2023-10-06T05:08:20.609 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.602+0000 7feedaffd700 1 -- 172.21.15.98:0/2210197207 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feecc007300 con 0x7feee40d09c0 2023-10-06T05:08:20.609 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.602+0000 7feee359e700 1 -- 172.21.15.98:0/2210197207 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feee40c8ca0 con 0x7feee40d09c0 2023-10-06T05:08:20.609 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.603+0000 7feee359e700 1 -- 172.21.15.98:0/2210197207 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7feee40ca730 con 0x7feee40d09c0 2023-10-06T05:08:20.609 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.603+0000 7feedaffd700 1 -- 172.21.15.98:0/2210197207 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7feecc01b3b0 con 0x7feee40d09c0 2023-10-06T05:08:20.609 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.604+0000 7feedaffd700 1 --2- 172.21.15.98:0/2210197207 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7feec4066f30 0x7feec40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:20.609 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.604+0000 7feedaffd700 1 -- 172.21.15.98:0/2210197207 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5612+0+0 (secure 0 0 0) 0x7feecc083a00 con 0x7feee40d09c0 2023-10-06T05:08:20.609 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.604+0000 7feee259c700 1 --2- 172.21.15.98:0/2210197207 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7feec4066f30 0x7feec40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:20.609 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.604+0000 7feedaffd700 1 --2- 172.21.15.98:0/2210197207 >> [v2:172.21.15.174:6824/90878,v1:172.21.15.174:6825/90878] conn(0x7feec406d850 0x7feec406fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:20.609 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.604+0000 7feedaffd700 1 -- 172.21.15.98:0/2210197207 --> [v2:172.21.15.174:6824/90878,v1:172.21.15.174:6825/90878] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7feec4070370 con 0x7feec406d850 2023-10-06T05:08:20.609 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.604+0000 7feedaffd700 1 -- 172.21.15.98:0/2210197207 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=46) v2 ==== 24+0+0 (secure 0 0 0) 0x7feecc088240 con 0x7feee40d09c0 2023-10-06T05:08:20.609 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.604+0000 7feee259c700 1 --2- 172.21.15.98:0/2210197207 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7feec4066f30 0x7feec40693e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7feed40093f0 tx=0x7feed4017000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:20.610 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.604+0000 7feee2d9d700 1 --2- 172.21.15.98:0/2210197207 >> [v2:172.21.15.174:6824/90878,v1:172.21.15.174:6825/90878] conn(0x7feec406d850 0x7feec406fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:20.610 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.605+0000 7feee2d9d700 1 --2- 172.21.15.98:0/2210197207 >> [v2:172.21.15.174:6824/90878,v1:172.21.15.174:6825/90878] conn(0x7feec406d850 0x7feec406fca0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:20.610 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.605+0000 7feedaffd700 1 -- 172.21.15.98:0/2210197207 <== osd.7 v2:172.21.15.174:6824/90878 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7feec4070370 con 0x7feec406d850 2023-10-06T05:08:20.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.619+0000 7feee359e700 1 -- 172.21.15.98:0/2210197207 --> [v2:172.21.15.174:6824/90878,v1:172.21.15.174:6825/90878] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7feee405f230 con 0x7feec406d850 2023-10-06T05:08:20.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.619+0000 7feedaffd700 1 -- 172.21.15.98:0/2210197207 <== osd.7 v2:172.21.15.174:6824/90878 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7feee405f230 con 0x7feec406d850 2023-10-06T05:08:20.622 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-10-06T05:08:20.622 INFO:teuthology.orchestra.run.smithi098.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-06T05:08:20.622 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-10-06T05:08:20.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.620+0000 7feee359e700 1 -- 172.21.15.98:0/2210197207 >> [v2:172.21.15.174:6824/90878,v1:172.21.15.174:6825/90878] conn(0x7feec406d850 msgr2=0x7feec406fca0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:20.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.620+0000 7feee359e700 1 --2- 172.21.15.98:0/2210197207 >> [v2:172.21.15.174:6824/90878,v1:172.21.15.174:6825/90878] conn(0x7feec406d850 0x7feec406fca0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:20.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.620+0000 7feee359e700 1 -- 172.21.15.98:0/2210197207 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7feec4066f30 msgr2=0x7feec40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:20.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.620+0000 7feee359e700 1 --2- 172.21.15.98:0/2210197207 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7feec4066f30 0x7feec40693e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7feed40093f0 tx=0x7feed4017000 comp rx=0 tx=0).stop 2023-10-06T05:08:20.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.620+0000 7feee359e700 1 -- 172.21.15.98:0/2210197207 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feee40d09c0 msgr2=0x7feee40b3cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:20.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.620+0000 7feee359e700 1 --2- 172.21.15.98:0/2210197207 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feee40d09c0 0x7feee40b3cb0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7feecc00c5b0 tx=0x7feecc001070 comp rx=0 tx=0).stop 2023-10-06T05:08:20.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.620+0000 7feee2d9d700 1 -- 172.21.15.98:0/2210197207 reap_dead start 2023-10-06T05:08:20.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.620+0000 7feee359e700 1 -- 172.21.15.98:0/2210197207 shutdown_connections 2023-10-06T05:08:20.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.620+0000 7feee359e700 1 -- 172.21.15.98:0/2210197207 >> 172.21.15.98:0/2210197207 conn(0x7feee40680c0 msgr2=0x7feee41580d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:20.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.620+0000 7feee359e700 1 -- 172.21.15.98:0/2210197207 shutdown_connections 2023-10-06T05:08:20.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.620+0000 7feee359e700 1 -- 172.21.15.98:0/2210197207 wait complete. 2023-10-06T05:08:20.634 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2023-10-06T05:08:20.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.746+0000 7f701ffff700 1 Processor -- start 2023-10-06T05:08:20.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.746+0000 7f701ffff700 1 -- start start 2023-10-06T05:08:20.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.747+0000 7f701ffff700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7020159440 0x7f70200b3090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:20.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.747+0000 7f701ffff700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f70200b2100 0x7f70200b2520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:20.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.747+0000 7f701ffff700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70200cb7a0 con 0x7f70200b35d0 2023-10-06T05:08:20.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.747+0000 7f701ffff700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70200b0860 con 0x7f7020159440 2023-10-06T05:08:20.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.747+0000 7f701ffff700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70200cb150 con 0x7f70200b2100 2023-10-06T05:08:20.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.747+0000 7f701effd700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7020159440 0x7f70200b3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:20.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.747+0000 7f70167fc700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f70200b35d0 0x7f70200cbc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:38834/0 (socket says 172.21.15.98:38834) 2023-10-06T05:08:20.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.747+0000 7f70167fc700 1 -- 172.21.15.98:0/3221114266 learned_addr learned my addr 172.21.15.98:0/3221114266 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:08:20.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.747+0000 7f701f7fe700 1 --2- 172.21.15.98:0/3221114266 >> v2:172.21.15.174:3300/0 conn(0x7f70200b2100 0x7f70200b2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:20.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.747+0000 7f701effd700 1 -- 172.21.15.98:0/3221114266 >> v2:172.21.15.174:3300/0 conn(0x7f70200b2100 msgr2=0x7f70200b2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:20.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.747+0000 7f701effd700 1 --2- 172.21.15.98:0/3221114266 >> v2:172.21.15.174:3300/0 conn(0x7f70200b2100 0x7f70200b2520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:20.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.747+0000 7f701effd700 1 -- 172.21.15.98:0/3221114266 >> v1:172.21.15.98:6789/0 conn(0x7f70200b35d0 legacy=0x7f70200cbc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:20.751 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.747+0000 7f701effd700 1 -- 172.21.15.98:0/3221114266 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70200cb430 con 0x7f7020159440 2023-10-06T05:08:20.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.747+0000 7f701e7fc700 1 -- 172.21.15.98:0/3221114266 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3377930312 0 0) 0x7f70200cb7a0 con 0x7f70200b35d0 2023-10-06T05:08:20.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.747+0000 7f701f7fe700 1 --2- 172.21.15.98:0/3221114266 >> v2:172.21.15.174:3300/0 conn(0x7f70200b2100 0x7f70200b2520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T05:08:20.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.747+0000 7f701effd700 1 --2- 172.21.15.98:0/3221114266 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7020159440 0x7f70200b3090 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f70200ca7f0 tx=0x7f70100091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d58ba81e31a24d server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:20.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.747+0000 7f701e7fc700 1 -- 172.21.15.98:0/3221114266 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f701000b6b0 con 0x7f7020159440 2023-10-06T05:08:20.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.748+0000 7f701e7fc700 1 -- 172.21.15.98:0/3221114266 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f701000b810 con 0x7f7020159440 2023-10-06T05:08:20.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.748+0000 7f701e7fc700 1 -- 172.21.15.98:0/3221114266 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f701000bbb0 con 0x7f7020159440 2023-10-06T05:08:20.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.748+0000 7f701ffff700 1 -- 172.21.15.98:0/3221114266 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7020159440 msgr2=0x7f70200b3090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:20.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.748+0000 7f701ffff700 1 --2- 172.21.15.98:0/3221114266 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7020159440 0x7f70200b3090 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f70200ca7f0 tx=0x7f70100091a0 comp rx=0 tx=0).stop 2023-10-06T05:08:20.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.748+0000 7f701ffff700 1 -- 172.21.15.98:0/3221114266 shutdown_connections 2023-10-06T05:08:20.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.748+0000 7f701ffff700 1 --2- 172.21.15.98:0/3221114266 >> v2:172.21.15.174:3300/0 conn(0x7f70200b2100 0x7f70200b2520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:20.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.748+0000 7f701ffff700 1 --2- 172.21.15.98:0/3221114266 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7020159440 0x7f70200b3090 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:20.752 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.748+0000 7f701ffff700 1 -- 172.21.15.98:0/3221114266 >> 172.21.15.98:0/3221114266 conn(0x7f702006ece0 msgr2=0x7f702006f0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:20.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.748+0000 7f701ffff700 1 -- 172.21.15.98:0/3221114266 shutdown_connections 2023-10-06T05:08:20.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.748+0000 7f701ffff700 1 -- 172.21.15.98:0/3221114266 wait complete. 2023-10-06T05:08:20.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.749+0000 7f701ffff700 1 Processor -- start 2023-10-06T05:08:20.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.749+0000 7f701ffff700 1 -- start start 2023-10-06T05:08:20.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.750+0000 7f701ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70200b2100 0x7f702015c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:20.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.750+0000 7f701ffff700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f70200b35d0 0x7f7020154750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:20.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.750+0000 7f701ffff700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7020159440 0x7f70200d0250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:20.755 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.750+0000 7f701ffff700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70200c7bb0 con 0x7f70200b2100 2023-10-06T05:08:20.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.750+0000 7f701ffff700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f702015f5c0 con 0x7f7020159440 2023-10-06T05:08:20.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.750+0000 7f701ffff700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f702015e9b0 con 0x7f70200b35d0 2023-10-06T05:08:20.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.750+0000 7f701effd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70200b2100 0x7f702015c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:20.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.750+0000 7f701effd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70200b2100 0x7f702015c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36612/0 (socket says 172.21.15.98:36612) 2023-10-06T05:08:20.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.750+0000 7f701effd700 1 -- 172.21.15.98:0/2741772518 learned_addr learned my addr 172.21.15.98:0/2741772518 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:08:20.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.750+0000 7f70167fc700 1 --2- 172.21.15.98:0/2741772518 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f70200b35d0 0x7f7020154750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:20.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.750+0000 7f701f7fe700 1 --2- 172.21.15.98:0/2741772518 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7020159440 0x7f70200d0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:20.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.750+0000 7f701effd700 1 -- 172.21.15.98:0/2741772518 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f70200b35d0 msgr2=0x7f7020154750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:20.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.750+0000 7f701effd700 1 --2- 172.21.15.98:0/2741772518 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f70200b35d0 0x7f7020154750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:20.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.750+0000 7f701effd700 1 -- 172.21.15.98:0/2741772518 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7020159440 msgr2=0x7f70200d0250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:20.756 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.750+0000 7f701effd700 1 --2- 172.21.15.98:0/2741772518 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7020159440 0x7f70200d0250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:20.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.750+0000 7f701effd700 1 -- 172.21.15.98:0/2741772518 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7010009420 con 0x7f70200b2100 2023-10-06T05:08:20.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.750+0000 7f70167fc700 1 --2- 172.21.15.98:0/2741772518 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f70200b35d0 0x7f7020154750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:08:20.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.750+0000 7f701f7fe700 1 --2- 172.21.15.98:0/2741772518 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7020159440 0x7f70200d0250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:08:20.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.751+0000 7f701effd700 1 --2- 172.21.15.98:0/2741772518 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70200b2100 0x7f702015c540 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f7020069150 tx=0x7f7010017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:20.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.751+0000 7f70177fe700 1 -- 172.21.15.98:0/2741772518 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7010011420 con 0x7f70200b2100 2023-10-06T05:08:20.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.751+0000 7f70177fe700 1 -- 172.21.15.98:0/2741772518 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7010011580 con 0x7f70200b2100 2023-10-06T05:08:20.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.751+0000 7f701ffff700 1 -- 172.21.15.98:0/2741772518 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f702015ec90 con 0x7f70200b2100 2023-10-06T05:08:20.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.751+0000 7f70177fe700 1 -- 172.21.15.98:0/2741772518 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7010011920 con 0x7f70200b2100 2023-10-06T05:08:20.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.751+0000 7f701ffff700 1 -- 172.21.15.98:0/2741772518 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f702015f250 con 0x7f70200b2100 2023-10-06T05:08:20.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.752+0000 7f701ffff700 1 -- 172.21.15.98:0/2741772518 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f702005f230 con 0x7f70200b2100 2023-10-06T05:08:20.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.752+0000 7f70177fe700 1 -- 172.21.15.98:0/2741772518 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f7010011d10 con 0x7f70200b2100 2023-10-06T05:08:20.757 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.752+0000 7f70177fe700 1 --2- 172.21.15.98:0/2741772518 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f7000066f30 0x7f70000693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:20.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.753+0000 7f70177fe700 1 -- 172.21.15.98:0/2741772518 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5612+0+0 (secure 0 0 0) 0x7f7010085b10 con 0x7f70200b2100 2023-10-06T05:08:20.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.753+0000 7f70167fc700 1 --2- 172.21.15.98:0/2741772518 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f7000066f30 0x7f70000693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:20.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.753+0000 7f70167fc700 1 --2- 172.21.15.98:0/2741772518 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f7000066f30 0x7f70000693e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f702015b410 tx=0x7f7008009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:20.758 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.757+0000 7f70177fe700 1 -- 172.21.15.98:0/2741772518 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7010052dc0 con 0x7f70200b2100 2023-10-06T05:08:20.944 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.941+0000 7f701ffff700 1 -- 172.21.15.98:0/2741772518 --> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f7020159810 con 0x7f7000066f30 2023-10-06T05:08:20.944 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.942+0000 7f70177fe700 1 -- 172.21.15.98:0/2741772518 <== mgr.4104 v2:172.21.15.98:6832/90397 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f7020159810 con 0x7f7000066f30 2023-10-06T05:08:20.947 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.944+0000 7f701ffff700 1 -- 172.21.15.98:0/2741772518 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f7000066f30 msgr2=0x7f70000693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:20.947 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.944+0000 7f701ffff700 1 --2- 172.21.15.98:0/2741772518 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f7000066f30 0x7f70000693e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f702015b410 tx=0x7f7008009040 comp rx=0 tx=0).stop 2023-10-06T05:08:20.947 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.944+0000 7f701ffff700 1 -- 172.21.15.98:0/2741772518 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70200b2100 msgr2=0x7f702015c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:20.947 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.944+0000 7f701ffff700 1 --2- 172.21.15.98:0/2741772518 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70200b2100 0x7f702015c540 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f7020069150 tx=0x7f7010017000 comp rx=0 tx=0).stop 2023-10-06T05:08:20.947 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.944+0000 7f701ffff700 1 -- 172.21.15.98:0/2741772518 shutdown_connections 2023-10-06T05:08:20.947 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.944+0000 7f701ffff700 1 --2- 172.21.15.98:0/2741772518 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7020159440 0x7f70200d0250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:20.947 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.944+0000 7f701ffff700 1 --2- 172.21.15.98:0/2741772518 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f7000066f30 0x7f70000693e0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:20.947 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.944+0000 7f701ffff700 1 --2- 172.21.15.98:0/2741772518 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70200b2100 0x7f702015c540 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:20.947 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.944+0000 7f701ffff700 1 --2- 172.21.15.98:0/2741772518 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f70200b35d0 0x7f7020154750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:20.947 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.944+0000 7f701ffff700 1 -- 172.21.15.98:0/2741772518 >> 172.21.15.98:0/2741772518 conn(0x7f702006ece0 msgr2=0x7f70200cd020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:20.947 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.944+0000 7f701ffff700 1 -- 172.21.15.98:0/2741772518 shutdown_connections 2023-10-06T05:08:20.947 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:20.944+0000 7f701ffff700 1 -- 172.21.15.98:0/2741772518 wait complete. 2023-10-06T05:08:20.948 INFO:teuthology.orchestra.run.smithi098.stderr:instructed osd(s) 7 to deep-scrub 2023-10-06T05:08:20.958 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-06T05:08:21.072 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.070+0000 7fe73c5b7700 1 Processor -- start 2023-10-06T05:08:21.072 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.070+0000 7fe73c5b7700 1 -- start start 2023-10-06T05:08:21.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.071+0000 7fe73c5b7700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe7340b0760 0x7fe7340ccf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:21.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.071+0000 7fe73c5b7700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7340b0c60 con 0x7fe7340b49a0 2023-10-06T05:08:21.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.071+0000 7fe73c5b7700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7340b0f40 con 0x7fe734159560 2023-10-06T05:08:21.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.071+0000 7fe73c5b7700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7340ab350 con 0x7fe7340b0760 2023-10-06T05:08:21.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.071+0000 7fe739b52700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fe7340b49a0 0x7fe7340aac40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:38838/0 (socket says 172.21.15.98:38838) 2023-10-06T05:08:21.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.071+0000 7fe739b52700 1 -- 172.21.15.98:0/930199655 learned_addr learned my addr 172.21.15.98:0/930199655 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:08:21.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.072+0000 7fe73ab54700 1 --2- 172.21.15.98:0/930199655 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe7340b0760 0x7fe7340ccf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:21.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.072+0000 7fe739351700 1 -- 172.21.15.98:0/930199655 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2266866364 0 0) 0x7fe7340b0c60 con 0x7fe7340b49a0 2023-10-06T05:08:21.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.072+0000 7fe739351700 1 -- 172.21.15.98:0/930199655 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe720003570 con 0x7fe7340b49a0 2023-10-06T05:08:21.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.072+0000 7fe739351700 1 -- 172.21.15.98:0/930199655 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4061101022 0 0) 0x7fe720003570 con 0x7fe7340b49a0 2023-10-06T05:08:21.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.072+0000 7fe739351700 1 -- 172.21.15.98:0/930199655 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe7340b0760 msgr2=0x7fe7340ccf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:21.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.072+0000 7fe739351700 1 --2- 172.21.15.98:0/930199655 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe7340b0760 0x7fe7340ccf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:21.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.072+0000 7fe739351700 1 -- 172.21.15.98:0/930199655 >> v1:172.21.15.174:6789/0 conn(0x7fe734159560 legacy=0x7fe7340aa530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:21.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.072+0000 7fe739351700 1 -- 172.21.15.98:0/930199655 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7340b1220 con 0x7fe7340b49a0 2023-10-06T05:08:21.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.072+0000 7fe739351700 1 -- 172.21.15.98:0/930199655 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1543991876 0 0) 0x7fe724001470 con 0x7fe7340b49a0 2023-10-06T05:08:21.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.072+0000 7fe739351700 1 -- 172.21.15.98:0/930199655 >> v1:172.21.15.98:6789/0 conn(0x7fe7340b49a0 legacy=0x7fe7340aac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:21.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.072+0000 7fe739351700 1 --2- 172.21.15.98:0/930199655 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe7340b0760 0x7fe720003a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:21.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.072+0000 7fe739351700 1 --2- 172.21.15.98:0/930199655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe720004010 0x7fe720004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:21.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.072+0000 7fe739351700 1 --2- 172.21.15.98:0/930199655 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe720004d50 0x7fe7200051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:21.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.072+0000 7fe739351700 1 -- 172.21.15.98:0/930199655 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7340ab350 con 0x7fe720004010 2023-10-06T05:08:21.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.072+0000 7fe739351700 1 -- 172.21.15.98:0/930199655 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe72000bbf0 con 0x7fe720004d50 2023-10-06T05:08:21.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.072+0000 7fe739351700 1 -- 172.21.15.98:0/930199655 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe72000bdb0 con 0x7fe7340b0760 2023-10-06T05:08:21.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.072+0000 7fe739b52700 1 --2- 172.21.15.98:0/930199655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe720004010 0x7fe720004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:21.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.072+0000 7fe73a353700 1 --2- 172.21.15.98:0/930199655 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe7340b0760 0x7fe720003a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:21.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.072+0000 7fe73ab54700 1 --2- 172.21.15.98:0/930199655 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe720004d50 0x7fe7200051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:21.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.072+0000 7fe739b52700 1 -- 172.21.15.98:0/930199655 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe7340b0760 msgr2=0x7fe720003a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:21.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.072+0000 7fe739b52700 1 --2- 172.21.15.98:0/930199655 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe7340b0760 0x7fe720003a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:21.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.072+0000 7fe739b52700 1 -- 172.21.15.98:0/930199655 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe720004d50 msgr2=0x7fe7200051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:21.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.072+0000 7fe739b52700 1 --2- 172.21.15.98:0/930199655 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe720004d50 0x7fe7200051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:21.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.072+0000 7fe739b52700 1 -- 172.21.15.98:0/930199655 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe720003570 con 0x7fe720004010 2023-10-06T05:08:21.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.073+0000 7fe739b52700 1 --2- 172.21.15.98:0/930199655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe720004010 0x7fe720004810 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fe734099f20 tx=0x7fe724001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=100d5469760cb51d server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:21.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.073+0000 7fe739351700 1 -- 172.21.15.98:0/930199655 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe72400cc00 con 0x7fe720004010 2023-10-06T05:08:21.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.073+0000 7fe739351700 1 -- 172.21.15.98:0/930199655 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe72400cd60 con 0x7fe720004010 2023-10-06T05:08:21.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.073+0000 7fe739351700 1 -- 172.21.15.98:0/930199655 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe724005b50 con 0x7fe720004010 2023-10-06T05:08:21.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.073+0000 7fe73c5b7700 1 -- 172.21.15.98:0/930199655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe720004010 msgr2=0x7fe720004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:21.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.073+0000 7fe73c5b7700 1 --2- 172.21.15.98:0/930199655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe720004010 0x7fe720004810 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fe734099f20 tx=0x7fe724001e50 comp rx=0 tx=0).stop 2023-10-06T05:08:21.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.073+0000 7fe73ab54700 1 -- 172.21.15.98:0/930199655 reap_dead start 2023-10-06T05:08:21.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.073+0000 7fe73c5b7700 1 -- 172.21.15.98:0/930199655 shutdown_connections 2023-10-06T05:08:21.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.073+0000 7fe73c5b7700 1 -- 172.21.15.98:0/930199655 >> 172.21.15.98:0/930199655 conn(0x7fe7340997f0 msgr2=0x7fe73406f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:21.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.073+0000 7fe73c5b7700 1 -- 172.21.15.98:0/930199655 shutdown_connections 2023-10-06T05:08:21.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.073+0000 7fe73c5b7700 1 -- 172.21.15.98:0/930199655 wait complete. 2023-10-06T05:08:21.081 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.074+0000 7fe73c5b7700 1 Processor -- start 2023-10-06T05:08:21.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.074+0000 7fe73c5b7700 1 -- start start 2023-10-06T05:08:21.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.075+0000 7fe73c5b7700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe720004010 0x7fe734147810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:21.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.075+0000 7fe73c5b7700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe7340ceb40 0x7fe7340cefb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:21.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.075+0000 7fe73c5b7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7340ab5b0 0x7fe7340aba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:21.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.075+0000 7fe73c5b7700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7340cf4f0 con 0x7fe7340ab5b0 2023-10-06T05:08:21.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.075+0000 7fe73c5b7700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe73409ac70 con 0x7fe720004010 2023-10-06T05:08:21.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.075+0000 7fe73c5b7700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe73409a060 con 0x7fe7340ceb40 2023-10-06T05:08:21.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.075+0000 7fe73ab54700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7340ab5b0 0x7fe7340aba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:21.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.075+0000 7fe73ab54700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7340ab5b0 0x7fe7340aba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36630/0 (socket says 172.21.15.98:36630) 2023-10-06T05:08:21.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.075+0000 7fe73ab54700 1 -- 172.21.15.98:0/3383711165 learned_addr learned my addr 172.21.15.98:0/3383711165 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:08:21.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.075+0000 7fe739b52700 1 --2- 172.21.15.98:0/3383711165 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe7340ceb40 0x7fe7340cefb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:21.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.076+0000 7fe73a353700 1 --2- 172.21.15.98:0/3383711165 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe720004010 0x7fe734147810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:21.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.076+0000 7fe73ab54700 1 -- 172.21.15.98:0/3383711165 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe7340ceb40 msgr2=0x7fe7340cefb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:21.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.076+0000 7fe73ab54700 1 --2- 172.21.15.98:0/3383711165 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe7340ceb40 0x7fe7340cefb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:21.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.076+0000 7fe73ab54700 1 -- 172.21.15.98:0/3383711165 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe720004010 msgr2=0x7fe734147810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:21.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.076+0000 7fe73ab54700 1 --2- 172.21.15.98:0/3383711165 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe720004010 0x7fe734147810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:21.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.076+0000 7fe73ab54700 1 -- 172.21.15.98:0/3383711165 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe73409a340 con 0x7fe7340ab5b0 2023-10-06T05:08:21.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.076+0000 7fe73ab54700 1 --2- 172.21.15.98:0/3383711165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7340ab5b0 0x7fe7340aba30 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fe72c006f00 tx=0x7fe72c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:21.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.076+0000 7fe72a7fc700 1 -- 172.21.15.98:0/3383711165 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe72c00f910 con 0x7fe7340ab5b0 2023-10-06T05:08:21.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.076+0000 7fe72a7fc700 1 -- 172.21.15.98:0/3383711165 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe72c00fa70 con 0x7fe7340ab5b0 2023-10-06T05:08:21.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.076+0000 7fe73c5b7700 1 -- 172.21.15.98:0/3383711165 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe73409a620 con 0x7fe7340ab5b0 2023-10-06T05:08:21.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.076+0000 7fe72a7fc700 1 -- 172.21.15.98:0/3383711165 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe72c00fe10 con 0x7fe7340ab5b0 2023-10-06T05:08:21.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.076+0000 7fe73c5b7700 1 -- 172.21.15.98:0/3383711165 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe734147d50 con 0x7fe7340ab5b0 2023-10-06T05:08:21.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.077+0000 7fe73c5b7700 1 -- 172.21.15.98:0/3383711165 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe73409ae20 con 0x7fe7340ab5b0 2023-10-06T05:08:21.084 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.077+0000 7fe72a7fc700 1 -- 172.21.15.98:0/3383711165 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fe72c014650 con 0x7fe7340ab5b0 2023-10-06T05:08:21.084 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.078+0000 7fe72a7fc700 1 --2- 172.21.15.98:0/3383711165 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fe718066f30 0x7fe7180693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:21.084 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.078+0000 7fe72a7fc700 1 -- 172.21.15.98:0/3383711165 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5612+0+0 (secure 0 0 0) 0x7fe72c084b50 con 0x7fe7340ab5b0 2023-10-06T05:08:21.084 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.078+0000 7fe73a353700 1 --2- 172.21.15.98:0/3383711165 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fe718066f30 0x7fe7180693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:21.084 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.078+0000 7fe73a353700 1 --2- 172.21.15.98:0/3383711165 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fe718066f30 0x7fe7180693e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fe7300042e0 tx=0x7fe730001a70 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:21.084 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.082+0000 7fe72a7fc700 1 -- 172.21.15.98:0/3383711165 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe73409ae20 con 0x7fe7340ab5b0 2023-10-06T05:08:21.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.263+0000 7fe73c5b7700 1 -- 172.21.15.98:0/3383711165 --> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe734159150 con 0x7fe718066f30 2023-10-06T05:08:21.268 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.266+0000 7fe72a7fc700 1 -- 172.21.15.98:0/3383711165 <== mgr.4104 v2:172.21.15.98:6832/90397 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+72543 (secure 0 0 0) 0x7fe734159150 con 0x7fe718066f30 2023-10-06T05:08:21.269 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:08:21.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.268+0000 7fe73c5b7700 1 -- 172.21.15.98:0/3383711165 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fe718066f30 msgr2=0x7fe7180693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:21.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.268+0000 7fe73c5b7700 1 --2- 172.21.15.98:0/3383711165 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fe718066f30 0x7fe7180693e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fe7300042e0 tx=0x7fe730001a70 comp rx=0 tx=0).stop 2023-10-06T05:08:21.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.268+0000 7fe73c5b7700 1 -- 172.21.15.98:0/3383711165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7340ab5b0 msgr2=0x7fe7340aba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:21.270 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.268+0000 7fe73c5b7700 1 --2- 172.21.15.98:0/3383711165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7340ab5b0 0x7fe7340aba30 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fe72c006f00 tx=0x7fe72c0091a0 comp rx=0 tx=0).stop 2023-10-06T05:08:21.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.268+0000 7fe73c5b7700 1 -- 172.21.15.98:0/3383711165 shutdown_connections 2023-10-06T05:08:21.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.268+0000 7fe73c5b7700 1 --2- 172.21.15.98:0/3383711165 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fe718066f30 0x7fe7180693e0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:21.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.268+0000 7fe73c5b7700 1 --2- 172.21.15.98:0/3383711165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7340ab5b0 0x7fe7340aba30 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:21.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.268+0000 7fe73c5b7700 1 --2- 172.21.15.98:0/3383711165 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe720004010 0x7fe734147810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:21.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.268+0000 7fe73c5b7700 1 --2- 172.21.15.98:0/3383711165 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe7340ceb40 0x7fe7340cefb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:21.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.268+0000 7fe73c5b7700 1 -- 172.21.15.98:0/3383711165 >> 172.21.15.98:0/3383711165 conn(0x7fe7340997f0 msgr2=0x7fe7340afce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:21.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.268+0000 7fe73c5b7700 1 -- 172.21.15.98:0/3383711165 shutdown_connections 2023-10-06T05:08:21.271 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:21.268+0000 7fe73c5b7700 1 -- 172.21.15.98:0/3383711165 wait complete. 2023-10-06T05:08:21.271 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2023-10-06T05:08:21.282 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":88,"stamp":"2023-10-06T05:08:20.536411+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":2383656,"kb_used_data":5352,"kb_used_omap":0,"kb_used_meta":2378240,"kb_avail":752591064,"statfs":{"total":773094113280,"available":770653249536,"internally_reserved":0,"allocated":5480448,"data_stored":3086880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2435317760},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-33554432,"num_objects":-8,"num_object_clones":0,"num_object_copies":-16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8,"num_whiteouts":0,"num_read":-9,"num_read_kb":-15872,"num_write":-8,"num_write_kb":-32768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.783071"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":32,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-06T05:08:12.845087+0000","last_change":"2023-10-06T05:06:52.627232+0000","last_active":"2023-10-06T05:08:12.845087+0000","last_peered":"2023-10-06T05:08:12.845087+0000","last_clean":"2023-10-06T05:08:12.845087+0000","last_became_active":"2023-10-06T05:06:52.626936+0000","last_became_peered":"2023-10-06T05:06:52.626936+0000","last_unstale":"2023-10-06T05:08:12.845087+0000","last_undegraded":"2023-10-06T05:08:12.845087+0000","last_fullsized":"2023-10-06T05:08:12.845087+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":34,"reported_epoch":45,"state":"active+clean","last_fresh":"2023-10-06T05:08:11.757641+0000","last_change":"2023-10-06T05:06:52.587795+0000","last_active":"2023-10-06T05:08:11.757641+0000","last_peered":"2023-10-06T05:08:11.757641+0000","last_clean":"2023-10-06T05:08:11.757641+0000","last_became_active":"2023-10-06T05:06:52.587407+0000","last_became_peered":"2023-10-06T05:06:52.587407+0000","last_unstale":"2023-10-06T05:08:11.757641+0000","last_undegraded":"2023-10-06T05:08:11.757641+0000","last_fullsized":"2023-10-06T05:08:11.757641+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:36:31.526532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":32,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-06T05:08:12.848492+0000","last_change":"2023-10-06T05:06:52.675238+0000","last_active":"2023-10-06T05:08:12.848492+0000","last_peered":"2023-10-06T05:08:12.848492+0000","last_clean":"2023-10-06T05:08:12.848492+0000","last_became_active":"2023-10-06T05:06:52.674952+0000","last_became_peered":"2023-10-06T05:06:52.674952+0000","last_unstale":"2023-10-06T05:08:12.848492+0000","last_undegraded":"2023-10-06T05:08:12.848492+0000","last_fullsized":"2023-10-06T05:08:12.848492+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":34,"reported_epoch":45,"state":"active+clean","last_fresh":"2023-10-06T05:08:11.757942+0000","last_change":"2023-10-06T05:06:52.587778+0000","last_active":"2023-10-06T05:08:11.757942+0000","last_peered":"2023-10-06T05:08:11.757942+0000","last_clean":"2023-10-06T05:08:11.757942+0000","last_became_active":"2023-10-06T05:06:52.587388+0000","last_became_peered":"2023-10-06T05:06:52.587388+0000","last_unstale":"2023-10-06T05:08:11.757942+0000","last_undegraded":"2023-10-06T05:08:11.757942+0000","last_fullsized":"2023-10-06T05:08:11.757942+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T14:45:52.502298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":185,"reported_epoch":45,"state":"active+clean","last_fresh":"2023-10-06T05:08:11.919801+0000","last_change":"2023-10-06T05:06:46.307169+0000","last_active":"2023-10-06T05:08:11.919801+0000","last_peered":"2023-10-06T05:08:11.919801+0000","last_clean":"2023-10-06T05:08:11.919801+0000","last_became_active":"2023-10-06T05:06:46.306107+0000","last_became_peered":"2023-10-06T05:06:46.306107+0000","last_unstale":"2023-10-06T05:08:11.919801+0000","last_undegraded":"2023-10-06T05:08:11.919801+0000","last_fullsized":"2023-10-06T05:08:11.919801+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:45.018006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:45.018006+0000","last_clean_scrub_stamp":"2023-10-06T05:06:45.018006+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:38:17.064131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":32,"reported_epoch":45,"state":"active+clean","last_fresh":"2023-10-06T05:08:11.919890+0000","last_change":"2023-10-06T05:06:52.588475+0000","last_active":"2023-10-06T05:08:11.919890+0000","last_peered":"2023-10-06T05:08:11.919890+0000","last_clean":"2023-10-06T05:08:11.919890+0000","last_became_active":"2023-10-06T05:06:52.588134+0000","last_became_peered":"2023-10-06T05:06:52.588134+0000","last_unstale":"2023-10-06T05:08:11.919890+0000","last_undegraded":"2023-10-06T05:08:11.919890+0000","last_fullsized":"2023-10-06T05:08:11.919890+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:35:05.269778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":32,"reported_epoch":45,"state":"active+clean","last_fresh":"2023-10-06T05:08:11.919876+0000","last_change":"2023-10-06T05:06:52.627289+0000","last_active":"2023-10-06T05:08:11.919876+0000","last_peered":"2023-10-06T05:08:11.919876+0000","last_clean":"2023-10-06T05:08:11.919876+0000","last_became_active":"2023-10-06T05:06:52.626937+0000","last_became_peered":"2023-10-06T05:06:52.626937+0000","last_unstale":"2023-10-06T05:08:11.919876+0000","last_undegraded":"2023-10-06T05:08:11.919876+0000","last_fullsized":"2023-10-06T05:08:11.919876+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:46:59.070674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":32,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-06T05:08:12.846113+0000","last_change":"2023-10-06T05:06:52.588853+0000","last_active":"2023-10-06T05:08:12.846113+0000","last_peered":"2023-10-06T05:08:12.846113+0000","last_clean":"2023-10-06T05:08:12.846113+0000","last_became_active":"2023-10-06T05:06:52.588561+0000","last_became_peered":"2023-10-06T05:06:52.588561+0000","last_unstale":"2023-10-06T05:08:12.846113+0000","last_undegraded":"2023-10-06T05:08:12.846113+0000","last_fullsized":"2023-10-06T05:08:12.846113+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":30,"reported_epoch":45,"state":"active+clean","last_fresh":"2023-10-06T05:08:11.920709+0000","last_change":"2023-10-06T05:06:52.589519+0000","last_active":"2023-10-06T05:08:11.920709+0000","last_peered":"2023-10-06T05:08:11.920709+0000","last_clean":"2023-10-06T05:08:11.920709+0000","last_became_active":"2023-10-06T05:06:52.589246+0000","last_became_peered":"2023-10-06T05:06:52.589246+0000","last_unstale":"2023-10-06T05:08:11.920709+0000","last_undegraded":"2023-10-06T05:08:11.920709+0000","last_fullsized":"2023-10-06T05:08:11.920709+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:06:51.570421+0000","last_clean_scrub_stamp":"2023-10-06T05:06:51.570421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:54:23.352764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509463,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297812,"kb_used_data":524,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074028,"statfs":{"total":96636764160,"available":96331804672,"internally_reserved":0,"allocated":536576,"data_stored":238268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.52700000000000002}]},{"osd":1,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.48299999999999998}]},{"osd":2,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.63500000000000001}]},{"osd":3,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.499},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.53800000000000003}]},{"osd":4,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.57099999999999995}]},{"osd":5,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.58599999999999997}]},{"osd":7,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.64200000000000002}]}]},{"osd":5,"up_from":15,"seq":64424509463,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297812,"kb_used_data":524,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074028,"statfs":{"total":96636764160,"available":96331804672,"internally_reserved":0,"allocated":536576,"data_stored":238268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.56999999999999995}]},{"osd":1,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.58499999999999996}]},{"osd":2,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.49199999999999999}]},{"osd":3,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.623}]},{"osd":4,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.64800000000000002}]},{"osd":6,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.63600000000000001}]},{"osd":7,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.66800000000000004}]}]},{"osd":0,"up_from":14,"seq":60129542167,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298392,"kb_used_data":1104,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073448,"statfs":{"total":96636764160,"available":96331210752,"internally_reserved":0,"allocated":1130496,"data_stored":828636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":1.2809999999999999,"5min":1.2809999999999999,"15min":1.2809999999999999},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.35899999999999999}]},{"osd":2,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":4.1529999999999996,"5min":4.1529999999999996,"15min":4.1529999999999996},"last":0.308},{"interface":"front","average":{"1min":0.372,"5min":0.372,"15min":0.372},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":1.105,"5min":1.105,"15min":1.105},"last":0.38500000000000001}]},{"osd":3,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":1.327,"5min":1.327,"15min":1.327},"last":0.377},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":4.1900000000000004,"5min":4.1900000000000004,"15min":4.1900000000000004},"last":0.40699999999999997}]},{"osd":4,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.60199999999999998}]},{"osd":5,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.304,"5min":1.304,"15min":1.304},"last":0.626}]},{"osd":6,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.73099999999999998}]},{"osd":7,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":1.079,"5min":1.079,"15min":1.079},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.67700000000000005}]}]},{"osd":7,"up_from":13,"seq":55834574870,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298392,"kb_used_data":1104,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073448,"statfs":{"total":96636764160,"available":96331210752,"internally_reserved":0,"allocated":1130496,"data_stored":828636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.621},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.45900000000000002}]},{"osd":1,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.64700000000000002}]},{"osd":2,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.504},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.60399999999999998}]},{"osd":3,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.57699999999999996}]},{"osd":4,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.436}]},{"osd":5,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.55600000000000005}]},{"osd":6,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.59299999999999997}]}]},{"osd":1,"up_from":14,"seq":60129542166,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297812,"kb_used_data":524,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074028,"statfs":{"total":96636764160,"available":96331804672,"internally_reserved":0,"allocated":536576,"data_stored":238268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":1.8879999999999999,"5min":1.8879999999999999,"15min":1.8879999999999999},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":1.595,"5min":1.595,"15min":1.595},"last":0.27700000000000002}]},{"osd":2,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.123,"5min":1.123,"15min":1.123},"last":0.309}]},{"osd":3,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.179,"5min":1.179,"15min":1.179},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1.657,"5min":1.657,"15min":1.657},"last":0.371}]},{"osd":4,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":1.6319999999999999,"5min":1.6319999999999999,"15min":1.6319999999999999},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.837,"5min":1.837,"15min":1.837},"last":0.436}]},{"osd":5,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.8100000000000001,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.49399999999999999}]},{"osd":6,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.6850000000000001,"5min":1.6850000000000001,"15min":1.6850000000000001},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.8620000000000001,"5min":1.8620000000000001,"15min":1.8620000000000001},"last":0.53900000000000003}]},{"osd":7,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.7789999999999999,"5min":1.7789999999999999,"15min":1.7789999999999999},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.52300000000000002}]}]},{"osd":2,"up_from":15,"seq":64424509462,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297812,"kb_used_data":524,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074028,"statfs":{"total":96636764160,"available":96331804672,"internally_reserved":0,"allocated":536576,"data_stored":238268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.311,"5min":0.311,"15min":0.311},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.35899999999999999}]},{"osd":1,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"last":0.33400000000000002}]},{"osd":3,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"last":0.31},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"last":0.29599999999999999}]},{"osd":4,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.496}]},{"osd":5,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.443}]},{"osd":6,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.55600000000000005}]},{"osd":7,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.36599999999999999}]}]},{"osd":3,"up_from":15,"seq":64424509462,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297812,"kb_used_data":524,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074028,"statfs":{"total":96636764160,"available":96331804672,"internally_reserved":0,"allocated":536576,"data_stored":238268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"last":0.309},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":3.391,"5min":3.391,"15min":3.391},"last":0.26000000000000001}]},{"osd":1,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":2.8740000000000001,"5min":2.8740000000000001,"15min":2.8740000000000001},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.376,"5min":0.376,"15min":0.376},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.34899999999999998}]},{"osd":2,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":7.2210000000000001,"5min":7.2210000000000001,"15min":7.2210000000000001},"last":0.38200000000000001}]},{"osd":4,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.018,"5min":1.018,"15min":1.018},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.6930000000000001,"5min":1.6930000000000001,"15min":1.6930000000000001},"last":0.57199999999999995}]},{"osd":5,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.607,"5min":1.607,"15min":1.607},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.52700000000000002}]},{"osd":6,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.495},{"interface":"front","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.657,"5min":1.657,"15min":1.657},"last":0.54700000000000004}]},{"osd":7,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.877,"5min":1.877,"15min":1.877},"last":0.53600000000000003}]}]},{"osd":4,"up_from":15,"seq":64424509463,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297812,"kb_used_data":524,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074028,"statfs":{"total":96636764160,"available":96331804672,"internally_reserved":0,"allocated":536576,"data_stored":238268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.52600000000000002}]},{"osd":1,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.53800000000000003}]},{"osd":2,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.441},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.627}]},{"osd":3,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":11.385999999999999,"5min":11.385999999999999,"15min":11.385999999999999},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.63800000000000001}]},{"osd":5,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.57199999999999995}]},{"osd":6,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.68899999999999995}]},{"osd":7,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.66800000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-06T05:08:21.286 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-06T05:06:51.570421+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=6, tm_hour=5, tm_min=6, tm_sec=51, tm_wday=4, tm_yday=279, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=6, tm_hour=5, tm_min=8, tm_sec=16, tm_wday=4, tm_yday=279, tm_isdst=0) 2023-10-06T05:08:21.286 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-06T05:06:51.570421+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=6, tm_hour=5, tm_min=6, tm_sec=51, tm_wday=4, tm_yday=279, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=6, tm_hour=5, tm_min=8, tm_sec=16, tm_wday=4, tm_yday=279, tm_isdst=0) 2023-10-06T05:08:21.286 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-06T05:06:51.570421+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=6, tm_hour=5, tm_min=6, tm_sec=51, tm_wday=4, tm_yday=279, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=6, tm_hour=5, tm_min=8, tm_sec=16, tm_wday=4, tm_yday=279, tm_isdst=0) 2023-10-06T05:08:21.286 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-06T05:06:51.570421+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=6, tm_hour=5, tm_min=6, tm_sec=51, tm_wday=4, tm_yday=279, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=6, tm_hour=5, tm_min=8, tm_sec=16, tm_wday=4, tm_yday=279, tm_isdst=0) 2023-10-06T05:08:21.286 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-10-06T05:06:45.018006+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=6, tm_hour=5, tm_min=6, tm_sec=45, tm_wday=4, tm_yday=279, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=6, tm_hour=5, tm_min=8, tm_sec=16, tm_wday=4, tm_yday=279, tm_isdst=0) 2023-10-06T05:08:21.286 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-06T05:06:51.570421+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=6, tm_hour=5, tm_min=6, tm_sec=51, tm_wday=4, tm_yday=279, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=6, tm_hour=5, tm_min=8, tm_sec=16, tm_wday=4, tm_yday=279, tm_isdst=0) 2023-10-06T05:08:21.287 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-06T05:06:51.570421+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=6, tm_hour=5, tm_min=6, tm_sec=51, tm_wday=4, tm_yday=279, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=6, tm_hour=5, tm_min=8, tm_sec=16, tm_wday=4, tm_yday=279, tm_isdst=0) 2023-10-06T05:08:21.287 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-06T05:06:51.570421+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=6, tm_hour=5, tm_min=6, tm_sec=51, tm_wday=4, tm_yday=279, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=6, tm_hour=5, tm_min=8, tm_sec=16, tm_wday=4, tm_yday=279, tm_isdst=0) 2023-10-06T05:08:21.287 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-06T05:06:51.570421+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=6, tm_hour=5, tm_min=6, tm_sec=51, tm_wday=4, tm_yday=279, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=6, tm_hour=5, tm_min=8, tm_sec=16, tm_wday=4, tm_yday=279, tm_isdst=0) 2023-10-06T05:08:21.287 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-06T05:08:27.067 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-06T05:08:27.070 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-06T05:08:41.288 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-06T05:08:41.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.413+0000 7feb1b020700 1 Processor -- start 2023-10-06T05:08:41.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.413+0000 7feb1b020700 1 -- start start 2023-10-06T05:08:41.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.414+0000 7feb1b020700 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7feb14159440 0x7feb140b3090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:41.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.414+0000 7feb1b020700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7feb140b2100 0x7feb140b2520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:41.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.414+0000 7feb1b020700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb140cb7a0 con 0x7feb140b35d0 2023-10-06T05:08:41.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.414+0000 7feb1b020700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb140b0860 con 0x7feb140b2100 2023-10-06T05:08:41.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.414+0000 7feb1b020700 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb140cb150 con 0x7feb14159440 2023-10-06T05:08:41.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.414+0000 7feb1a81f700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7feb140b2100 0x7feb140b2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:41.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.414+0000 7feb1a81f700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7feb140b2100 0x7feb140b2520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51616/0 (socket says 172.21.15.98:51616) 2023-10-06T05:08:41.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.414+0000 7feb1a81f700 1 -- 172.21.15.98:0/2270938450 learned_addr learned my addr 172.21.15.98:0/2270938450 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:08:41.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.414+0000 7feb1a01e700 1 --2- 172.21.15.98:0/2270938450 >> v2:172.21.15.174:3300/0 conn(0x7feb14159440 0x7feb140b3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:41.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.414+0000 7feb1a81f700 1 -- 172.21.15.98:0/2270938450 >> v2:172.21.15.174:3300/0 conn(0x7feb14159440 msgr2=0x7feb140b3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:41.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.414+0000 7feb1a81f700 1 --2- 172.21.15.98:0/2270938450 >> v2:172.21.15.174:3300/0 conn(0x7feb14159440 0x7feb140b3090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:41.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.414+0000 7feb1a81f700 1 -- 172.21.15.98:0/2270938450 >> v1:172.21.15.174:6789/0 conn(0x7feb140b35d0 legacy=0x7feb140cbc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:41.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.414+0000 7feb1a81f700 1 -- 172.21.15.98:0/2270938450 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb140cb430 con 0x7feb140b2100 2023-10-06T05:08:41.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.415+0000 7feb1a81f700 1 --2- 172.21.15.98:0/2270938450 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb140b2100 0x7feb140b2520 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7feb1406f9f0 tx=0x7feb0c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=194c1c854298eab1 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:41.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.415+0000 7feb1901c700 1 -- 172.21.15.98:0/2270938450 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb0c00b820 con 0x7feb140b2100 2023-10-06T05:08:41.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.415+0000 7feb1901c700 1 -- 172.21.15.98:0/2270938450 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb0c00b980 con 0x7feb140b2100 2023-10-06T05:08:41.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.415+0000 7feb1901c700 1 -- 172.21.15.98:0/2270938450 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb0c00bd20 con 0x7feb140b2100 2023-10-06T05:08:41.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.415+0000 7feb1b020700 1 -- 172.21.15.98:0/2270938450 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb140b2100 msgr2=0x7feb140b2520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:41.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.415+0000 7feb1b020700 1 --2- 172.21.15.98:0/2270938450 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb140b2100 0x7feb140b2520 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7feb1406f9f0 tx=0x7feb0c009250 comp rx=0 tx=0).stop 2023-10-06T05:08:41.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.415+0000 7feb1b020700 1 -- 172.21.15.98:0/2270938450 shutdown_connections 2023-10-06T05:08:41.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.415+0000 7feb1b020700 1 --2- 172.21.15.98:0/2270938450 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb140b2100 0x7feb140b2520 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:41.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.415+0000 7feb1b020700 1 --2- 172.21.15.98:0/2270938450 >> v2:172.21.15.174:3300/0 conn(0x7feb14159440 0x7feb140b3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:41.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.415+0000 7feb1b020700 1 -- 172.21.15.98:0/2270938450 >> 172.21.15.98:0/2270938450 conn(0x7feb1406ece0 msgr2=0x7feb1406f0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:41.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.415+0000 7feb1b020700 1 -- 172.21.15.98:0/2270938450 shutdown_connections 2023-10-06T05:08:41.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.415+0000 7feb1b020700 1 -- 172.21.15.98:0/2270938450 wait complete. 2023-10-06T05:08:41.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.417+0000 7feb1b020700 1 Processor -- start 2023-10-06T05:08:41.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.417+0000 7feb1b020700 1 -- start start 2023-10-06T05:08:41.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.417+0000 7feb1b020700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7feb140b2100 0x7feb1415c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:41.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.417+0000 7feb1b020700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb140b35d0 0x7feb140c87a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:41.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.417+0000 7feb1b020700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feb14159440 0x7feb140c4a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:41.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.417+0000 7feb1b020700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb140c4590 con 0x7feb140b35d0 2023-10-06T05:08:41.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.417+0000 7feb1b020700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb1415c690 con 0x7feb14159440 2023-10-06T05:08:41.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.417+0000 7feb1b020700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb14154c90 con 0x7feb140b2100 2023-10-06T05:08:41.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.417+0000 7feb1981d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb140b35d0 0x7feb140c87a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:41.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.417+0000 7feb1981d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb140b35d0 0x7feb140c87a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51618/0 (socket says 172.21.15.98:51618) 2023-10-06T05:08:41.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.417+0000 7feb1981d700 1 -- 172.21.15.98:0/4049468715 learned_addr learned my addr 172.21.15.98:0/4049468715 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:08:41.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.418+0000 7feb1981d700 1 -- 172.21.15.98:0/4049468715 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7feb140b2100 msgr2=0x7feb1415c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:41.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.418+0000 7feb1981d700 1 --2- 172.21.15.98:0/4049468715 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7feb140b2100 0x7feb1415c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:41.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.418+0000 7feb1981d700 1 -- 172.21.15.98:0/4049468715 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feb14159440 msgr2=0x7feb140c4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:41.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.418+0000 7feb1981d700 1 --2- 172.21.15.98:0/4049468715 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feb14159440 0x7feb140c4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:41.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.418+0000 7feb1981d700 1 -- 172.21.15.98:0/4049468715 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb0c0094f0 con 0x7feb140b35d0 2023-10-06T05:08:41.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.418+0000 7feb1981d700 1 --2- 172.21.15.98:0/4049468715 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb140b35d0 0x7feb140c87a0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7feb0400c5d0 tx=0x7feb04007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:41.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.418+0000 7feb0a7fc700 1 -- 172.21.15.98:0/4049468715 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb04011590 con 0x7feb140b35d0 2023-10-06T05:08:41.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.418+0000 7feb0a7fc700 1 -- 172.21.15.98:0/4049468715 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb04011710 con 0x7feb140b35d0 2023-10-06T05:08:41.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.418+0000 7feb1b020700 1 -- 172.21.15.98:0/4049468715 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb140c3c60 con 0x7feb140b35d0 2023-10-06T05:08:41.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.418+0000 7feb0a7fc700 1 -- 172.21.15.98:0/4049468715 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb04011ad0 con 0x7feb140b35d0 2023-10-06T05:08:41.423 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.418+0000 7feb1b020700 1 -- 172.21.15.98:0/4049468715 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb140c4220 con 0x7feb140b35d0 2023-10-06T05:08:41.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.419+0000 7feb1b020700 1 -- 172.21.15.98:0/4049468715 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb1405f230 con 0x7feb140b35d0 2023-10-06T05:08:41.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.419+0000 7feb0a7fc700 1 -- 172.21.15.98:0/4049468715 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7feb0401c460 con 0x7feb140b35d0 2023-10-06T05:08:41.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.420+0000 7feb0a7fc700 1 --2- 172.21.15.98:0/4049468715 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7feaf8066f30 0x7feaf80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:41.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.420+0000 7feb0a7fc700 1 -- 172.21.15.98:0/4049468715 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5612+0+0 (secure 0 0 0) 0x7feb040842d0 con 0x7feb140b35d0 2023-10-06T05:08:41.424 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.420+0000 7feb1a01e700 1 --2- 172.21.15.98:0/4049468715 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7feaf8066f30 0x7feaf80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:41.425 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.420+0000 7feb1a01e700 1 --2- 172.21.15.98:0/4049468715 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7feaf8066f30 0x7feaf80693e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7feb10007eb0 tx=0x7feb100073a0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:41.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.424+0000 7feb0a7fc700 1 -- 172.21.15.98:0/4049468715 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7feb040515a0 con 0x7feb140b35d0 2023-10-06T05:08:41.604 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.602+0000 7feb1b020700 1 -- 172.21.15.98:0/4049468715 --> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7feb140c46d0 con 0x7feaf8066f30 2023-10-06T05:08:41.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.605+0000 7feb0a7fc700 1 -- 172.21.15.98:0/4049468715 <== mgr.4104 v2:172.21.15.98:6832/90397 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+72790 (secure 0 0 0) 0x7feb140c46d0 con 0x7feaf8066f30 2023-10-06T05:08:41.608 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:08:41.610 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.607+0000 7feb1b020700 1 -- 172.21.15.98:0/4049468715 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7feaf8066f30 msgr2=0x7feaf80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:41.610 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.607+0000 7feb1b020700 1 --2- 172.21.15.98:0/4049468715 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7feaf8066f30 0x7feaf80693e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7feb10007eb0 tx=0x7feb100073a0 comp rx=0 tx=0).stop 2023-10-06T05:08:41.610 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.607+0000 7feb1b020700 1 -- 172.21.15.98:0/4049468715 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb140b35d0 msgr2=0x7feb140c87a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:41.610 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.607+0000 7feb1b020700 1 --2- 172.21.15.98:0/4049468715 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb140b35d0 0x7feb140c87a0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7feb0400c5d0 tx=0x7feb04007f60 comp rx=0 tx=0).stop 2023-10-06T05:08:41.610 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.607+0000 7feb1b020700 1 -- 172.21.15.98:0/4049468715 shutdown_connections 2023-10-06T05:08:41.610 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.607+0000 7feb1b020700 1 --2- 172.21.15.98:0/4049468715 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feb14159440 0x7feb140c4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:41.610 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.607+0000 7feb1b020700 1 --2- 172.21.15.98:0/4049468715 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7feb140b2100 0x7feb1415c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:41.610 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.607+0000 7feb1b020700 1 --2- 172.21.15.98:0/4049468715 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7feaf8066f30 0x7feaf80693e0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:41.611 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.607+0000 7feb1b020700 1 --2- 172.21.15.98:0/4049468715 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb140b35d0 0x7feb140c87a0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:41.611 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.607+0000 7feb1b020700 1 -- 172.21.15.98:0/4049468715 >> 172.21.15.98:0/4049468715 conn(0x7feb1406ece0 msgr2=0x7feb140cbc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:41.611 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.607+0000 7feb1b020700 1 -- 172.21.15.98:0/4049468715 shutdown_connections 2023-10-06T05:08:41.611 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.607+0000 7feb1b020700 1 -- 172.21.15.98:0/4049468715 wait complete. 2023-10-06T05:08:41.611 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2023-10-06T05:08:41.621 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":98,"stamp":"2023-10-06T05:08:40.540631+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":2383656,"kb_used_data":5352,"kb_used_omap":0,"kb_used_meta":2378240,"kb_avail":752591064,"statfs":{"total":773094113280,"available":770653249536,"internally_reserved":0,"allocated":5480448,"data_stored":3086880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2435317760},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002354"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":42,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-06T05:08:23.648532+0000","last_change":"2023-10-06T05:08:23.648506+0000","last_active":"2023-10-06T05:08:23.648532+0000","last_peered":"2023-10-06T05:08:23.648532+0000","last_clean":"2023-10-06T05:08:23.648532+0000","last_became_active":"2023-10-06T05:06:52.626936+0000","last_became_peered":"2023-10-06T05:06:52.626936+0000","last_unstale":"2023-10-06T05:08:23.648532+0000","last_undegraded":"2023-10-06T05:08:23.648532+0000","last_fullsized":"2023-10-06T05:08:23.648532+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:08:23.648411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:08:23.648411+0000","last_clean_scrub_stamp":"2023-10-06T05:08:23.648411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:43:49.725374+0000","scrub_duration":0.0014977860000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":45,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-06T05:08:21.469708+0000","last_change":"2023-10-06T05:08:21.469679+0000","last_active":"2023-10-06T05:08:21.469708+0000","last_peered":"2023-10-06T05:08:21.469708+0000","last_clean":"2023-10-06T05:08:21.469708+0000","last_became_active":"2023-10-06T05:06:52.587407+0000","last_became_peered":"2023-10-06T05:06:52.587407+0000","last_unstale":"2023-10-06T05:08:21.469708+0000","last_undegraded":"2023-10-06T05:08:21.469708+0000","last_fullsized":"2023-10-06T05:08:21.469708+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:08:21.469570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:08:21.469570+0000","last_clean_scrub_stamp":"2023-10-06T05:08:21.469570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:22:01.730111+0000","scrub_duration":0.001424437,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":42,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-06T05:08:20.906882+0000","last_change":"2023-10-06T05:08:20.906862+0000","last_active":"2023-10-06T05:08:20.906882+0000","last_peered":"2023-10-06T05:08:20.906882+0000","last_clean":"2023-10-06T05:08:20.906882+0000","last_became_active":"2023-10-06T05:06:52.674952+0000","last_became_peered":"2023-10-06T05:06:52.674952+0000","last_unstale":"2023-10-06T05:08:20.906882+0000","last_undegraded":"2023-10-06T05:08:20.906882+0000","last_fullsized":"2023-10-06T05:08:20.906882+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:08:20.906786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:08:20.906786+0000","last_clean_scrub_stamp":"2023-10-06T05:08:20.906786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:50:14.401073+0000","scrub_duration":0.001294243,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":45,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-06T05:08:20.462839+0000","last_change":"2023-10-06T05:08:20.462804+0000","last_active":"2023-10-06T05:08:20.462839+0000","last_peered":"2023-10-06T05:08:20.462839+0000","last_clean":"2023-10-06T05:08:20.462839+0000","last_became_active":"2023-10-06T05:06:52.587388+0000","last_became_peered":"2023-10-06T05:06:52.587388+0000","last_unstale":"2023-10-06T05:08:20.462839+0000","last_undegraded":"2023-10-06T05:08:20.462839+0000","last_fullsized":"2023-10-06T05:08:20.462839+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:08:20.462708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:08:20.462708+0000","last_clean_scrub_stamp":"2023-10-06T05:08:20.462708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:53:15.776045+0000","scrub_duration":0.001866656,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"46'119","reported_seq":198,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-06T05:08:21.471576+0000","last_change":"2023-10-06T05:08:21.471548+0000","last_active":"2023-10-06T05:08:21.471576+0000","last_peered":"2023-10-06T05:08:21.471576+0000","last_clean":"2023-10-06T05:08:21.471576+0000","last_became_active":"2023-10-06T05:06:46.306107+0000","last_became_peered":"2023-10-06T05:06:46.306107+0000","last_unstale":"2023-10-06T05:08:21.471576+0000","last_undegraded":"2023-10-06T05:08:21.471576+0000","last_fullsized":"2023-10-06T05:08:21.471576+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'119","last_scrub_stamp":"2023-10-06T05:08:21.471458+0000","last_deep_scrub":"46'119","last_deep_scrub_stamp":"2023-10-06T05:08:21.471458+0000","last_clean_scrub_stamp":"2023-10-06T05:08:21.471458+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:23:33.990344+0000","scrub_duration":0.0053759539999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":43,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-06T05:08:26.353688+0000","last_change":"2023-10-06T05:08:26.353636+0000","last_active":"2023-10-06T05:08:26.353688+0000","last_peered":"2023-10-06T05:08:26.353688+0000","last_clean":"2023-10-06T05:08:26.353688+0000","last_became_active":"2023-10-06T05:06:52.588134+0000","last_became_peered":"2023-10-06T05:06:52.588134+0000","last_unstale":"2023-10-06T05:08:26.353688+0000","last_undegraded":"2023-10-06T05:08:26.353688+0000","last_fullsized":"2023-10-06T05:08:26.353688+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:08:26.353498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:08:26.353498+0000","last_clean_scrub_stamp":"2023-10-06T05:08:26.353498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:53:14.491549+0000","scrub_duration":0.0020934460000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":43,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-06T05:08:24.423029+0000","last_change":"2023-10-06T05:08:24.423000+0000","last_active":"2023-10-06T05:08:24.423029+0000","last_peered":"2023-10-06T05:08:24.423029+0000","last_clean":"2023-10-06T05:08:24.423029+0000","last_became_active":"2023-10-06T05:06:52.626937+0000","last_became_peered":"2023-10-06T05:06:52.626937+0000","last_unstale":"2023-10-06T05:08:24.423029+0000","last_undegraded":"2023-10-06T05:08:24.423029+0000","last_fullsized":"2023-10-06T05:08:24.423029+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:08:24.422892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:08:24.422892+0000","last_clean_scrub_stamp":"2023-10-06T05:08:24.422892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:55:23.107581+0000","scrub_duration":0.0019805309999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":42,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-06T05:08:21.648183+0000","last_change":"2023-10-06T05:08:21.648157+0000","last_active":"2023-10-06T05:08:21.648183+0000","last_peered":"2023-10-06T05:08:21.648183+0000","last_clean":"2023-10-06T05:08:21.648183+0000","last_became_active":"2023-10-06T05:06:52.588561+0000","last_became_peered":"2023-10-06T05:06:52.588561+0000","last_unstale":"2023-10-06T05:08:21.648183+0000","last_undegraded":"2023-10-06T05:08:21.648183+0000","last_fullsized":"2023-10-06T05:08:21.648183+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:08:21.648087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:08:21.648087+0000","last_clean_scrub_stamp":"2023-10-06T05:08:21.648087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:44:22.434173+0000","scrub_duration":0.0020664699999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":41,"reported_epoch":46,"state":"active+clean","last_fresh":"2023-10-06T05:08:23.406076+0000","last_change":"2023-10-06T05:08:23.406047+0000","last_active":"2023-10-06T05:08:23.406076+0000","last_peered":"2023-10-06T05:08:23.406076+0000","last_clean":"2023-10-06T05:08:23.406076+0000","last_became_active":"2023-10-06T05:06:52.589246+0000","last_became_peered":"2023-10-06T05:06:52.589246+0000","last_unstale":"2023-10-06T05:08:23.406076+0000","last_undegraded":"2023-10-06T05:08:23.406076+0000","last_fullsized":"2023-10-06T05:08:23.406076+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T05:08:23.405946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T05:08:23.405946+0000","last_clean_scrub_stamp":"2023-10-06T05:08:23.405946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T15:52:00.634924+0000","scrub_duration":0.0015277240000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509467,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297812,"kb_used_data":524,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074028,"statfs":{"total":96636764160,"available":96331804672,"internally_reserved":0,"allocated":536576,"data_stored":238268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.45400000000000001}]},{"osd":1,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.437},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.51300000000000001}]},{"osd":2,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.495}]},{"osd":3,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.57399999999999995}]},{"osd":4,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.376},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.39700000000000002}]},{"osd":5,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.46400000000000002}]},{"osd":7,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.47599999999999998}]}]},{"osd":5,"up_from":15,"seq":64424509467,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297812,"kb_used_data":524,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074028,"statfs":{"total":96636764160,"available":96331804672,"internally_reserved":0,"allocated":536576,"data_stored":238268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.48299999999999998}]},{"osd":1,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.68000000000000005}]},{"osd":2,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.625}]},{"osd":3,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.58799999999999997}]},{"osd":4,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.59999999999999998}]},{"osd":6,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.57199999999999995}]},{"osd":7,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.70799999999999996}]}]},{"osd":0,"up_from":14,"seq":60129542171,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298392,"kb_used_data":1104,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073448,"statfs":{"total":96636764160,"available":96331210752,"internally_reserved":0,"allocated":1130496,"data_stored":828636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":1.2809999999999999,"5min":1.2809999999999999,"15min":1.2809999999999999},"last":0.371},{"interface":"front","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.30499999999999999}]},{"osd":2,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":4.1529999999999996,"5min":4.1529999999999996,"15min":4.1529999999999996},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.372,"5min":0.372,"15min":0.372},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":1.105,"5min":1.105,"15min":1.105},"last":0.34100000000000003}]},{"osd":3,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":1.327,"5min":1.327,"15min":1.327},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":4.1900000000000004,"5min":4.1900000000000004,"15min":4.1900000000000004},"last":0.42199999999999999}]},{"osd":4,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.434}]},{"osd":5,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.304,"5min":1.304,"15min":1.304},"last":0.52100000000000002}]},{"osd":6,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.53900000000000003}]},{"osd":7,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":1.079,"5min":1.079,"15min":1.079},"last":0.499},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.55600000000000005}]}]},{"osd":7,"up_from":13,"seq":55834574874,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298392,"kb_used_data":1104,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073448,"statfs":{"total":96636764160,"available":96331210752,"internally_reserved":0,"allocated":1130496,"data_stored":828636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.46999999999999997}]},{"osd":1,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.5},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.45200000000000001}]},{"osd":2,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.46100000000000002}]},{"osd":3,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.48299999999999998}]},{"osd":4,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.374}]},{"osd":5,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.52000000000000002}]},{"osd":6,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.39100000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542170,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297812,"kb_used_data":524,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074028,"statfs":{"total":96636764160,"available":96331804672,"internally_reserved":0,"allocated":536576,"data_stored":238268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":1.8879999999999999,"5min":1.8879999999999999,"15min":1.8879999999999999},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":1.595,"5min":1.595,"15min":1.595},"last":0.48599999999999999}]},{"osd":2,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.123,"5min":1.123,"15min":1.123},"last":0.437}]},{"osd":3,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.179,"5min":1.179,"15min":1.179},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1.657,"5min":1.657,"15min":1.657},"last":0.53700000000000003}]},{"osd":4,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":1.6319999999999999,"5min":1.6319999999999999,"15min":1.6319999999999999},"last":0.871},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.837,"5min":1.837,"15min":1.837},"last":0.71799999999999997}]},{"osd":5,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.8100000000000001,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.89600000000000002}]},{"osd":6,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.6850000000000001,"5min":1.6850000000000001,"15min":1.6850000000000001},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.8620000000000001,"5min":1.8620000000000001,"15min":1.8620000000000001},"last":0.84899999999999998}]},{"osd":7,"last update":"Fri Oct 6 05:07:45 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.7789999999999999,"5min":1.7789999999999999,"15min":1.7789999999999999},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.93200000000000005}]}]},{"osd":2,"up_from":15,"seq":64424509466,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297812,"kb_used_data":524,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074028,"statfs":{"total":96636764160,"available":96331804672,"internally_reserved":0,"allocated":536576,"data_stored":238268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.311,"5min":0.311,"15min":0.311},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"last":0.378},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.45300000000000001}]},{"osd":1,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"last":0.497}]},{"osd":3,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"last":0.47599999999999998}]},{"osd":4,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.71499999999999997}]},{"osd":5,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.66700000000000004}]},{"osd":6,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.56599999999999995}]},{"osd":7,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.58799999999999997}]}]},{"osd":3,"up_from":15,"seq":64424509466,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297812,"kb_used_data":524,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074028,"statfs":{"total":96636764160,"available":96331804672,"internally_reserved":0,"allocated":536576,"data_stored":238268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":3.391,"5min":3.391,"15min":3.391},"last":0.42899999999999999}]},{"osd":1,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":2.8740000000000001,"5min":2.8740000000000001,"15min":2.8740000000000001},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.376,"5min":0.376,"15min":0.376},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.41199999999999998}]},{"osd":2,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":7.2210000000000001,"5min":7.2210000000000001,"15min":7.2210000000000001},"last":0.44900000000000001}]},{"osd":4,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.018,"5min":1.018,"15min":1.018},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.6930000000000001,"5min":1.6930000000000001,"15min":1.6930000000000001},"last":0.752}]},{"osd":5,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.607,"5min":1.607,"15min":1.607},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.72999999999999998}]},{"osd":6,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.657,"5min":1.657,"15min":1.657},"last":0.68999999999999995}]},{"osd":7,"last update":"Fri Oct 6 05:07:47 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.877,"5min":1.877,"15min":1.877},"last":0.80000000000000004}]}]},{"osd":4,"up_from":15,"seq":64424509467,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297812,"kb_used_data":524,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074028,"statfs":{"total":96636764160,"available":96331804672,"internally_reserved":0,"allocated":536576,"data_stored":238268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.66300000000000003}]},{"osd":1,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.54200000000000004}]},{"osd":2,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.58299999999999996}]},{"osd":3,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":11.385999999999999,"5min":11.385999999999999,"15min":11.385999999999999},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.67700000000000005}]},{"osd":5,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.496},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.70799999999999996}]},{"osd":6,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.68999999999999995}]},{"osd":7,"last update":"Fri Oct 6 05:07:44 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.627},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.72099999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-06T05:08:41.625 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-10-06T05:08:41.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.736+0000 7fba14f6a700 1 Processor -- start 2023-10-06T05:08:41.739 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.737+0000 7fba14f6a700 1 -- start start 2023-10-06T05:08:41.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.737+0000 7fba14f6a700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fba101594d0 0x7fba100cb560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:41.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.737+0000 7fba14f6a700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba100cbee0 con 0x7fba100cbaa0 2023-10-06T05:08:41.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.737+0000 7fba14f6a700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba100cf5d0 con 0x7fba101598a0 2023-10-06T05:08:41.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.737+0000 7fba14f6a700 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba100cf8b0 con 0x7fba101594d0 2023-10-06T05:08:41.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.738+0000 7fba0e59c700 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fba101594d0 0x7fba100cb560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:41.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.738+0000 7fba0ed9d700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fba100cbaa0 0x7fba100c5690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:47442/0 (socket says 172.21.15.98:47442) 2023-10-06T05:08:41.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.738+0000 7fba0ed9d700 1 -- 172.21.15.98:0/3824532819 learned_addr learned my addr 172.21.15.98:0/3824532819 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:08:41.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.738+0000 7fba0e59c700 1 -- 172.21.15.98:0/3824532819 >> v1:172.21.15.174:6789/0 conn(0x7fba101598a0 legacy=0x7fba100ce7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:41.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.738+0000 7fba0e59c700 1 -- 172.21.15.98:0/3824532819 >> v1:172.21.15.98:6789/0 conn(0x7fba100cbaa0 legacy=0x7fba100c5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:41.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.738+0000 7fba0e59c700 1 -- 172.21.15.98:0/3824532819 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba100cfb90 con 0x7fba101594d0 2023-10-06T05:08:41.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.738+0000 7fba0dd9b700 1 -- 172.21.15.98:0/3824532819 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4277095809 0 0) 0x7fba100cbee0 con 0x7fba100cbaa0 2023-10-06T05:08:41.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.738+0000 7fba0e59c700 1 --2- 172.21.15.98:0/3824532819 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba101594d0 0x7fba100cb560 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fba000098e0 tx=0x7fba000091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be1b712efedc94c7 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:41.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.738+0000 7fba0dd9b700 1 -- 172.21.15.98:0/3824532819 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba00009e80 con 0x7fba101594d0 2023-10-06T05:08:41.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.738+0000 7fba0dd9b700 1 -- 172.21.15.98:0/3824532819 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba0000b820 con 0x7fba101594d0 2023-10-06T05:08:41.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.738+0000 7fba0dd9b700 1 -- 172.21.15.98:0/3824532819 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba0000bba0 con 0x7fba101594d0 2023-10-06T05:08:41.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.739+0000 7fba14f6a700 1 -- 172.21.15.98:0/3824532819 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba101594d0 msgr2=0x7fba100cb560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:41.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.739+0000 7fba14f6a700 1 --2- 172.21.15.98:0/3824532819 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba101594d0 0x7fba100cb560 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fba000098e0 tx=0x7fba000091a0 comp rx=0 tx=0).stop 2023-10-06T05:08:41.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.739+0000 7fba14f6a700 1 -- 172.21.15.98:0/3824532819 shutdown_connections 2023-10-06T05:08:41.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.739+0000 7fba14f6a700 1 --2- 172.21.15.98:0/3824532819 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba101594d0 0x7fba100cb560 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:41.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.739+0000 7fba14f6a700 1 -- 172.21.15.98:0/3824532819 >> 172.21.15.98:0/3824532819 conn(0x7fba10089e50 msgr2=0x7fba10099770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:41.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.739+0000 7fba14f6a700 1 -- 172.21.15.98:0/3824532819 shutdown_connections 2023-10-06T05:08:41.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.739+0000 7fba14f6a700 1 -- 172.21.15.98:0/3824532819 wait complete. 2023-10-06T05:08:41.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.740+0000 7fba14f6a700 1 Processor -- start 2023-10-06T05:08:41.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.740+0000 7fba14f6a700 1 -- start start 2023-10-06T05:08:41.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.741+0000 7fba14f6a700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fba100cbaa0 0x7fba100c9c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:41.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.741+0000 7fba14f6a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba101594d0 0x7fba10069670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:41.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.741+0000 7fba14f6a700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fba101598a0 0x7fba1013ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:41.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.741+0000 7fba14f6a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba1015afc0 con 0x7fba101594d0 2023-10-06T05:08:41.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.741+0000 7fba14f6a700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba1015b2a0 con 0x7fba100cbaa0 2023-10-06T05:08:41.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.741+0000 7fba14f6a700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba1015b580 con 0x7fba101598a0 2023-10-06T05:08:41.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.741+0000 7fba05d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba101594d0 0x7fba10069670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:41.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.741+0000 7fba05d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba101594d0 0x7fba10069670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51638/0 (socket says 172.21.15.98:51638) 2023-10-06T05:08:41.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.741+0000 7fba05d9b700 1 -- 172.21.15.98:0/4034498653 learned_addr learned my addr 172.21.15.98:0/4034498653 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:08:41.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.741+0000 7fba0e59c700 1 --2- 172.21.15.98:0/4034498653 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fba100cbaa0 0x7fba100c9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:41.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.741+0000 7fba0ed9d700 1 --2- 172.21.15.98:0/4034498653 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fba101598a0 0x7fba1013ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:41.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.741+0000 7fba05d9b700 1 -- 172.21.15.98:0/4034498653 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fba101598a0 msgr2=0x7fba1013ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:41.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.741+0000 7fba05d9b700 1 --2- 172.21.15.98:0/4034498653 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fba101598a0 0x7fba1013ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:41.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.741+0000 7fba05d9b700 1 -- 172.21.15.98:0/4034498653 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fba100cbaa0 msgr2=0x7fba100c9c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:41.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.741+0000 7fba05d9b700 1 --2- 172.21.15.98:0/4034498653 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fba100cbaa0 0x7fba100c9c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:41.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.741+0000 7fba05d9b700 1 -- 172.21.15.98:0/4034498653 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba00009420 con 0x7fba101594d0 2023-10-06T05:08:41.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.741+0000 7fba0ed9d700 1 --2- 172.21.15.98:0/4034498653 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fba101598a0 0x7fba1013ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:08:41.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.741+0000 7fba05d9b700 1 --2- 172.21.15.98:0/4034498653 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba101594d0 0x7fba10069670 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fb9f800b510 tx=0x7fb9f80090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:41.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.741+0000 7fba06ffd700 1 -- 172.21.15.98:0/4034498653 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9f800f510 con 0x7fba101594d0 2023-10-06T05:08:41.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.741+0000 7fba06ffd700 1 -- 172.21.15.98:0/4034498653 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9f800f670 con 0x7fba101594d0 2023-10-06T05:08:41.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.741+0000 7fba14f6a700 1 -- 172.21.15.98:0/4034498653 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba1015b860 con 0x7fba101594d0 2023-10-06T05:08:41.749 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.741+0000 7fba06ffd700 1 -- 172.21.15.98:0/4034498653 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9f800fa30 con 0x7fba101594d0 2023-10-06T05:08:41.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.741+0000 7fba14f6a700 1 -- 172.21.15.98:0/4034498653 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba1015ac70 con 0x7fba101594d0 2023-10-06T05:08:41.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.742+0000 7fba14f6a700 1 -- 172.21.15.98:0/4034498653 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba100b4970 con 0x7fba101594d0 2023-10-06T05:08:41.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.743+0000 7fba06ffd700 1 -- 172.21.15.98:0/4034498653 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fb9f801a400 con 0x7fba101594d0 2023-10-06T05:08:41.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.743+0000 7fba06ffd700 1 --2- 172.21.15.98:0/4034498653 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fb9f0066f30 0x7fb9f00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:41.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.743+0000 7fba0e59c700 1 --2- 172.21.15.98:0/4034498653 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fb9f0066f30 0x7fb9f00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:41.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.743+0000 7fba06ffd700 1 -- 172.21.15.98:0/4034498653 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5612+0+0 (secure 0 0 0) 0x7fb9f8083ed0 con 0x7fba101594d0 2023-10-06T05:08:41.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.744+0000 7fba0e59c700 1 --2- 172.21.15.98:0/4034498653 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fb9f0066f30 0x7fb9f00693e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fba000030c0 tx=0x7fba00016000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:41.750 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.747+0000 7fba06ffd700 1 -- 172.21.15.98:0/4034498653 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb9f8051220 con 0x7fba101594d0 2023-10-06T05:08:41.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.930+0000 7fba14f6a700 1 -- 172.21.15.98:0/4034498653 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7fba10064a90 con 0x7fba101594d0 2023-10-06T05:08:41.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.934+0000 7fba06ffd700 1 -- 172.21.15.98:0/4034498653 <== mon.0 v2:172.21.15.98:3300/0 7 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fb9f8051220 con 0x7fba101594d0 2023-10-06T05:08:41.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.934+0000 7fba06ffd700 1 -- 172.21.15.98:0/4034498653 <== mon.0 v2:172.21.15.98:3300/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v12)=0 v12) v1 ==== 123+0+0 (secure 0 0 0) 0x7fb9f8054840 con 0x7fba101594d0 2023-10-06T05:08:41.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.937+0000 7fba14f6a700 1 -- 172.21.15.98:0/4034498653 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fb9f0066f30 msgr2=0x7fb9f00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:41.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.937+0000 7fba14f6a700 1 --2- 172.21.15.98:0/4034498653 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fb9f0066f30 0x7fb9f00693e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fba000030c0 tx=0x7fba00016000 comp rx=0 tx=0).stop 2023-10-06T05:08:41.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.937+0000 7fba14f6a700 1 -- 172.21.15.98:0/4034498653 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba101594d0 msgr2=0x7fba10069670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:41.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.937+0000 7fba14f6a700 1 --2- 172.21.15.98:0/4034498653 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba101594d0 0x7fba10069670 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fb9f800b510 tx=0x7fb9f80090f0 comp rx=0 tx=0).stop 2023-10-06T05:08:41.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.937+0000 7fba14f6a700 1 -- 172.21.15.98:0/4034498653 shutdown_connections 2023-10-06T05:08:41.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.937+0000 7fba14f6a700 1 --2- 172.21.15.98:0/4034498653 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fba101598a0 0x7fba1013ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:41.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.937+0000 7fba14f6a700 1 --2- 172.21.15.98:0/4034498653 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fba100cbaa0 0x7fba100c9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:41.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.937+0000 7fba14f6a700 1 --2- 172.21.15.98:0/4034498653 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7fb9f0066f30 0x7fb9f00693e0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:41.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.937+0000 7fba14f6a700 1 --2- 172.21.15.98:0/4034498653 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba101594d0 0x7fba10069670 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:41.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.937+0000 7fba14f6a700 1 -- 172.21.15.98:0/4034498653 >> 172.21.15.98:0/4034498653 conn(0x7fba10089e50 msgr2=0x7fba100688d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:41.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.937+0000 7fba14f6a700 1 -- 172.21.15.98:0/4034498653 shutdown_connections 2023-10-06T05:08:41.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:41.937+0000 7fba14f6a700 1 -- 172.21.15.98:0/4034498653 wait complete. 2023-10-06T05:08:41.951 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-06T05:08:41.951 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-06T05:08:42.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.064+0000 7f88ae680700 1 Processor -- start 2023-10-06T05:08:42.066 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.064+0000 7f88ae680700 1 -- start start 2023-10-06T05:08:42.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.065+0000 7f88ae680700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f88a8140b40 0x7f88a80aec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:42.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.065+0000 7f88ae680700 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88a806d920 con 0x7f88a8140770 2023-10-06T05:08:42.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.065+0000 7f88ae680700 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88a80c6500 con 0x7f88a80af1c0 2023-10-06T05:08:42.069 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.065+0000 7f88ae680700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88a80c67e0 con 0x7f88a8140b40 2023-10-06T05:08:42.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.065+0000 7f88ad67e700 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f88a8140770 0x7f88a806d210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:47444/0 (socket says 172.21.15.98:47444) 2023-10-06T05:08:42.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.065+0000 7f88ad67e700 1 -- 172.21.15.98:0/1044173417 learned_addr learned my addr 172.21.15.98:0/1044173417 (peer_addr_for_me v1:172.21.15.98:0/0) 2023-10-06T05:08:42.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.065+0000 7f889ffff700 1 -- 172.21.15.98:0/1044173417 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2554831648 0 0) 0x7f88a4001110 con 0x7f88a8140770 2023-10-06T05:08:42.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.065+0000 7f88ace7d700 1 --2- 172.21.15.98:0/1044173417 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f88a8140b40 0x7f88a80aec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:42.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.065+0000 7f889ffff700 1 -- 172.21.15.98:0/1044173417 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8894003570 con 0x7f88a8140770 2023-10-06T05:08:42.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.065+0000 7f889ffff700 1 -- 172.21.15.98:0/1044173417 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2484178115 0 0) 0x7f8894003570 con 0x7f88a8140770 2023-10-06T05:08:42.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.065+0000 7f889ffff700 1 -- 172.21.15.98:0/1044173417 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f88a8140b40 msgr2=0x7f88a80aec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:42.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.065+0000 7f889ffff700 1 --2- 172.21.15.98:0/1044173417 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f88a8140b40 0x7f88a80aec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:42.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.065+0000 7f889ffff700 1 -- 172.21.15.98:0/1044173417 >> v1:172.21.15.174:6789/0 conn(0x7f88a80af1c0 legacy=0x7f88a815adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:42.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.065+0000 7f889ffff700 1 -- 172.21.15.98:0/1044173417 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88a80c6ac0 con 0x7f88a8140770 2023-10-06T05:08:42.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.065+0000 7f889ffff700 1 -- 172.21.15.98:0/1044173417 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1543991876 0 0) 0x7f88a4001600 con 0x7f88a8140770 2023-10-06T05:08:42.070 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.066+0000 7f889ffff700 1 -- 172.21.15.98:0/1044173417 >> v1:172.21.15.98:6789/0 conn(0x7f88a8140770 legacy=0x7f88a806d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:42.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.066+0000 7f889ffff700 1 --2- 172.21.15.98:0/1044173417 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8894003a90 0x7f8894003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:42.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.066+0000 7f889ffff700 1 --2- 172.21.15.98:0/1044173417 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88940043e0 0x7f88940048b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:42.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.066+0000 7f889ffff700 1 --2- 172.21.15.98:0/1044173417 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88940050b0 0x7f889400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:42.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.066+0000 7f889ffff700 1 -- 172.21.15.98:0/1044173417 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88a80c67e0 con 0x7f88940050b0 2023-10-06T05:08:42.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.066+0000 7f889ffff700 1 -- 172.21.15.98:0/1044173417 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f889400bfa0 con 0x7f88940043e0 2023-10-06T05:08:42.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.066+0000 7f889ffff700 1 -- 172.21.15.98:0/1044173417 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f889400c160 con 0x7f8894003a90 2023-10-06T05:08:42.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.066+0000 7f88ade7f700 1 --2- 172.21.15.98:0/1044173417 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88940050b0 0x7f889400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:42.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.066+0000 7f88ad67e700 1 --2- 172.21.15.98:0/1044173417 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8894003a90 0x7f8894003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:42.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.066+0000 7f88ace7d700 1 --2- 172.21.15.98:0/1044173417 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88940043e0 0x7f88940048b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:42.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.066+0000 7f88ade7f700 1 -- 172.21.15.98:0/1044173417 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8894003a90 msgr2=0x7f8894003e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:42.071 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.066+0000 7f88ade7f700 1 --2- 172.21.15.98:0/1044173417 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8894003a90 0x7f8894003e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:42.072 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.066+0000 7f88ade7f700 1 -- 172.21.15.98:0/1044173417 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88940043e0 msgr2=0x7f88940048b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T05:08:42.072 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.066+0000 7f88ade7f700 1 --2- 172.21.15.98:0/1044173417 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88940043e0 0x7f88940048b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:42.072 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.066+0000 7f88ade7f700 1 -- 172.21.15.98:0/1044173417 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8894003570 con 0x7f88940050b0 2023-10-06T05:08:42.072 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.066+0000 7f88ade7f700 1 --2- 172.21.15.98:0/1044173417 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88940050b0 0x7f889400ba60 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f88a0001fc0 tx=0x7f88a0001640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee409415ea75d6e2 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:42.072 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.066+0000 7f889ffff700 1 -- 172.21.15.98:0/1044173417 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a0001950 con 0x7f88940050b0 2023-10-06T05:08:42.072 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.066+0000 7f889ffff700 1 -- 172.21.15.98:0/1044173417 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f88a000c8b0 con 0x7f88940050b0 2023-10-06T05:08:42.072 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.066+0000 7f889ffff700 1 -- 172.21.15.98:0/1044173417 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a000cc70 con 0x7f88940050b0 2023-10-06T05:08:42.072 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.067+0000 7f88ae680700 1 -- 172.21.15.98:0/1044173417 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88940050b0 msgr2=0x7f889400ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:42.072 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.067+0000 7f88ae680700 1 --2- 172.21.15.98:0/1044173417 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88940050b0 0x7f889400ba60 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f88a0001fc0 tx=0x7f88a0001640 comp rx=0 tx=0).stop 2023-10-06T05:08:42.072 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.067+0000 7f88ade7f700 1 -- 172.21.15.98:0/1044173417 reap_dead start 2023-10-06T05:08:42.072 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.067+0000 7f88ae680700 1 -- 172.21.15.98:0/1044173417 shutdown_connections 2023-10-06T05:08:42.072 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.067+0000 7f88ae680700 1 -- 172.21.15.98:0/1044173417 >> 172.21.15.98:0/1044173417 conn(0x7f88a8068080 msgr2=0x7f88a80cbec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:42.073 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.067+0000 7f88ae680700 1 -- 172.21.15.98:0/1044173417 shutdown_connections 2023-10-06T05:08:42.073 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.067+0000 7f88ae680700 1 -- 172.21.15.98:0/1044173417 wait complete. 2023-10-06T05:08:42.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.068+0000 7f88ae680700 1 Processor -- start 2023-10-06T05:08:42.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.068+0000 7f88ae680700 1 -- start start 2023-10-06T05:08:42.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.069+0000 7f88ae680700 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f88940050b0 0x7f88a816b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:42.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.069+0000 7f88ae680700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88a81584e0 0x7f88a8158950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:42.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.069+0000 7f88ae680700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88a8166b60 0x7f88a8166fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:42.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.069+0000 7f88ae680700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88a80a5f40 con 0x7f88a8166b60 2023-10-06T05:08:42.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.069+0000 7f88ae680700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88a80a5610 con 0x7f88a81584e0 2023-10-06T05:08:42.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.069+0000 7f88ae680700 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88a80a58f0 con 0x7f88940050b0 2023-10-06T05:08:42.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.069+0000 7f88ade7f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88a8166b60 0x7f88a8166fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:42.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.069+0000 7f88ade7f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88a8166b60 0x7f88a8166fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51654/0 (socket says 172.21.15.98:51654) 2023-10-06T05:08:42.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.069+0000 7f88ade7f700 1 -- 172.21.15.98:0/3583212066 learned_addr learned my addr 172.21.15.98:0/3583212066 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-10-06T05:08:42.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.069+0000 7f88ad67e700 1 --2- 172.21.15.98:0/3583212066 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f88940050b0 0x7f88a816b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:42.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.069+0000 7f88ace7d700 1 --2- 172.21.15.98:0/3583212066 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88a81584e0 0x7f88a8158950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:42.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.069+0000 7f88ade7f700 1 -- 172.21.15.98:0/3583212066 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f88940050b0 msgr2=0x7f88a816b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:42.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.069+0000 7f88ade7f700 1 --2- 172.21.15.98:0/3583212066 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f88940050b0 0x7f88a816b110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:42.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.069+0000 7f88ade7f700 1 -- 172.21.15.98:0/3583212066 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88a81584e0 msgr2=0x7f88a8158950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:42.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.069+0000 7f88ade7f700 1 --2- 172.21.15.98:0/3583212066 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88a81584e0 0x7f88a8158950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:42.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.069+0000 7f88ade7f700 1 -- 172.21.15.98:0/3583212066 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88a80a5bd0 con 0x7f88a8166b60 2023-10-06T05:08:42.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.069+0000 7f88ace7d700 1 --2- 172.21.15.98:0/3583212066 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88a81584e0 0x7f88a8158950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T05:08:42.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.069+0000 7f88ade7f700 1 --2- 172.21.15.98:0/3583212066 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88a8166b60 0x7f88a8166fd0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f88a0004000 tx=0x7f88a0001640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:42.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.070+0000 7f889d7fa700 1 -- 172.21.15.98:0/3583212066 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a0001950 con 0x7f88a8166b60 2023-10-06T05:08:42.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.070+0000 7f889d7fa700 1 -- 172.21.15.98:0/3583212066 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f88a000cea0 con 0x7f88a8166b60 2023-10-06T05:08:42.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.070+0000 7f88ae680700 1 -- 172.21.15.98:0/3583212066 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88a812e850 con 0x7f88a8166b60 2023-10-06T05:08:42.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.070+0000 7f889d7fa700 1 -- 172.21.15.98:0/3583212066 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a0006e10 con 0x7f88a8166b60 2023-10-06T05:08:42.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.070+0000 7f88ae680700 1 -- 172.21.15.98:0/3583212066 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88a816b650 con 0x7f88a8166b60 2023-10-06T05:08:42.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.071+0000 7f88ae680700 1 -- 172.21.15.98:0/3583212066 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88a806abe0 con 0x7f88a8166b60 2023-10-06T05:08:42.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.071+0000 7f889d7fa700 1 -- 172.21.15.98:0/3583212066 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f88a0019070 con 0x7f88a8166b60 2023-10-06T05:08:42.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.071+0000 7f889d7fa700 1 --2- 172.21.15.98:0/3583212066 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f888c066f30 0x7f888c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T05:08:42.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.071+0000 7f889d7fa700 1 -- 172.21.15.98:0/3583212066 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5612+0+0 (secure 0 0 0) 0x7f88a0085b10 con 0x7f88a8166b60 2023-10-06T05:08:42.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.071+0000 7f88ad67e700 1 --2- 172.21.15.98:0/3583212066 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f888c066f30 0x7f888c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T05:08:42.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.072+0000 7f88ad67e700 1 --2- 172.21.15.98:0/3583212066 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f888c066f30 0x7f888c0693e0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f88a4001e70 tx=0x7f88a4007a50 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T05:08:42.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.076+0000 7f889d7fa700 1 -- 172.21.15.98:0/3583212066 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f88a0052fd0 con 0x7f88a8166b60 2023-10-06T05:08:42.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.300+0000 7f88ae680700 1 -- 172.21.15.98:0/3583212066 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f88a80a5d60 con 0x7f88a8166b60 2023-10-06T05:08:42.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.300+0000 7f889d7fa700 1 -- 172.21.15.98:0/3583212066 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+3535 (secure 0 0 0) 0x7f88a0017020 con 0x7f88a8166b60 2023-10-06T05:08:42.303 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-10-06T05:08:42.303 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":19,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4966,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6834/2320896583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2320896583},{"type":"v1","addr":"172.21.15.174:6834","nonce":2320896583}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":4969,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/1423644921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6833","nonce":1423644921},{"type":"v1","addr":"172.21.15.174:6835","nonce":1423644921}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5100,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6836/4291295874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":4291295874},{"type":"v1","addr":"172.21.15.98:6836","nonce":4291295874}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5394,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/3513742657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6835","nonce":3513742657},{"type":"v1","addr":"172.21.15.98:6837","nonce":3513742657}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19}],"filesystems":[]} 2023-10-06T05:08:42.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.302+0000 7f88ae680700 1 -- 172.21.15.98:0/3583212066 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f888c066f30 msgr2=0x7f888c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:42.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.302+0000 7f88ae680700 1 --2- 172.21.15.98:0/3583212066 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f888c066f30 0x7f888c0693e0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f88a4001e70 tx=0x7f88a4007a50 comp rx=0 tx=0).stop 2023-10-06T05:08:42.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.302+0000 7f88ae680700 1 -- 172.21.15.98:0/3583212066 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88a8166b60 msgr2=0x7f88a8166fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T05:08:42.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.302+0000 7f88ae680700 1 --2- 172.21.15.98:0/3583212066 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88a8166b60 0x7f88a8166fd0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f88a0004000 tx=0x7f88a0001640 comp rx=0 tx=0).stop 2023-10-06T05:08:42.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.302+0000 7f88ae680700 1 -- 172.21.15.98:0/3583212066 shutdown_connections 2023-10-06T05:08:42.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.302+0000 7f88ae680700 1 --2- 172.21.15.98:0/3583212066 >> [v2:172.21.15.98:6832/90397,v1:172.21.15.98:6833/90397] conn(0x7f888c066f30 0x7f888c0693e0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:42.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.302+0000 7f88ae680700 1 --2- 172.21.15.98:0/3583212066 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88a8166b60 0x7f88a8166fd0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:42.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.302+0000 7f88ae680700 1 --2- 172.21.15.98:0/3583212066 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f88940050b0 0x7f88a816b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:42.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.302+0000 7f88ae680700 1 --2- 172.21.15.98:0/3583212066 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88a81584e0 0x7f88a8158950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T05:08:42.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.302+0000 7f88ae680700 1 -- 172.21.15.98:0/3583212066 >> 172.21.15.98:0/3583212066 conn(0x7f88a8068080 msgr2=0x7f88a80c0c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T05:08:42.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.303+0000 7f88ae680700 1 -- 172.21.15.98:0/3583212066 shutdown_connections 2023-10-06T05:08:42.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-10-06T05:08:42.303+0000 7f88ae680700 1 -- 172.21.15.98:0/3583212066 wait complete. 2023-10-06T05:08:42.306 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 19 2023-10-06T05:08:42.316 DEBUG:tasks.cephfs.filesystem:already dead... 2023-10-06T05:08:42.317 INFO:teuthology.misc:Shutting down mds daemons... 2023-10-06T05:08:42.317 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-06T05:08:42.317 INFO:teuthology.orchestra.run:waiting for 300 2023-10-06T05:08:42.325 INFO:tasks.ceph.mds.a:Stopped 2023-10-06T05:08:42.326 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-06T05:08:42.326 INFO:teuthology.orchestra.run:waiting for 300 2023-10-06T05:08:42.375 INFO:tasks.ceph.mds.c:Stopped 2023-10-06T05:08:42.375 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-06T05:08:42.375 INFO:teuthology.orchestra.run:waiting for 300 2023-10-06T05:08:42.386 INFO:tasks.ceph.mds.b:Stopped 2023-10-06T05:08:42.386 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-06T05:08:42.386 INFO:teuthology.orchestra.run:waiting for 300 2023-10-06T05:08:42.437 INFO:tasks.ceph.mds.d:Stopped 2023-10-06T05:08:42.437 INFO:teuthology.misc:Shutting down osd daemons... 2023-10-06T05:08:42.437 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-10-06T05:08:42.437 INFO:teuthology.orchestra.run:waiting for 300 2023-10-06T05:08:42.464 INFO:tasks.ceph.osd.0:Stopped 2023-10-06T05:08:42.464 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-10-06T05:08:42.465 INFO:teuthology.orchestra.run:waiting for 300 2023-10-06T05:08:42.533 INFO:tasks.ceph.osd.1:Stopped 2023-10-06T05:08:42.533 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-10-06T05:08:42.533 INFO:teuthology.orchestra.run:waiting for 300 2023-10-06T05:08:42.596 INFO:tasks.ceph.osd.2:Stopped 2023-10-06T05:08:42.596 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-10-06T05:08:42.596 INFO:teuthology.orchestra.run:waiting for 300 2023-10-06T05:08:42.662 INFO:tasks.ceph.osd.3:Stopped 2023-10-06T05:08:42.662 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-10-06T05:08:42.663 INFO:teuthology.orchestra.run:waiting for 300 2023-10-06T05:08:42.699 INFO:tasks.ceph.osd.4:Stopped 2023-10-06T05:08:42.699 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-10-06T05:08:42.699 INFO:teuthology.orchestra.run:waiting for 300 2023-10-06T05:08:42.770 INFO:tasks.ceph.osd.5:Stopped 2023-10-06T05:08:42.771 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2023-10-06T05:08:42.771 INFO:teuthology.orchestra.run:waiting for 300 2023-10-06T05:08:42.839 INFO:tasks.ceph.osd.6:Stopped 2023-10-06T05:08:42.840 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2023-10-06T05:08:42.840 INFO:teuthology.orchestra.run:waiting for 300 2023-10-06T05:08:42.911 INFO:tasks.ceph.osd.7:Stopped 2023-10-06T05:08:42.911 INFO:teuthology.misc:Shutting down mgr daemons... 2023-10-06T05:08:42.912 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-10-06T05:08:42.912 INFO:teuthology.orchestra.run:waiting for 300 2023-10-06T05:08:42.936 INFO:tasks.ceph.mgr.y:Stopped 2023-10-06T05:08:42.936 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-10-06T05:08:42.936 INFO:teuthology.orchestra.run:waiting for 300 2023-10-06T05:08:42.975 INFO:tasks.ceph.mgr.x:Stopped 2023-10-06T05:08:42.975 INFO:teuthology.misc:Shutting down mon daemons... 2023-10-06T05:08:42.975 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-10-06T05:08:42.975 INFO:teuthology.orchestra.run:waiting for 300 2023-10-06T05:08:42.986 INFO:tasks.ceph.mon.a:Stopped 2023-10-06T05:08:42.986 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-10-06T05:08:42.986 INFO:teuthology.orchestra.run:waiting for 300 2023-10-06T05:08:43.025 INFO:tasks.ceph.mon.b:Stopped 2023-10-06T05:08:43.025 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-10-06T05:08:43.025 INFO:teuthology.orchestra.run:waiting for 300 2023-10-06T05:08:43.075 INFO:tasks.ceph.mon.c:Stopped 2023-10-06T05:08:43.075 INFO:tasks.ceph:Checking cluster log for badness... 2023-10-06T05:08:43.076 DEBUG:teuthology.orchestra.run.smithi098:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-10-06T05:08:43.103 INFO:teuthology.orchestra.run.smithi098.stdout:1696568812.576576 mon.a (mon.0) 200 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-10-06T05:08:43.103 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2023-10-06T05:08:43.103 DEBUG:teuthology.orchestra.run.smithi098:> sudo egrep '\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-10-06T05:08:43.170 DEBUG:teuthology.orchestra.run.smithi098:> sudo egrep '\[ERR\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-10-06T05:08:43.239 DEBUG:teuthology.orchestra.run.smithi098:> sudo egrep '\[WRN\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-10-06T05:08:43.306 INFO:teuthology.orchestra.run.smithi098.stdout:1696568812.576576 mon.a (mon.0) 200 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-10-06T05:08:43.306 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi098.front.sepia.ceph.com 2023-10-06T05:08:43.306 DEBUG:teuthology.orchestra.run.smithi098:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-10-06T05:08:43.571 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi098.front.sepia.ceph.com 2023-10-06T05:08:43.572 DEBUG:teuthology.orchestra.run.smithi098:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-10-06T05:08:43.708 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi098.front.sepia.ceph.com 2023-10-06T05:08:43.708 DEBUG:teuthology.orchestra.run.smithi098:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-10-06T05:08:43.767 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi098.front.sepia.ceph.com 2023-10-06T05:08:43.767 DEBUG:teuthology.orchestra.run.smithi098:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-10-06T05:08:43.871 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi174.front.sepia.ceph.com 2023-10-06T05:08:43.871 DEBUG:teuthology.orchestra.run.smithi174:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-10-06T05:08:44.066 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi174.front.sepia.ceph.com 2023-10-06T05:08:44.066 DEBUG:teuthology.orchestra.run.smithi174:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-10-06T05:08:44.163 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi174.front.sepia.ceph.com 2023-10-06T05:08:44.164 DEBUG:teuthology.orchestra.run.smithi174:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2023-10-06T05:08:44.228 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi174.front.sepia.ceph.com 2023-10-06T05:08:44.228 DEBUG:teuthology.orchestra.run.smithi174:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2023-10-06T05:08:44.287 INFO:tasks.ceph:Archiving mon data... 2023-10-06T05:08:44.289 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2023-10-05_23:54:57-fs-wip-yuri3-testing-2023-10-05-0943-quincy-distro-default-smithi/7412314/data/mon.a.tgz 2023-10-06T05:08:44.289 DEBUG:teuthology.orchestra.run.smithi098:> mktemp 2023-10-06T05:08:44.304 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/tmp.AXTvd4N9Dh 2023-10-06T05:08:44.304 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.AXTvd4N9Dh 2023-10-06T05:08:44.432 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0666 /tmp/tmp.AXTvd4N9Dh 2023-10-06T05:08:44.513 DEBUG:teuthology.orchestra.remote:smithi098:/tmp/tmp.AXTvd4N9Dh is 295KB 2023-10-06T05:08:44.609 DEBUG:teuthology.orchestra.run.smithi098:> rm -fr /tmp/tmp.AXTvd4N9Dh 2023-10-06T05:08:44.623 DEBUG:teuthology.misc:Transferring archived files from smithi174:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2023-10-05_23:54:57-fs-wip-yuri3-testing-2023-10-05-0943-quincy-distro-default-smithi/7412314/data/mon.b.tgz 2023-10-06T05:08:44.624 DEBUG:teuthology.orchestra.run.smithi174:> mktemp 2023-10-06T05:08:44.639 INFO:teuthology.orchestra.run.smithi174.stdout:/tmp/tmp.uhIvfb598e 2023-10-06T05:08:44.639 DEBUG:teuthology.orchestra.run.smithi174:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.uhIvfb598e 2023-10-06T05:08:44.779 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod 0666 /tmp/tmp.uhIvfb598e 2023-10-06T05:08:44.860 DEBUG:teuthology.orchestra.remote:smithi174:/tmp/tmp.uhIvfb598e is 392KB 2023-10-06T05:08:44.958 DEBUG:teuthology.orchestra.run.smithi174:> rm -fr /tmp/tmp.uhIvfb598e 2023-10-06T05:08:44.972 DEBUG:teuthology.misc:Transferring archived files from smithi174:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2023-10-05_23:54:57-fs-wip-yuri3-testing-2023-10-05-0943-quincy-distro-default-smithi/7412314/data/mon.c.tgz 2023-10-06T05:08:44.973 DEBUG:teuthology.orchestra.run.smithi174:> mktemp 2023-10-06T05:08:45.026 INFO:teuthology.orchestra.run.smithi174.stdout:/tmp/tmp.fv9KGAWefh 2023-10-06T05:08:45.027 DEBUG:teuthology.orchestra.run.smithi174:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.fv9KGAWefh 2023-10-06T05:08:45.160 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod 0666 /tmp/tmp.fv9KGAWefh 2023-10-06T05:08:45.242 DEBUG:teuthology.orchestra.remote:smithi174:/tmp/tmp.fv9KGAWefh is 393KB 2023-10-06T05:08:45.308 DEBUG:teuthology.orchestra.run.smithi174:> rm -fr /tmp/tmp.fv9KGAWefh 2023-10-06T05:08:45.322 INFO:tasks.ceph:Cleaning ceph cluster... 2023-10-06T05:08:45.322 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-06T05:08:45.325 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-06T05:08:45.386 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-10-06T05:08:45.386 DEBUG:teuthology.orchestra.run.smithi098:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-06T05:08:45.392 DEBUG:teuthology.orchestra.run.smithi174:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-06T05:08:45.416 INFO:teuthology.orchestra.run.smithi098.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-06T05:08:45.453 INFO:teuthology.orchestra.run.smithi174.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-06T05:08:45.459 INFO:tasks.ceph:Archiving crash dumps... 2023-10-06T05:08:45.460 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-10-05_23:54:57-fs-wip-yuri3-testing-2023-10-05-0943-quincy-distro-default-smithi/7412314/remote/smithi098/crash 2023-10-06T05:08:45.461 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-06T05:08:45.489 DEBUG:teuthology.misc:Transferring archived files from smithi174:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-10-05_23:54:57-fs-wip-yuri3-testing-2023-10-05-0943-quincy-distro-default-smithi/7412314/remote/smithi174/crash 2023-10-06T05:08:45.490 DEBUG:teuthology.orchestra.run.smithi174:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-06T05:08:45.522 INFO:tasks.ceph:Shutting down logrotate 2023-10-06T05:08:45.523 DEBUG:teuthology.orchestra.run.smithi098:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-06T05:08:45.534 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-06T05:08:45.635 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-10-06T05:08:45.660 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-10-06T05:08:45.684 INFO:tasks.ceph:Compressing logs... 2023-10-06T05:08:45.684 DEBUG:teuthology.orchestra.run.smithi098:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-06T05:08:45.702 DEBUG:teuthology.orchestra.run.smithi174:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-06T05:08:48.968 INFO:tasks.ceph:Archiving logs... 2023-10-06T05:08:48.968 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-10-05_23:54:57-fs-wip-yuri3-testing-2023-10-05-0943-quincy-distro-default-smithi/7412314/remote/smithi098/log 2023-10-06T05:08:48.969 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-06T05:08:49.551 DEBUG:teuthology.misc:Transferring archived files from smithi174:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-10-05_23:54:57-fs-wip-yuri3-testing-2023-10-05-0943-quincy-distro-default-smithi/7412314/remote/smithi174/log 2023-10-06T05:08:49.552 DEBUG:teuthology.orchestra.run.smithi174:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-06T05:08:49.910 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-06T05:08:49.918 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-10-06T05:08:49.919 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-06T05:08:49.921 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-06T05:08:50.014 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-10-06T05:08:50.023 INFO:teuthology.task.clock:Checking final clock skew... 2023-10-06T05:08:50.024 DEBUG:teuthology.orchestra.run.smithi098:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-06T05:08:50.026 DEBUG:teuthology.orchestra.run.smithi174:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-06T05:08:50.038 INFO:teuthology.orchestra.run.smithi098.stderr:bash: ntpq: command not found 2023-10-06T05:08:50.039 INFO:teuthology.orchestra.run.smithi174.stderr:bash: ntpq: command not found 2023-10-06T05:08:50.041 INFO:teuthology.orchestra.run.smithi098.stdout:210 Number of sources = 4 2023-10-06T05:08:50.042 INFO:teuthology.orchestra.run.smithi098.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-06T05:08:50.042 INFO:teuthology.orchestra.run.smithi098.stdout:=============================================================================== 2023-10-06T05:08:50.042 INFO:teuthology.orchestra.run.smithi098.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 37 +262us[ +262us] +/- 87ms 2023-10-06T05:08:50.042 INFO:teuthology.orchestra.run.smithi098.stdout:^* hv02.front.sepia.ceph.com 3 6 377 38 -348us[ -352us] +/- 59ms 2023-10-06T05:08:50.042 INFO:teuthology.orchestra.run.smithi098.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 36 +200us[ +200us] +/- 65ms 2023-10-06T05:08:50.042 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-10-06T05:08:50.043 INFO:teuthology.orchestra.run.smithi174.stdout:210 Number of sources = 4 2023-10-06T05:08:50.043 INFO:teuthology.orchestra.run.smithi174.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-06T05:08:50.043 INFO:teuthology.orchestra.run.smithi174.stdout:=============================================================================== 2023-10-06T05:08:50.043 INFO:teuthology.orchestra.run.smithi174.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 37 +269us[ +269us] +/- 87ms 2023-10-06T05:08:50.043 INFO:teuthology.orchestra.run.smithi174.stdout:^* hv02.front.sepia.ceph.com 3 6 377 37 -342us[ -329us] +/- 59ms 2023-10-06T05:08:50.043 INFO:teuthology.orchestra.run.smithi174.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 36 +187us[ +187us] +/- 65ms 2023-10-06T05:08:50.044 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-10-06T05:08:50.044 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-10-06T05:08:50.053 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-10-06T05:08:50.053 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-10-06T05:08:50.062 DEBUG:teuthology.orchestra.run.smithi098:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-06T05:08:50.178 DEBUG:teuthology.orchestra.run.smithi174:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-06T05:08:50.267 DEBUG:teuthology.orchestra.run.smithi098:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-06T05:08:50.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-06T05:08:50.297 DEBUG:teuthology.orchestra.run.smithi174:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-06T05:08:50.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-06T05:08:50.326 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-10-06T05:08:50.352 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-10-06T05:08:50.361 INFO:teuthology.task.internal:Duration was 793.437578 seconds 2023-10-06T05:08:50.362 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-10-06T05:08:50.371 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-10-06T05:08:50.371 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-06T05:08:50.374 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-06T05:08:50.416 INFO:teuthology.orchestra.run.smithi098.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-06T05:08:50.418 INFO:teuthology.orchestra.run.smithi174.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-06T05:08:50.664 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-10-06T05:08:50.665 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi098.front.sepia.ceph.com 2023-10-06T05:08:50.665 DEBUG:teuthology.orchestra.run.smithi098:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-06T05:08:50.724 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi174.front.sepia.ceph.com 2023-10-06T05:08:50.725 DEBUG:teuthology.orchestra.run.smithi174:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-06T05:08:50.744 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-10-06T05:08:50.744 DEBUG:teuthology.orchestra.run.smithi098:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-06T05:08:50.767 DEBUG:teuthology.orchestra.run.smithi174:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-06T05:08:50.823 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-10-06T05:08:50.823 DEBUG:teuthology.orchestra.run.smithi098:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-06T05:08:50.847 DEBUG:teuthology.orchestra.run.smithi174:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-06T05:08:50.936 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-10-06T05:08:50.946 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-10-06T05:08:50.946 DEBUG:teuthology.orchestra.run.smithi098:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-06T05:08:50.970 DEBUG:teuthology.orchestra.run.smithi174:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-06T05:08:50.995 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-10-06T05:08:51.005 DEBUG:teuthology.orchestra.run.smithi098:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-06T05:08:51.013 DEBUG:teuthology.orchestra.run.smithi174:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-06T05:08:51.034 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern = core 2023-10-06T05:08:51.058 INFO:teuthology.orchestra.run.smithi174.stdout:kernel.core_pattern = core 2023-10-06T05:08:51.072 DEBUG:teuthology.orchestra.run.smithi098:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-06T05:08:51.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-06T05:08:51.102 DEBUG:teuthology.orchestra.run.smithi174:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-06T05:08:51.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-06T05:08:51.127 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-10-06T05:08:51.137 INFO:teuthology.task.internal:Transferring archived files... 2023-10-06T05:08:51.137 DEBUG:teuthology.misc:Transferring archived files from smithi098:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-10-05_23:54:57-fs-wip-yuri3-testing-2023-10-05-0943-quincy-distro-default-smithi/7412314/remote/smithi098 2023-10-06T05:08:51.138 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-06T05:08:51.177 DEBUG:teuthology.misc:Transferring archived files from smithi174:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-10-05_23:54:57-fs-wip-yuri3-testing-2023-10-05-0943-quincy-distro-default-smithi/7412314/remote/smithi174 2023-10-06T05:08:51.178 DEBUG:teuthology.orchestra.run.smithi174:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-06T05:08:51.213 INFO:teuthology.task.internal:Removing archive directory... 2023-10-06T05:08:51.214 DEBUG:teuthology.orchestra.run.smithi098:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-06T05:08:51.216 DEBUG:teuthology.orchestra.run.smithi174:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-06T05:08:51.262 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-10-06T05:08:51.278 INFO:teuthology.task.internal:Not uploading archives. 2023-10-06T05:08:51.278 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-10-06T05:08:51.288 INFO:teuthology.task.internal:Tidying up after the test... 2023-10-06T05:08:51.288 DEBUG:teuthology.orchestra.run.smithi098:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-06T05:08:51.290 DEBUG:teuthology.orchestra.run.smithi174:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-06T05:08:51.302 INFO:teuthology.orchestra.run.smithi098.stdout: 263007 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 6 05:08 /home/ubuntu/cephtest 2023-10-06T05:08:51.317 INFO:teuthology.orchestra.run.smithi174.stdout: 263007 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 6 05:08 /home/ubuntu/cephtest 2023-10-06T05:08:51.318 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-10-06T05:08:51.329 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-10-06T05:08:51.351 INFO:teuthology.nuke:Checking targets against current locks 2023-10-06T05:08:51.375 DEBUG:teuthology.nuke:shortname: smithi098 2023-10-06T05:08:51.375 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-06T05:08:51.400 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi098.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-05_23:54:57-fs-wip-yuri3-testing-2023-10-05-0943-quincy-distro-default-smithi/7412314', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-06 04:48:18.746010', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAdLN1jqITOyOs2/LG9xBqbeR1dU57UfESY/Js/od2mIfqcEEbswvtH7X0gj0zBZVDFXiTLyYoFCX8PqhzYepWI='} 2023-10-06T05:08:51.403 DEBUG:teuthology.nuke:shortname: smithi174 2023-10-06T05:08:51.404 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-06T05:08:51.429 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi174.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-05_23:54:57-fs-wip-yuri3-testing-2023-10-05-0943-quincy-distro-default-smithi/7412314', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-06 04:48:18.747732', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKTO+hRfafQJmPn7d061Qdv6KXPKph9+gBvigvYovlxzFKbU3ueuwcnmiJKd4IHyY+NWJGJPMku58GXgpv6R/oQ='} 2023-10-06T05:08:51.456 INFO:teuthology.orchestra.console.smithi098:Power off 2023-10-06T05:08:51.456 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-06T05:08:51.478 DEBUG:teuthology.orchestra.console.smithi098:power off output: Chassis Power Control: Down/Off 2023-10-06T05:08:51.478 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-06T05:08:51.493 INFO:teuthology.orchestra.console.smithi174:Power off 2023-10-06T05:08:51.493 DEBUG:teuthology.orchestra.console.smithi174:pexpect command: ipmitool -H smithi174.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-06T05:08:51.505 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is on 2023-10-06T05:08:51.517 DEBUG:teuthology.orchestra.console.smithi174:power off output: Chassis Power Control: Down/Off 2023-10-06T05:08:51.517 DEBUG:teuthology.orchestra.console.smithi174:pexpect command: ipmitool -H smithi174.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-06T05:08:51.552 DEBUG:teuthology.orchestra.console.smithi174:check power output: Chassis Power is on 2023-10-06T05:08:55.506 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-06T05:08:55.553 DEBUG:teuthology.orchestra.console.smithi174:pexpect command: ipmitool -H smithi174.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-06T05:08:55.615 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is on 2023-10-06T05:08:55.663 DEBUG:teuthology.orchestra.console.smithi174:check power output: Chassis Power is on 2023-10-06T05:08:59.618 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-06T05:08:59.665 DEBUG:teuthology.orchestra.console.smithi174:pexpect command: ipmitool -H smithi174.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-06T05:08:59.731 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is on 2023-10-06T05:08:59.775 DEBUG:teuthology.orchestra.console.smithi174:check power output: Chassis Power is off 2023-10-06T05:08:59.876 INFO:teuthology.orchestra.console.smithi174:Power off completed 2023-10-06T05:09:03.736 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-06T05:09:03.845 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is off 2023-10-06T05:09:03.946 INFO:teuthology.orchestra.console.smithi098:Power off completed 2023-10-06T05:09:04.050 INFO:teuthology.run:Summary data: description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-readahead} duration: 793.437578201294 failure_reason: '"1696568812.576576 mon.a (mon.0) 200 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log' flavor: default owner: scheduled_yuriw@teuthology success: false 2023-10-06T05:09:04.051 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-06T05:09:04.117 INFO:teuthology.run:FAIL